Collaborative Data Sharing with Mappings and Provenance

Abstract

A key challenge in science today involves integrating data from databases managed by different collaborating scientists. In this dissertation, we develop the foundations and applications of collaborative data sharing systems (CDSSs), which address this challenge. A CDSS allows collaborators to define loose confederations of heterogeneous databases, relating them through schema mappings that establish how data should flow from one site to the next. In addition to simply propagating data along the mappings, it is critical to record data provenance (annotations describing where and how data originated) and to support policies allowing scientists to specify whose data they trust, and when. Since a large data sharing confederation is certain to evolve over time, the CDSS must also efficiently handle incremental changes to data, schemas, and mappings. We focus in this dissertation on the formal foundations of CDSSs, as well as practical issues of its implementation in a prototype CDSS called Orchestra. We propose a novel model of data provenance appropriate for CDSSs, based on a framework of semiring-annotated relations. This framework elegantly generalizes a number of other important database semantics involving annotated relations, including ranked results, prior provenance models, and probabilistic databases. We describe the design and implementation of the Orchestra prototype, which supports update propagation across schema mappings while maintaining data provenance and filtering data according to trust policies. We investigate fundamental questions of query containment and equivalence in the context of provenance information. We use the results of these investigations to develop novel approaches to efficiently propagating changes to data and mappings in a CDSS. Our approaches highlight unexpected connections between the two problems and with the problem of optimizing queries using materialized views. Finally, we show that semiring annotations also make sense for XML and nested relational data, paving the way towards a future extension of CDSS to these richer data models.

Full text

Page 1

COLLABORATIVE DATA SHARING WITH MAPPINGS AND
PROVENANCE
Todd J. Green
A DISSERTATION
in
Computer and Information Science
Presented to the Faculties of the University of Pennsylvania in Partial
Fulfillment of the Requirements for the Degree of Doctor of Philosophy
2009
Zachary G. Ives
Supervisor of Dissertation
Val Tannen
Supervisor of Dissertation
Jianbo Shi, Associate Professor, Computer and Information Science
Graduate Group Chairperson
Dissertation Committee
Susan B. Davidson, Professor, Computer and Information Science
Sanjeev Khanna, Professor, Computer and Information Science
Jeffrey F. Naughton, Professor, Computer Science, University of Wisconsin-Madison
Benjamin C. Pierce, Professor, Computer and Information Science

Page 2

COPYRIGHT
Todd J. Green
2009

Page 3

To Elisabeth
iii

Page 4

ABSTRACT
COLLABORATIVE DATA SHARING WITH MAPPINGS AND PROVENANCE
Todd J. Green
Supervisors: Zachary G. Ives and Val Tannen
A key challenge in science today involves integrating data from databases managed by differ-
ent collaborating scientists. In this dissertation, we develop the foundations and applications of
collaborative data sharing systems (CDSSs), which address this challenge. A CDSS allows collab-
orators to define loose confederations of heterogeneous databases, relating them through schema
mappings that establish how data should flow from one site to the next. In addition to simply
propagating data along the mappings, it is critical to record data provenance (annotations describ-
ing where and how data originated) and to support policies allowing scientists to specify whose
data they trust, and when. Since a large data sharing confederation is certain to evolve over time,
the CDSS must also efficiently handle incremental changes to data, schemas, and mappings.
We focus in this dissertation on the formal foundations of CDSSs, as well as practical issues
of its implementation in a prototype CDSS called Orchestra. We propose a novel model of data
provenance appropriate for CDSSs, based on a framework of semiring-annotated relations. This
framework elegantly generalizes a number of other important database semantics involving an-
notated relations, including ranked results, prior provenance models, and probabilistic databases.
We describe the design and implementation of the Orchestra prototype, which supports update
propagation across schema mappings while maintaining data provenance and filtering data ac-
cording to trust policies. We investigate fundamental questions of query containment and equiv-
alence in the context of provenance information. We use the results of these investigations to
develop novel approaches to efficiently propagating changes to data and mappings in a CDSS.
Our approaches highlight unexpected connections between the two problems and with the prob-
lem of optimizing queries using materialized views. Finally, we show that semiring annotations
also make sense for XML and nested relational data, paving the way towards a future extension
of CDSS to these richer data models.
iv

Page 5

Contents
1Introduction1
1.1Overview of CDSS and Orchestra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
1.2Overview of Technical Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
1.3Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25
2Provenance Semirings27
2.1Queries on Annotated Relations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28
2.2Positive Relational Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
2.3Polynomials for Provenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
2.4 A Hierarchy of Provenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36
2.5Datalog on K-Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
2.6Formal Power Series for Provenance . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
2.7Computing Provenance Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46
2.8Application to Incomplete/Probabilistic Databases. . . . . . . . . . . . . . . . . . .49
2.9Provenance-Annotated Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49
3Update Exchange in Orchestra53
3.1CDSS Update Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
3.2Update Exchange Formalized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.3Performing Update Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67
3.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75
3.5Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4Optimizing Queries on Annotated Relations 85
4.1Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87
4.2Containment Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89
4.3Bounds from Semiring Homomorphisms . . . . . . . . . . . . . . . . . . . . . . . . .90
v