Executive Summary
Data wrangling is an essential step for curating any new dataset into a form suitable for analytics.  The data must be coerced into a relational schema, validated, and any errors must be repaired.  Unfortunately, data wrangling takes time, effort, and money, most of which will go to curate data that never actually gets queried.  Worse still, these costs are ongoing: data sources, formats, and curation requirements regularly change over time, requiring continuous effort to maintain data sets that are both properly curated and up-to-date.  The Mimir project has been working to limit the need for up-front data wrangling by enabling a deferred, 'dynamic' form of data curation, where manual wrangling is only used when absolutely required.  

Our initial efforts focused on repairing data-level errors.  We next propose to build on these efforts by developing a data model and interactive analytics environment for repairing schema-level errors such as mismatched schemas, untyped data (e.g., CSV or JSON), and hierarchical data shredding.

Technical Objectives
-	Extend Mimir with a data model that supports schema-level uncertainty.
-	Develop a 360 degree lens.
-	Develop Mimir’s interface to enable intuitive, interactive, on-demand schema construction.
-	Improve Mimir’s techniques for presenting uncertainty to end users, focusing on presentations of schema-level uncertainty.

Research Summary
Our preliminary work resulted in ‘lenses’, a relational operator that applies a data cleaning heuristic (e.g., data interpolation), but requires little to no upfront configuration or validation.  Instead of asking the user to manually perform data wrangling tasks before analyzing her data, the Mimir system makes a best-effort heuristic guess.  Mimir annotates the output of lenses with provenance markers that track what query results are affected by these guesses, and how big of an effect the guess has.  

The majority of our efforts have so far focused on lenses that validate and repair data.  A ‘domain constraint repair’ lens allows users to validate individual fields, using simple machine learning to make an educated guess about how the data should be fixed.  An ‘archival’ lens uses periodic batch-copies (e.g., OLTP infrastructure to OLAP infrastructure) annotated with error terms that measure how stale the data is.  

Our next goal is to apply the same methodology to create an adaptive, interactive tool for on-demand schema construction and schema mapping, two of the remaining parts of the data wrangling process, and time permitting, to incorporate these techniques into a variety of back-end data management infrastructures for in-situ analytics.  Once complete, Mimir will support interactive exploration using SQL queries over raw data files (i.e., in-situ) provided in CSV, XML, and JSON formats, and stored on local filesystems, in HDFS, or already loaded into a backend database.  In addition to building on the prior efforts of Mimir, we will incorporate existing work on In-Situ query processing (Alagiannis et. al. 2012), Schema-Free Databases (Li et. al. 2014), Flexible Schema Data (Liu/Gawlick 2015) and Dynamic Hierarchical Schema Mapping (Dyreson et al. 2010).  We anticipate three primary challenges

The first challenge will be to develop a data and query model that captures the possibility of a probabilistic schema.  One approach to this problem was initially proposed by Li et al in ‘Schema-Free Databases’, which introduced a database that could automatically detect and repair minor errors in queries such as typos in column/table names or missing join conditions.  We will create a data model based on a virtual schema, or ‘360 degree lens’.  Rather than identifying one specific database configuration, the lens will identify a set of possible database schemas and mappings between them: 
- CSV files do not contain type information, but simple schemes like majority vote can be used to identify a set of possible types.
- Hierarchical data models (e.g., JSON and XML) can be shredded into a flat relational schema, but admit multiple different key/foreign key relationship patterns.
- JSON (and to a lesser degree XML) files are not required to conform to a rigid schema, permitting the use of multiple distinct flat relational schemas for the same data (e.g., see the DataGuide project).  A problem specific to hierarchical data like JSON or XML is that more than one entity is typically captured in the same data set. Data stored in NoSQL stores that lack the capability of joins typically has even more entities stored in the same record. An example is a MongoDB project [Migrating from Oracle: How Apollo Group evaluated MongoDB] where one JSON record contains the entities Student, Teacher and Course in one hierarchy. In order to join such record with other data sources (e.g. relational or JSON) one would have to identify entities based on subschema and subschema-mappings. 

- Flat relational queries admit multiple different interpretations over hierarchical data.  For example, take the query:
	SELECT Name FROM People WHERE Address.City = ‘Buffalo’
which could conceivably be issued on the JSON object:
	[ { name: ’Bob’, address: [ 
		{ city: ‘Buffalo’, … }, 
		{ city: ’Redwood Shores’, … } 
	    ] }, … ]
This query could be interpreted as an EXISTS query (The name of anyone with an address in ‘Buffalo’, a FOR ALL query (The name of anyone who lives only in Buffalo).  
- CSV files are not required to contain attribute names; It may be possible to use, for example, freely available ontological relationships to assign possible labels to attributes in a CSV file.

Our first goal is to develop a model for databases with probabilistic schemas.  This model will need to admit efficient enumeration of possible schemas, efficient assignment of probabilities or weights to each possible schema.

The second challenge will be to identify vectors for extracting implicit feedback about the schema from users to supplement manual feedback.  One potentially useful vector for feedback is the queries themselves.  By issuing queries, database users can implicitly define foreign key references through equi-joins, and implicitly assign schemas to the input relations by using columns.  This information can be collected and used to refine the 360 degree lens.

Schemas should be generates/derived/ mapped for single collections (tables) as well as across collections (tables) to enable (suggest) joins. For Example: Detect ‘Courses’ in Student/Teacher/Course hierarchy and allow then to join with a relational course table. 

Our final goal is to implement the lens.  The primary challenge will be to develop a technique for efficiently enumerating possible schemas that are viable for a given query, exploring small perturbations of an invalid query that make possible schemas viable, and identifying a most-likely query/schema pair to use when producing initial results for the user.  We will also have to produce a query rewriter that transforms this flat relational query over the virtual schema into an in-situ query posed over the raw data.  Time permitting, we will also explore techniques for efficiently computing result quality metrics such as bounds, confidence levels, entropy, and variance.

Prior Work
To date, the Mimir project has produced 2 workshop papers (Yang 2014, Kennedy et. al. 2014), a VLDB paper (Yang et. al. 2015) and a Technical Report (Nandi/Yang et. al. 2015) currently under submission to SIGMOD 2016.  The project has also produced a proof-of-concept software tool: The Mimir Database Overlay, also under submission to the demo track of SIGMOD 2016.  Mimir uses JDBC to wrap around an existing database such as Oracle, and allows users to pose queries that combine dynamically curated data-sources with data already in the database.  Although Mimir supports a command-line interface, one of our key accomplishments is a browser-based graphical frontend that enables interactive data exploration.  We have been exploring techniques for effectively presenting uncertainty to database users.  Our current approach takes a gradual approach, first identifying which results depend on a heuristic guess, and then explaining the nature and impact of these guesses in simple user-friendly English terms.  A demo screencast and further details are available on our project page.

Benefits to Oracle
Dynamic curation is important for classical ETL data warehouses/data-marts/data lakes.  Additionally, an opportunity exists to revolutionize cloud data analytic services for the general public.  Using Mimir, even a novice with minimal knowledge of SQL, E/R design, or statistics can reliably explore and visualize the ever-increasing array of publicly available "open-data" resources.  With a growing availability of ‘free’ open-source database (-like) products Oracle need to provide higher-level services to its customers. Allowing a DBA and/or data-analyst to quickly analyze and query data in different formats from different sources is a great time/cost saver and guarantees that Oracle continues to dominate the Data Warehousing market even if customers chose to store their data outside of the Oracle Database. Identifiying schema with personal data (credit cards, social security numbers, health?) would also position this product as a security/data privacy feature which is very important for JSON where DBAs have little control over what fields are inserted by developers into the JSON objects.

In short, the lens abstraction making E/F Codd's E/R model and SQL relational algebra accessible to the public for data analytical services in the cloud.  This stands to significantly benefit Oracle, which is a provider of both cloud and classical DB services and software.