Vizier

Data curation (also called preparation, wrangling, or cleaning) is a critical stage in data science in which raw data is structured, validated, and repaired. Data validation and repair establish trust in analytical results, while appropriate structuring streamlines analytics. A new collaboration between the University at Buffalo, New York University, and the Illinois Institute of Technology is looking to build software to streamline this process, making it easier and faster to explore and analyze raw data.

Our goal is to make curation more...

efficient, by modeling your workflow and using the model to offer suggestions.
simple, by using educated guesses to set default parameters.
reliable, by tracking changes and guesses, and communicating their effects on analytical results.
exploratory, by making it easier to quickly "try on" some changes.
speedy, by immediately updating visualizations to reflect small changes to your data or workflow.

Our tool, Vizier, will combine a simple "notebook-style" interface based on JuPyTer with powerful back-end tools that track changes, edits, and the effects of automation. These forms of "provenance" capture both the exploratory curation process---how the cleaning workflows evolve, and how data changes over time. By connecting these different types of provenance, Vizier will not only support the auditing of curation processes, but also explain the context in which they were applied, making it faster and easier to curate data.

Provenance for Curation and Exploration

Vizier enables worry-free exploration. A simple notebook interface mirrors a spreadsheet view of your data, tracking the provenance of your edits. Provenance is at the heart of Vizier, making it easy to undo and redo actions and allowing Vizier to suggest new curation steps, visualizations, or to make guesses about your data. Finally provenance allows you to develop curation workflows on small data sets and then seamlessly deploy them to larger datasets (e.g., via Spark or Hadoop)

The Team

Heiko Mueller (NYU)
Jacob Varghese (UB)
Juliana Freire (NYU)
Oliver Kennedy (UB)
Boris Glavic (IIT)

Publications

Overlay Spreadsheets HILDA ( preprint | slides | video )

Social determinants of health derived from people with opioid use disorder: Improving data collection, integration and use with cross-domain collaboration and reproducible, data-centric, notebook-style workflows Frontiers in Medicine ( paper )

Efficient Approximation of Certain and Possible Answers for Ranking and Window Queries over Uncertain Data pVLDB ( paper )

The Right Tool for the Job: Data-Centric Workflows in Vizier IEEE-DEB ( paper )

Runtime Provenance Refinement for Notebooks TaPP ( paper )

Efficient Uncertainty Tracking for Complex Queries with Attribute-level Bounds SIGMOD ( paper )

Reducing Ambiguity in Json Schema Discovery SIGMOD ( paper )

DataSense: Display Agnostic Data Documentation CIDR ( abstract | extended )

Loki: Streamlining Integration and Enrichment HILDA ( paper )

Your notebook is not crumby enough, REPLace it CIDR ( short | full )

CAPE: Explaining Outliers by Counterbalancing pVLDB

Data Debugging and Exploration with Vizier SIGMOD ( paper )

Uncertainty Annotated Databases - A Lightweight Approach for Approximating Certain Answers SIGMOD ( paper | extended )

SchemaDrill: Interactive Semi-Structured Schema Design HILDA ( paper )

Uncertainty Annotated Databases - A Lightweight Approach for Dealing with Uncertainty IIT-TR ( report )

Adaptive Schema Databases CIDR ( paper )

The Exception That Improves The Rule HILDA ( paper | slides )

Provenance-aware Versioned Dataworkspaces TaPP ( paper )