UW Interactive Data Lab
IDL logo

Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment

Sean Kandel, Ravi Parikh, Andreas Paepcke, Joseph Hellerstein, Jeffrey Heer. Advanced Visual Interfaces, 2012
Figure for Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment
The Profiler Architecture. An (a) input table is analyzed to (b) infer types for each column. Type information is used to (c) generate features prior to running (d) anomaly detection routines. The results of anomaly detection and mutual information analysis are used to perform (e) view recommendation and populate a set of (f) interactive visualizations.
Materials
Abstract
Data quality issues such as missing, erroneous, extreme and duplicate values undermine analysis and are time-consuming to find and fix. Automated methods can help identify anomalies, but determining what constitutes an error is context-dependent and so requires human judgment. While visualization tools can facilitate this process, analysts must often manually construct the necessary views, requiring significant expertise. We present Profiler, a visual analysis tool for assessing quality issues in tabular data. Profiler applies data mining methods to automatically flag problematic data and suggests coordinated summary visualizations for assessing the data in context. The system contributes novel methods for integrated statistical and visual analysis, automatic view suggestion, and scalable visual summaries that support real-time interaction with millions of data points. We present Profiler's architecture -- including modular components for custom data types, anomaly detection routines and summary visualizations -- and describe its application to motion picture, natural disaster and water quality data sets.
BibTeX
@inproceedings{2012-profiler,
  title = {Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment},
  author = {Kandel, Sean AND Parikh, Ravi AND Paepcke, Andreas AND Hellerstein, Joseph AND Heer, Jeffrey},
  booktitle = {Advanced Visual Interfaces},
  year = {2012},
  url = {https://idl.uw.edu/papers/profiler},
  doi = {10.1145/2254556.2254659}
}