Abstract:In enterprise data pipelines, data insertions occur periodically and may impact downstream services if data quality issues are not addressed. Typically, such problems can be investigated and fixed by on-call engineers, but locating the cause of such problems and fixing errors are often time-consuming. Therefore, automatic data validation is a better solution to defend the system and downstream services by enabling early detection of errors and providing detailed error messages for quick resolution. This paper proposes a self-validate data management system with automatic pattern discovery techniques to verify the correctness of semi-structural string data in enterprise data pipelines. Our solution extracts patterns from historical data and detects erroneous incoming data in a top-down fashion. High-level information of historical data is analyzed to discover the format skeleton of correct values. Fine-grained semantic patterns are then extracted to strike a balance between generalization and specification of the discovered pattern, thus covering as many correct values as possible while avoiding over-fitting. To tackle cold start and rapid data growth, we propose an incremental update strategy and example generalization strategy. Experiments on large-scale industrial and public datasets demonstrate the effectiveness and efficiency of our method compared to alternative solutions. Furthermore, a case study on an industrial platform (Ant Group Inc.) with thousands of applications shows that our system captures meaningful data patterns in daily operations and helps engineers quickly identify errors.

Data Validation Based on Dojo

A Field-Oriented Approach To Web Form Validation For Database-Isolated Rule

An Interactive Mode to Aid Web Accessibility Evaluation

ETC-Based Methodology on Data Processing Validation

Research and development on data quality assessment management system

Application of Ajax in Web-based Process Control and Monitoring System

Study on Quality Evaluation in DFQ

A Petri Net-Based Method for Data Validation of Web Services Composition

A Design Tool for Dynamic HTML Form on J2EE Platform

Model Validation Via Uncertainty Propagation and Data Transformations

Data Validation Infrastructure for R

Quality Checking Method of Land Data Based on Rule Engine

Static Validation of WS-CDL Documents.

Automatic String Data Validation with Pattern Discovery

Testing Forms in Web Applications Automatically

A Design Tool for Dynamic WYSWYG Form on J2EE Platform

Dynamic Forms UI: Flexible and Portable Tool for easy UI Design

A Semantic Web Approach to Validating Semistructured Data

A Situation-Aware and Interactive System for Assisting People Fill out Paper Forms

Simulated or Physical? an Empirical Study on Input Validation for Context-aware Systems in Different Environments.

Validation Obligations: A Novel Approach to Check Compliance between Requirements and their Formal Specification