Abstract:Abstract Big Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.

ETC-Based Methodology on Data Processing Validation

Assessing Data Quality Within Available Context

Targeting High Quality For Complex Internet Based Information Integration Through Xp Extensions

Research and development on data quality assessment management system

Data Quality Transaction On Different Distributed Ledger Technologies

Automatic String Data Validation with Pattern Discovery

Generic and Automated Data Evaluation in Analytical Measurement

Big Data Quality Prediction in the Process Industry: A Distributed Parallel Modeling Framework

An Automatic Online System for Detecting and Analyzing Quality Data of Products in Manufacturing Process

Quality Assessment for Large-Scale Industrial Software Systems: Experience Report at Alibaba

Data Reconciliation Based on Large Scale Rigorous Model with Equation Oriented Method

A digital twin–driven method for online quality control in process industry

A Validation and Quality Assessment Method with Metamorphic Relations for Unsupervised Machine Learning Software.

A Machine Learning Based Framework for Verification and Validation of Massive Scale Image Data

Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines

Collaboration-Based Automatic Data Validation Framework for Enterprise Asset Management

Novel Collaborative Automated Testing Framework Using Ddf

TODQA: Efficient Task-Oriented Data Quality Assessment

ECS -- an Interactive Tool for Data Quality Assurance

Quality Assurance Technologies of Big Data Applications: A Systematic Literature Review

Big data quality framework: a holistic approach to continuous quality management