Abstract:Abstract Big Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.

Ontology-Based Data Quality Management for Data Streams

Assessing Data Quality Within Available Context

Data Quality Transaction On Different Distributed Ledger Technologies

DAQUA-MASS: An ISO 8000-61 Based Data Quality Management Methodology for Sensor Data

Big Data Quality: A Survey

Load Shedding in Stream Databases: A Control-Based Approach.

DQSOps: Data Quality Scoring Operations Framework for Data-Driven Applications

Research on Design and Analysis Techniques for Surface Water Quality Monitoring Information Management

Big data quality framework: a holistic approach to continuous quality management

Representing Dataset Quality Metadata using Multi-Dimensional Views

A survey of open-source data quality tools: shedding light on the materialization of data quality dimensions in practice

Data Quality Management Framework for Smart Grid Systems

Data Quality Assessment: Challenges and Opportunities

A Real-Time Semantic Annotation to the Sensor Stream Data for the Water Quality Monitoring

Provenance Management for Data Quality Assessment.

Goal-oriented Data Warehouse Quality Measurement

Definition and Evaluation of Data Quality: User-Oriented Data Object-Driven Approach to Data Quality Assessment

An Exploratory Study of How Specialists Deal with Testing in Data Stream Processing Applications

Beyond Accuracy: What Data Quality Means to Data Consumers

THE DESIGN AND IMPLEMENTATION OF INFORMATION SECURITY-ORIENTED HIGH DATA STREAM MANAGEMENT SYSTEM