Abstract:Abstract Big Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.

Goal-oriented Data Warehouse Quality Measurement

Assessing Data Quality Within Available Context

Quality-based framework for requirement analysis in data warehouse

A Simplified Approach for Quality Management in Data Warehouse

Data warehouse for quality management system

A Methodology for Information Quality Assessment in Data Warehousing

Data Warehouse for Quality Management Systems

Applying MDA to the development of data warehouses

Developing a Data Element Repository to Support EHR-driven Phenotype Algorithm Authoring and Execution

Ontology-Based Data Quality Management for Data Streams

Proposed Quality Evaluation Framework to Incorporate Quality Aspects in Web Warehouse Creation

Modeling Information Quality Risk for Data Mining in Data Warehouses

Managing data quality in an existing medical data warehouse using business intelligence technologies

Data Quality Assessment: Challenges and Opportunities

Big data quality framework: a holistic approach to continuous quality management

Assessing information quality for On-Line Analytical Processing in data warehouses

Towards augmented data quality management: Automation of Data Quality Rule Definition in Data Warehouses

Architecting Data-Intensive Applications : From Data Architecture Design to Its Quality Assurance

Data Warehouse Success and Strategic Oriented Business Intelligence: A Theoretical Framework

GQM+Strategies: A Comprehensive Methodology for Aligning Business Strategies with Software Measurement

Improving Data Quality During ERP Implementation Based on Information Product Map.