Abstract:Real-world data contains various kinds of errors. Before analyzing data, one usually needs to process the raw data. However, traditional data processing based on exactly match often misses lots of valid information. To get high-quality analysis results and fit in the big data era, this thesis studies the error-tolerant big data processing. As most of the data in real world can be represented as a sequence or a set, this thesis utilizes the widely-used sequence-based and set-based similar functions to tolerate errors in data processing and studies the approximate entity extraction, similarity join and similarity search problems. The main contributions of this thesis include: 1. This thesis proposes a unified framework to support approximate entity extraction with both sequence-based and set-based similarity functions simultaneously. The experiments show that the unified framework can improve the state-of-the-art methods by 1 to 2 orders of magnitude. 2. This thesis designs two methods respectively for the sequence and the set similarity joins. For the sequence similarity join, this thesis proposes to evenly partition the sequences to segments. It is guaranteed that two sequences are similar only if one sequence has a subsequence identical to a segment of another sequence. For the set similarity join, this thesis proposes to partition all the sets into segments based on the universe. This thesis further extends the two partition-based methods to support the large-scale data processing framework, Map-Reduce and Spark. The partition-based method won the string similarity join competition held by EDBT and beat the second place by 10 times. 3. This thesis proposes a pivotal prefix filter technique to solve the sequence similarity search problem. This thesis shows that the pivotal prefix filter has stronger pruning power and less filtering cost compared to the state-of-the-art filters.

Ensuring High Data Quality and Error Resilience in Autonomous Self-Schedulable Libraries for Heterogeneous Data Sources in NearReal-Time Ingestion Pipelines

ETC-Based Methodology on Data Processing Validation

A Scalable, Adaptive, Self-management and Fault-Tolerant Architecture for Digital Library

AI-Driven Frameworks for Enhancing Data Quality in Big Data Ecosystems: Error_Detection, Correction, and Metadata Integration

A Theoretical Framework for AI-driven data quality monitoring in high-volume data environments

Enhancing Dependability in Big Data Analytics Enterprise Pipelines

Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines

A Scalable and Robust Framework for Data Stream Ingestion

Big Data Quality Prediction in the Process Industry: A Distributed Parallel Modeling Framework

Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers

Enabling data-centric AI through data quality management and data literacy

Data Quality Over Quantity: Pitfalls and Guidelines for Process Analytics

Exploring Real-Time Data Processing Using Big Data Frameworks

Big Data Assurance: An Approach Based on Service-Level Agreements

Adaptive data quality scoring operations framework using drift-aware mechanism for industrial applications

A review on big data real-time stream processing and its scheduling techniques

Data Quality Improvement Strategies for Data Analysis Specific to the IT Industry

Architecting Data-Intensive Applications : From Data Architecture Design to Its Quality Assurance

A Data Analytics Framework for Aggregate Data Analysis

Error-Tolerant Big Data Processing

Putting Data Science Pipelines on the Edge