Abstract:To cleanse mislabeled examples from a training dataset for efficient and effective induction, most existing approaches adopt a major set oriented scheme: the training dataset is separated into two parts (a major set and a minor set). The classifiers learned from the major set are used to identify noise in the minor set. The obvious drawbacks of such a scheme are twofold: (1) when the underlying data volume keeps growing, it would be either physically impossible or time consuming to load the major set into the memory for inductive learning; and (2) for multiple or distributed datasets, it can be either technically infeasible or factitiously forbidden to download data from other sites (for security or privacy reasons). Therefore, these approaches have severe limitations in conducting effective global data cleansing from large, distributed datasets.In this paper, we propose a solution to bridge the local and global analysis for noise cleansing. More specifically, the proposed effort tries to identify and eliminate mislabeled data items from large or distributed datasets through local analysis and global incorporation. For this purpose, we make use of distributed datasets or partition a large dataset into subsets, each of which is regarded as a local subset and is small enough to be processed by an induction algorithm at one time to construct a local model for noise identification. We construct good rules from each subset, and use the good rules to evaluate the whole dataset. For a given instance I-k , two error count variables are used to count the number of times it has been identified as noise by all data subsets. The instance with higher error values will have a higher probability of being a mislabeled example. Two threshold schemes, majority and non-objection, are used to identify and eliminate the noisy examples. Experimental results and comparative studies on both real-world and synthetic datasets are reported to evaluate the effectiveness and efficiency of the proposed approach.

Class Noise Vs Attribute Noise: Their Impacts, Detection And Cleansing

Class Noise Vs. Attribute Noise: A Quantitative Study

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Dealing with predictive-but-unpredictable attributes in noisy data sources

Knowledge Discovery from Noisy Datasets

Mining with Noise Knowledge: Error-Aware Data Mining

Error-Tolerant Data Mining.

A Method for Eliminating Class Noise in Text Classification Based on Feature Class Attribute

A review of addressing class noise problems of remote sensing classification

A boosting method to detect noisy data

A Survey on Classifying Big Data with Label Noise

Bridging Local and Global Data Cleansing: Identifying Class Noise in Large, Distributed Data Datasets

Which noise affects algorithm robustness for learning to rank

A Novel Class Noise Estimation Method and Application in Classification.

The Classification of Noise-Afflicted Remotely Sensed Data Using Three Machine-Learning Techniques: Effect of Different Levels and Types of Noise on Accuracy

Noisy Label Processing for Classification: A Survey

Eliminating Class Noise in Large Datasets.

A study of the effect of different types of noise on the precision of supervised learning techniques

Impacts of Dirty Data: and Experimental Evaluation

Integrating Induction and Deduction for Noisy Data Mining

What Makes Data Robust