Abstract:To cleanse mislabeled examples from a training dataset for efficient and effective induction, most existing approaches adopt a major set oriented scheme: the training dataset is separated into two parts (a major set and a minor set). The classifiers learned from the major set are used to identify noise in the minor set. The obvious drawbacks of such a scheme are twofold: (1) when the underlying data volume keeps growing, it would be either physically impossible or time consuming to load the major set into the memory for inductive learning; and (2) for multiple or distributed datasets, it can be either technically infeasible or factitiously forbidden to download data from other sites (for security or privacy reasons). Therefore, these approaches have severe limitations in conducting effective global data cleansing from large, distributed datasets.In this paper, we propose a solution to bridge the local and global analysis for noise cleansing. More specifically, the proposed effort tries to identify and eliminate mislabeled data items from large or distributed datasets through local analysis and global incorporation. For this purpose, we make use of distributed datasets or partition a large dataset into subsets, each of which is regarded as a local subset and is small enough to be processed by an induction algorithm at one time to construct a local model for noise identification. We construct good rules from each subset, and use the good rules to evaluate the whole dataset. For a given instance I-k , two error count variables are used to count the number of times it has been identified as noise by all data subsets. The instance with higher error values will have a higher probability of being a mislabeled example. Two threshold schemes, majority and non-objection, are used to identify and eliminate the noisy examples. Experimental results and comparative studies on both real-world and synthetic datasets are reported to evaluate the effectiveness and efficiency of the proposed approach.

Discretization for Inductive Learning from Continuous Sample Data

Discretization of Continuous-Valued Attributes and Instance-Based Learning

An ICA-Based multivariate discretization algorithm

Dynamic Discretization: A Combination Approach

A New Discretization Approach of Continuous Attributes

A Tuning Aid for Discretization in Rule Induction

A New Supervised Discretization Method Based on Bayes Measure

A New Discretization Algorithm for Continuous Interval-valued Attributes

Discretisation in Lazy Learning Algorithms

A Bayesian Discretizer for Real-Valued Attributes.

A Survey of Discretization Techniques: Taxonomy and Empirical Analysis in Supervised Learning

Fuzzy Interpretation of Induction Results.

A UMDA-Based Discretization Method for Continuous Attributes

Continuous Invariance Learning

Discretization Invariant Networks for Learning Maps between Neural Fields

Method of Discretization of Continuous Attributes of Decision Table

Designing Discontinuities

The Unsupervised Discretization Method of Continuous Attributes Study: Based on Normal Distribution Characteristics

CAUSAL INFERENCE FOR CONTINUOUS-TIME PROCESSES WHEN COVARIATES ARE OBSERVED ONLY AT DISCRETE TIMES

Scalable Inductive Learning on Partitioned Data.

Bridging Local and Global Data Cleansing: Identifying Class Noise in Large, Distributed Data Datasets