Exploring the potential of prototype-based soft-labels data distillation for imbalanced data classification

Radu-Andrei Rosu,Mihaela-Elena Breaban,Henri Luchian

2024-03-26

Abstract:Dataset distillation aims at synthesizing a dataset by a small number of artificially generated data items, which, when used as training data, reproduce or approximate a machine learning (ML) model as if it were trained on the entire original dataset. Consequently, data distillation methods are usually tied to a specific ML algorithm. While recent literature deals mainly with distillation of large collections of images in the context of neural network models, tabular data distillation is much less represented and mainly focused on a theoretical perspective. The current paper explores the potential of a simple distillation technique previously proposed in the context of Less-than-one shot learning. The main goal is to push further the performance of prototype-based soft-labels distillation in terms of classification accuracy, by integrating optimization steps in the distillation process. The analysis is performed on real-world data sets with various degrees of imbalance. Experimental studies trace the capability of the method to distill the data, but also the opportunity to act as an augmentation method, i.e. to generate new data that is able to increase model accuracy when used in conjunction with - as opposed to instead of - the original data.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the application of data distillation in imbalanced data classification and explore how to improve classification performance by optimizing soft-labels and prototype generation.

Specifically:
1. **Impact of Imbalanced Data**: Investigate how imbalanced data affects the quality of distilled data and propose mitigation measures.
2. **Algorithm Generality**: Explore whether distilled data is only applicable to specific algorithms (such as k-nearest neighbors classifier) or can be applied to other scenarios.
3. **Data Augmentation Techniques**: Investigate the possibility of transforming the original purpose (i.e., reducing dataset size while maintaining classification performance) into data augmentation techniques to further enhance classification performance.

The paper mainly achieves these goals by introducing two improved methods:
- **Soft-Label Optimization**: Gradually improve the quality of soft-labels through iterative training of initial prototype lines.
- **Prototype Generation Based on Boosting Methods**: Use an ensemble of multiple prototype instances to better cover the feature space, ultimately for data augmentation.

The experimental section verifies the effectiveness of these methods, especially in handling imbalanced datasets.