Abstract:In numerous academic and practical settings, it is common to progressively add data features to existing models as data evolves due to new discoveries and ongoing data collection efforts. Incremental attribute learning focuses on incorporating these new feature dimensions efficiently without retraining the entire model. The key challenge for incremental attribute learning is adapting models to include new attributes dynamically while maintaining their performance on existing data formats. This research leverages strategies from existing incremental learning practices, applying them to incremental attribute learning through a new framework that utilizes knowledge distillation to prevent catastrophic forgetting. Additionally, this study proposes an embedding layer algorithm to manage the variability of input features in incremental learning effectively. Extensive experiments across 11 tabular datasets, covering binary classification, multi-class classification, and regression tasks, indicate that our approach surpasses traditional incremental attribute learning methods and conventional retraining in terms of accuracy and efficiency. Compared to existing incremental attribute learning models, our model requires only a fraction of the training time and achieves a relative improvement of 7.97% in classification tasks. Furthermore, our approach exhibits minimal forgetfulness of old format data, with a forgetting rate of approximately 1% per increment stage, while maintaining strong performance on new format data. Despite its advantages, our method is limited to tabular data and requires additional storage space. We encourage future research to delve deeper into incremental attribute learning and its broader applications, aiming to consistently enhance model performance as new attributes are introduced. Additionally, we suggest exploring methods for refining and selecting soft labels or intermediate vectors to optimize storage efficiency while effectively leveraging the original model's knowledge.

Hyper-feature aggregation and relaxed distillation for class incremental learning

Class Incremental Learning Via Multi-hinge Distillation

Maintaining Discrimination and Fairness in Class Incremental Learning

Hybrid Rotation Self-Supervision and Feature Space Normalization for Class Incremental Learning

Incremental Scene Classification Using Dual Knowledge Distillation and Classifier Discrepancy on Natural and Remote Sensing Images

Incremental Attribute Learning by Knowledge distillation method

Knowledge Aggregation Networks for Class Incremental Learning.

Class incremental learning of remote sensing images based on class similarity distillation

Class-Incremental Learning for Semantic Segmentation in Aerial Imagery Via Distillation in All Aspects

Multi-granularity knowledge distillation and prototype consistency regularization for class-incremental learning

Model Behavior Preserving for Class-Incremental Learning

Class Incremental Learning with Deep Contrastive Learning and Attention Distillation

Class-Incremental Semantic Segmentation of Aerial Images Via Pixel-Level Feature Generation and Task-Wise Distillation

Feature Expansion and enhanced Compression for Class Incremental Learning

Large Scale Incremental Learning

Class similarity weighted knowledge distillation for few shot incremental learning

Few-shot Class Incremental Learning Via Prompt Transfer and Knowledge Distillation

A Class Incremental Learning Method with Weight Re-Initialization and Multiple Distillation for Image Classification

Distillation-Constrained Prototype Representation Network for Hyperspectral Image Incremental Classification

Adaptive knowledge transfer for class incremental learning

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning