Abstract:Modern natural language processing (NLP) state-of-the-art (SoTA) deep learning (DL) models have hundreds of millions of parameters, making them extremely complex. Large datasets are required for training these models, and while pretraining has reduced this requirement, human-labelled datasets are still necessary for fine-tuning. Few-shot learning (FSL) techniques, such as meta-learning, try to train models from smaller datasets to mitigate this cost. However, the tasks used to evaluate these meta-learners frequently diverge from the problems in the real world that they are meant to resolve. This work aims to apply meta-learning to a problem that is more pertinent to the real world: class incremental learning (IL). In this scenario, after completing its training, the model learns to classify newly introduced classes. One unique quality of meta-learners is that they can generalise from a small sample size to classes that have never been seen before, which makes them especially useful for class incremental learning (IL). The method describes how to emulate class IL using proxy new classes. This method allows a meta-learner to complete the task without the need for retraining. To generate predictions, the transformer-based aggregation function in a meta-learner that modifies data from examples across all classes has been proposed. The principal contributions of the model include concurrently considering the entire support and query sets, and prioritising attention to crucial samples, such as the question, to increase the significance of its impact during inference. The outcomes demonstrate that the model surpasses prevailing benchmarks in the industry. Notably, most meta-learners demonstrate significant generalisation in the context of class IL even without specific training for this task. This paper establishes a high-performing baseline for subsequent transformer-based aggregation techniques, thereby emphasising the practical significance of meta-learners in class IL.

Distilled Meta-learning for Multi-Class Incremental Learning

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Incremental Meta-Learning via Indirect Discriminant Alignment

Class Incremental Learning with Multi-Teacher Distillation

Online Continual Learning Via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation

Class Incremental Learning with Deep Contrastive Learning and Attention Distillation

Class Incremental Learning Via Multi-hinge Distillation

Incremental Meta-Learning Via Episodic Replay Distillation for Few-Shot Image Recognition

Rebalancing Multi-Label Class-Incremental Learning

A Class Incremental Learning Method with Weight Re-Initialization and Multiple Distillation for Image Classification

Knowledge Restore and Transfer for Multi-label Class-Incremental Learning

Multi-granularity knowledge distillation and prototype consistency regularization for class-incremental learning

Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification

Class similarity weighted knowledge distillation for few shot incremental learning

Hyper-feature aggregation and relaxed distillation for class incremental learning

Meta-learning for real-world class incremental learning: a transformer-based approach

Structural Knowledge Organization and Transfer for Class-Incremental Learning.

Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning

Maintaining Discrimination and Fairness in Class Incremental Learning

Multi-Granularity Regularized Re-Balancing for Class Incremental Learning

Learn To Learn More Precisely