Abstract:Modern natural language processing (NLP) state-of-the-art (SoTA) deep learning (DL) models have hundreds of millions of parameters, making them extremely complex. Large datasets are required for training these models, and while pretraining has reduced this requirement, human-labelled datasets are still necessary for fine-tuning. Few-shot learning (FSL) techniques, such as meta-learning, try to train models from smaller datasets to mitigate this cost. However, the tasks used to evaluate these meta-learners frequently diverge from the problems in the real world that they are meant to resolve. This work aims to apply meta-learning to a problem that is more pertinent to the real world: class incremental learning (IL). In this scenario, after completing its training, the model learns to classify newly introduced classes. One unique quality of meta-learners is that they can generalise from a small sample size to classes that have never been seen before, which makes them especially useful for class incremental learning (IL). The method describes how to emulate class IL using proxy new classes. This method allows a meta-learner to complete the task without the need for retraining. To generate predictions, the transformer-based aggregation function in a meta-learner that modifies data from examples across all classes has been proposed. The principal contributions of the model include concurrently considering the entire support and query sets, and prioritising attention to crucial samples, such as the question, to increase the significance of its impact during inference. The outcomes demonstrate that the model surpasses prevailing benchmarks in the industry. Notably, most meta-learners demonstrate significant generalisation in the context of class IL even without specific training for this task. This paper establishes a high-performing baseline for subsequent transformer-based aggregation techniques, thereby emphasising the practical significance of meta-learners in class IL.

Bootstrapped Meta-Learning

Unleash Model Potential: Bootstrapped Meta Self-supervised Learning

Making Scalable Meta Learning Practical

Optimistic Meta-Gradients

Meta-Learning Requires Meta-Augmentation

Probabilistic Active Meta-Learning

Unsupervised Meta-Learning for Reinforcement Learning

Understanding Transfer Learning and Gradient-Based Meta-Learning Techniques

Reptile: a Scalable Metalearning Algorithm

Adaptive Gradient-Based Meta-Learning Methods

EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Curriculum in Gradient-Based Meta-Reinforcement Learning

GreedyAgent: Crafting Efficient Agents for Meta-learning from Learning Curves Via Greedy Algorithm Selection

Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practice

Meta-learning for real-world class incremental learning: a transformer-based approach

Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments

Meta-Learning to Improve Pre-Training

Metacontrol for Adaptive Imagination-Based Optimization

Meta weight learning via model-agnostic meta-learning

Model-Agnostic Learning to Meta-Learn