Abstract:Meta-learning aims to solve unseen tasks with few labelled instances. Nevertheless, despite its effectiveness for quick learning in existing optimization-based methods, it has several flaws. Inconsequential connections are frequently seen during meta-training, which results in an over-parameterized neural network. Because of this, meta-testing observes unnecessary computations and extra memory overhead. To overcome such flaws. We propose a novel meta-learning method called Meta-LTH that includes indispensible (necessary) connections. We applied the lottery ticket hypothesis technique known as magnitude pruning to generate these crucial connections that can effectively solve few-shot learning problem. We aim to perform two things: (a) to find a sub-network capable of more adaptive meta-learning and (b) to learn new low-level features of unseen tasks and recombine those features with the already learned features during the meta-test phase. Experimental results show that our proposed Met-LTH method outperformed existing first-order MAML algorithm for three different classification datasets. Our method improves the classification accuracy by approximately 2% (20-way 1-shot task setting) for omniglot dataset.

What problem does this paper attempt to address?

The paper aims to address the following key issues: 1. **Optimizing meta-learning algorithms to improve efficiency and reduce resource consumption**: Existing meta-learning methods, especially optimization-based methods (such as MAML), can quickly adapt to new tasks but usually require a large amount of computational resources during training and generate unnecessary computational burdens during the meta-testing phase. This is because these methods often use over-parameterized neural network structures during meta-training. 2. **Achieving lightweight meta-learning models**: To overcome the above issues, the paper proposes a new method called Meta-LTH, which combines meta-learning techniques with network pruning techniques (especially magnitude-based pruning). The goal of this method is to find a sparse subnetwork while ensuring that the model can quickly adapt to new tasks, thereby significantly reducing the model size and computational cost. 3. **Addressing the problem of feature reuse and reorganization**: When encountering new unseen tasks during the meta-testing phase, relying solely on feature reuse may not be sufficient for rapid learning. Therefore, the paper also explores how to learn new low-level features and reorganize these features with the learned features to better adapt to new tasks. Specifically, the main contributions of the paper include: - Proposing a technique based on the Lottery Ticket Hypothesis to identify "indispensable connections" in neural networks and prune unimportant connections. - Designing a simple yet effective meta-learning algorithm, Meta-LTH, which first pre-trains the model using FOMAML, then finds a sparse subnetwork through pruning, and finally retrains this subnetwork to obtain optimized meta-parameters. - Conducting experimental validation on three different classification datasets (Omniglot, MiniImagenet, and FC100), showing that Meta-LTH significantly reduces the model size and computational cost while maintaining or improving classification accuracy compared to the baseline algorithm FOMAML. In summary, by combining meta-learning and network pruning techniques, the paper aims to develop an efficient and lightweight meta-learning method that achieves better performance in learning tasks with a small number of samples.

Learning to Learn with Indispensable Connections

Incremental Meta-Learning via Indirect Discriminant Alignment

Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks

Task Attended Meta-Learning for Few-Shot Learning

Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs

Rethinking Meta-Learning from a Learning Lens

Meta-Learning Requires Meta-Augmentation

MetaConcept: Learn to Abstract Via Concept Graph for Weakly-Supervised Few-Shot Learning

Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Meta-learning for real-world class incremental learning: a transformer-based approach

Perturbing the Gradient for Alleviating Meta Overfitting

Learning Tensor Representations for Meta-Learning

Meta-Learning in Neural Networks: A Survey

Meta-Learning to Improve Pre-Training

A Meta Understanding of Meta-Learning

Deep Meta-Learning: Learning to Learn in the Concept Space

Meta-Learning for Semi-Supervised Few-Shot Classification

Meta-SGD: Learning to Learn Quickly for Few-Shot Learning

Meta-Transfer Learning Through Hard Tasks

Meta-Learning Loss Functions for Deep Neural Networks

Meta-learning Approaches for Few-Shot Learning: A Survey of Recent Advances