Abstract:Continual Learning (CL) is a field dedicated to devise algorithms able to achieve lifelong learning. Overcoming the knowledge disruption of previously acquired concepts, a drawback affecting deep learning models and that goes by the name of catastrophic forgetting, is a hard challenge. Currently, deep learning methods can attain impressive results when the data modeled does not undergo a considerable distributional shift in subsequent learning sessions, but whenever we expose such systems to this incremental setting, performance drop very quickly. Overcoming this limitation is fundamental as it would allow us to build truly intelligent systems showing stability and plasticity. Secondly, it would allow us to overcome the onerous limitation of retraining these architectures from scratch with the new updated data. In this thesis, we tackle the problem from multiple directions. In a first study, we show that in rehearsal-based techniques (systems that use memory buffer), the quantity of data stored in the rehearsal buffer is a more important factor over the quality of the data. Secondly, we propose one of the early works of incremental learning on ViTs architectures, comparing functional, weight and attention regularization approaches and propose effective novel a novel asymmetric loss. At the end we conclude with a study on pretraining and how it affects the performance in Continual Learning, raising some questions about the effective progression of the field. We then conclude with some future directions and closing remarks.

Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering

Progressive Learning without Forgetting

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Continual VQA for Disaster Response Systems

Controlling Forgetting with Test-Time Data in Continual Learning

Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Measuring Catastrophic Forgetting in Neural Networks

Slowing Down Forgetting in Continual Learning

Quantum Continual Learning Overcoming Catastrophic Forgetting

Reviving Dormant Memories: Investigating Catastrophic Forgetting in Language Models through Rationale-Guidance Difficulty

Dissecting Continual Learning a Structural and Data Analysis

Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy

Don't Stop Learning: Towards Continual Learning for the CLIP Model

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Overcoming Catastrophic Forgetting in Tabular Data Classification: A Pseudorehearsal-based approach

Attention-Based Structural-Plasticity

Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering

A Biologically Plausible Audio-Visual Integration Model for Continual Learning

Overcoming catastrophic forgetting in neural networks