Abstract:Catastrophic forgetting is a significant challenge in the field of machine learning, particularly in neural networks. When a neural network learns to perform well on a new task, it often forgets its previously acquired knowledge or experiences. This phenomenon occurs because the network adjusts its weights and connections to minimize the loss on the new task, which can inadvertently overwrite or disrupt the representations that were crucial for the previous tasks. As a result, the the performance of the network on earlier tasks deteriorates, limiting its ability to learn and adapt to a sequence of tasks. In this paper, we propose a novel method for preventing catastrophic forgetting in machine learning applications, specifically focusing on neural networks. Our approach aims to preserve the knowledge of the network across multiple tasks while still allowing it to learn new information effectively. We demonstrate the effectiveness of our method by conducting experiments on various benchmark datasets, including Split MNIST, Split CIFAR10, Split Fashion MNIST, and Split CIFAR100. These datasets are created by dividing the original datasets into separate, non overlapping tasks, simulating a continual learning scenario where the model needs to learn multiple tasks sequentially without forgetting the previous ones. Our proposed method tackles the catastrophic forgetting problem by incorporating negotiated representations into the learning process, which allows the model to maintain a balance between retaining past experiences and adapting to new tasks. By evaluating our method on these challenging datasets, we aim to showcase its potential for addressing catastrophic forgetting and improving the performance of neural networks in continual learning settings.

Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks

Progressive Learning without Forgetting

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model

Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

Disentangling and Mitigating the Impact of Task Similarity for Continual Learning

Understanding Forgetting in Continual Learning with Linear Regression

Measuring Catastrophic Forgetting in Neural Networks

A Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks

Continual Learning and Catastrophic Forgetting

Localizing Catastrophic Forgetting in Neural Networks

Optimal Protocols for Continual Learning via Statistical Physics and Control Theory

Wide Neural Networks Forget Less Catastrophically

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Memory Recall: A Simple Neural Network Training Framework Against Catastrophic Forgetting

Negotiated Representations to Prevent Forgetting in Machine Learning Applications

Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation

Quantum Continual Learning Overcoming Catastrophic Forgetting

Class-Incremental Learning via Knowledge Amalgamation

Catastrophic Importance of Catastrophic Forgetting