Mohamed Elsayed,A. Rupam Mahmood
Abstract:Deep representation learning methods struggle with continual learning, suffering from both catastrophic forgetting of useful units and loss of plasticity, often due to rigid and unuseful units. While many methods address these two issues separately, only a few currently deal with both simultaneously. In this paper, we introduce Utility-based Perturbed Gradient Descent (UPGD) as a novel approach for the continual learning of representations. UPGD combines gradient updates with perturbations, where it applies smaller modifications to more useful units, protecting them from forgetting, and larger modifications to less useful units, rejuvenating their plasticity. We use a challenging streaming learning setup where continual learning problems have hundreds of non-stationarities and unknown task boundaries. We show that many existing methods suffer from at least one of the issues, predominantly manifested by their decreasing accuracy over tasks. On the other hand, UPGD continues to improve performance and surpasses or is competitive with all methods in all problems. Finally, in extended reinforcement learning experiments with PPO, we show that while Adam exhibits a performance drop after initial learning, UPGD avoids it by addressing both continual learning issues.
What problem does this paper attempt to address?
The problems that this paper attempts to solve are the problems of **catastrophic forgetting and loss of plasticity in continual learning**. Specifically:
1. **Catastrophic Forgetting**: In continual learning, neural network models often forget the knowledge they have previously learned when learning new tasks. This phenomenon will lead to a decline in the performance of the model when dealing with old tasks, and it may even be completely unable to recall the previously learned knowledge.
2. **Loss of Plasticity**: As more tasks are learned, the learning ability of the model gradually weakens, and it becomes difficult to adapt to new tasks or data. This makes the model less flexible when facing new challenges.
To solve these two problems, the author proposes a new method - **Utility - based Perturbed Gradient Descent (UPGD)**. UPGD protects useful units by combining gradient updates and perturbations, and at the same time applies greater modifications to less useful units, thereby maintaining their plasticity. This method aims to overcome the limitations of existing methods in dealing with non - stationary streaming learning environments, and does not require additional conditions such as prior knowledge of task boundaries, replay buffers, or pre - training.
### Specific Problem Description
- **Catastrophic Forgetting**: Many existing continual learning methods will gradually forget the previously learned knowledge when facing multiple consecutive tasks, resulting in a decline in performance.
- **Loss of Plasticity**: As time goes by, the learning ability of the model gradually weakens, and it becomes difficult to adapt to new tasks or data.
### Proposed Solution
The author proposes the UPGD method, and its main features include:
- **Utility Evaluation**: The importance of weights is measured by defining their utility. Weights with higher utility are considered more important for model performance and should therefore be protected.
- **Perturbation Mechanism**: Greater perturbations are applied to less important weights to restore their plasticity.
- **Streaming Learning Environment**: It is suitable for non - stationary streaming learning environments, that is, the data is non - independently and identically distributed, and the task boundaries are unknown.
In this way, UPGD can effectively deal with the problems of catastrophic forgetting and loss of plasticity in the process of continual learning, thereby improving the overall performance of the model in a multi - task environment.
### Experimental Verification
The author conducted experiments on multiple benchmark datasets, including MNIST, EMNIST, CIFAR - 10, and ImageNet, and used different types of models such as multi - layer perceptrons, convolutional neural networks, and residual neural networks for testing. The experimental results show that UPGD performs excellently in these tasks, can effectively alleviate the problems of catastrophic forgetting and loss of plasticity, and is significantly better than other existing methods.
In addition, the author also verified the effectiveness of UPGD in extended reinforcement learning experiments, and the results show that UPGD has better stability and performance when dealing with continuous learning problems.
In summary, the main contribution of this paper is to propose a novel method UPGD, which can effectively solve the two major problems in continual learning - catastrophic forgetting and loss of plasticity - in complex streaming learning environments.