Abstract:Forgetting refers to the loss or deterioration of previously acquired knowledge. While existing surveys on forgetting have primarily focused on continual learning, forgetting is a prevalent phenomenon observed in various other research domains within deep learning. Forgetting manifests in research fields such as generative models due to generator shifts, and federated learning due to heterogeneous data distributions across clients. Addressing forgetting encompasses several challenges, including balancing the retention of old task knowledge with fast learning of new task, managing task interference with conflicting goals, and preventing privacy leakage, etc. Moreover, most existing surveys on continual learning implicitly assume that forgetting is always harmful. In contrast, our survey argues that forgetting is a double-edged sword and can be beneficial and desirable in certain cases, such as privacy-preserving scenarios. By exploring forgetting in a broader context, we present a more nuanced understanding of this phenomenon and highlight its potential advantages. Through this comprehensive survey, we aspire to uncover potential solutions by drawing upon ideas and approaches from various fields that have dealt with forgetting. By examining forgetting beyond its conventional boundaries, we hope to encourage the development of novel strategies for mitigating, harnessing, or even embracing forgetting in real applications. A comprehensive list of papers about forgetting in various research fields is available at \url{<a class="link-external link-https" href="https://github.com/EnnengYang/Awesome-Forgetting-in-Deep-Learning" rel="external noopener nofollow">this https URL</a>}.
What problem does this paper attempt to address?
### What problems does this paper attempt to solve?
This paper aims to explore the extensive impact of the forgetting phenomenon in deep learning, going beyond the traditional Continual Learning (CL) field. Specifically, the paper focuses on the following problems:
1. **Universality of Forgetting**:
- Forgetting occurs not only in continual learning but also is widespread in other research fields such as generative models, federated learning, transfer learning, meta - learning, test - time adaptation, reinforcement learning, and foundation models.
- The forgetting phenomena in these fields are caused by different factors, such as changes in data distribution, changes in task distribution, heterogeneity of client data, etc.
2. **Dual Nature of Forgetting**:
- The existing literature mainly focuses on the negative impacts of forgetting, while this paper emphasizes that forgetting is a double - edged sword, having both harmful and beneficial aspects.
- Harmful Forgetting: When a model needs to retain previously learned knowledge while adapting to new tasks, domains, or environments, forgetting is disadvantageous. In this case, forgetting will damage the performance of the model.
- Beneficial Forgetting: In some cases, forgetting is beneficial, for example:
- Reducing over - fitting: By selectively forgetting irrelevant or noisy information, the model can better generalize to unseen data.
- Eliminating bias and irrelevant information: Removing biased or irrelevant information to improve performance or learn new tasks.
- Machine Unlearning: To protect user privacy, forget specific training data.
3. **Challenges in Dealing with Forgetting**:
- Data Availability: Due to storage limitations or privacy issues, it is impossible to access data from previous tasks, which is especially complex in fields such as continual learning, meta - learning, and transfer learning.
- Resource Constraints: In resource - limited environments, such as those with limited memory and computing power, effectively dealing with forgetting is a challenge.
- Adapting to New Environments/Distributions: The target environment or data distribution changes over time, and the model must adapt to new scenarios, but often forgets previous knowledge when the data distribution changes.
- Task Interference/Inconsistency: Conflicting goals between different tasks will lead to task interference, especially in continual learning and federated learning.
- Privacy Leakage Prevention: Retaining old knowledge may inadvertently expose private information, so attention should be paid to how to forget or delete traces of training data to protect privacy.
4. **Scope of Review and Contributions**:
- The paper provides a comprehensive review, covering the existence and impact of forgetting in various key research fields.
- It proposes a more systematic classification method for continual learning and extends it to other research fields, providing a comprehensive understanding of the forgetting phenomenon.
- It emphasizes the potential benefits of forgetting in privacy - protection scenarios, revealing the nature of forgetting as a double - edged sword.
Through these discussions, the paper hopes to encourage the development of new strategies to deal with, utilize, or even embrace the forgetting phenomenon in order to achieve better practical application effects.