Abstract:In continual learning (CL), model growth enhances adaptability over new data, improving knowledge retention for more tasks. However, improper model growth can lead to severe degradation of previously learned knowledge, an issue we name as growth-induced forgetting (GIFt), especially in task-agnostic CL using entire grown model for inference. Existing works, despite adopting model growth and random initialization for better adaptability, often fail to recognize the presence of GIFt caused by improper model growth. This oversight limits comprehensive control of forgetting and hinders full utilization of model growth. We are the first in CL to identify this issue and conduct an in-depth study on root cause of GIFt, where layer expansion stands out among model growth strategies, widening layers without affecting model functionality. Yet, direct adoption of layer expansion presents challenges. It lacks data-driven control and initialization of expanded parameters to balance adaptability and knowledge retention. This paper presents a novel SparseGrow approach to overcome the issue of GIFt while enhancing adaptability over new data. SparseGrow employs data-driven sparse layer expansion to control efficient parameter usage during growth, reducing GIFt from excessive growth and functionality changes. It also combines sparse growth with on-data initialization at training late-stage to create partially 0-valued expansions that fit learned distribution, enhancing retention and adaptability. To further minimize forgetting, freezing is applied by calculating the sparse mask, allowing data-driven preservation of important parameters. Through experiments across datasets with various settings, cases, and task numbers, we demonstrate the necessity of layer expansion and showcase the effectiveness of SparseGrow in overcoming GIFt, highlighting its adaptability and knowledge retention for incremental tasks.

Overcoming Catastrophic Forgetting in Continual Learning by Exploring Eigenvalues of Hessian Matrix.

Progressive Learning without Forgetting

Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint

Understanding Forgetting in Continual Learning with Linear Regression

Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

Overcoming Catastrophic Forgetting with Classifier Expander.

Continual Learning in Human Activity Recognition: an Empirical Analysis of Regularization

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model

Defying Catastrophic Forgetting via Influence Function

Slowing Down Forgetting in Continual Learning

Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modification

A Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks

Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping

Weight Friction: A Simple Method to Overcome Catastrophic Forgetting and Enable Continual Learning

Measuring Catastrophic Forgetting in Neural Networks

Overcoming Growth-Induced Forgetting in Task-Agnostic Continual Learning

Regularization Shortcomings for Continual Learning

Elephant Neural Networks: Born to Be a Continual Learner

Quantum Continual Learning Overcoming Catastrophic Forgetting

The Ideal Continual Learner: An Agent That Never Forgets