Abstract:Continual Learning (CL) plays a crucial role in enhancing learning performance for both new and previous tasks in continuous data streams, thus contributing to the advancement of cognitive computing. However, CL faces a fundamental challenge known as the stability-plasticity quandary. In this research, we present an innovative and effective CL algorithm called Primary Null Space Projection (PNSP) to strike a balance between network plasticity and stability. PNSP consists of three main components. Firstly, it leverages the NSP-LRA algorithm to project the gradient of network parameters from previous tasks into a meticulously designed null space. NSP-LRA harnesses high-dimensional geometric information extracted from the feature covariance matrix through low-rank approximation algorithm to obtain the basis of null space dynamically. This process constructs an innovation null space and ensures the continuous updating of orthonormal bases to accommodate changes in the input data. Secondly, we propose a Consistency-guided Task-specific Feature Learning (CTFL) mechanism to tackle the issue of catastrophic forgetting and facilitate continual learning. CTFL achieves this by aligning feature vectors and maintaining consistent feature learning directions, thereby preventing the loss of previously acquired knowledge. Lastly, we introduce Label Guided Self-Distillation (LGSD), a technique that utilizes true labels to guide the distillation process and incorporates a dynamic temperature mechanism to enhance performance. To evaluate the effectiveness of our proposed method, we conduct experiments on the CIFAR100 and TinyImageNet datasets. The results demonstrate significant improvements over state-of-the-art methods. We have made the implementation code of our approach available for reference.

Latent Spectral Regularization for Continual Learning

Learning Continually by Spectral Regularization

Continual Learning in Human Activity Recognition: an Empirical Analysis of Regularization

On the Effectiveness of Equivariant Regularization for Robust Online Continual Learning

Forget but Recall: Incremental Latent Rectification in Continual Learning

IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning

Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning

Regularization Shortcomings for Continual Learning

EsaCL: Efficient Continual Learning of Sparse Models

Bio-inspired, task-free continual learning through activity regularization

A Closer Look at Rehearsal-Free Continual Learning

Regularizing Second-Order Influences for Continual Learning

TS-ACL: A Time Series Analytic Continual Learning Framework for Privacy-Preserving and Class-Incremental Pattern Recognition

Dissecting Continual Learning a Structural and Data Analysis

PNSP: Overcoming Catastrophic Forgetting Using Primary Null Space Projection in Continual Learning

Calibration of Continual Learning Models

An Attention-based Representation Distillation Baseline for Multi-Label Continual Learning

Continual Learning for LiDAR Semantic Segmentation: Class-Incremental and Coarse-to-Fine strategies on Sparse Data

Regularization-Based Efficient Continual Learning in Deep State-Space Models

Can Continual Learning Improve Long-Tailed Recognition? Toward a Unified Framework