Abstract:Continual Learning (CL) plays a crucial role in enhancing learning performance for both new and previous tasks in continuous data streams, thus contributing to the advancement of cognitive computing. However, CL faces a fundamental challenge known as the stability-plasticity quandary. In this research, we present an innovative and effective CL algorithm called Primary Null Space Projection (PNSP) to strike a balance between network plasticity and stability. PNSP consists of three main components. Firstly, it leverages the NSP-LRA algorithm to project the gradient of network parameters from previous tasks into a meticulously designed null space. NSP-LRA harnesses high-dimensional geometric information extracted from the feature covariance matrix through low-rank approximation algorithm to obtain the basis of null space dynamically. This process constructs an innovation null space and ensures the continuous updating of orthonormal bases to accommodate changes in the input data. Secondly, we propose a Consistency-guided Task-specific Feature Learning (CTFL) mechanism to tackle the issue of catastrophic forgetting and facilitate continual learning. CTFL achieves this by aligning feature vectors and maintaining consistent feature learning directions, thereby preventing the loss of previously acquired knowledge. Lastly, we introduce Label Guided Self-Distillation (LGSD), a technique that utilizes true labels to guide the distillation process and incorporates a dynamic temperature mechanism to enhance performance. To evaluate the effectiveness of our proposed method, we conduct experiments on the CIFAR100 and TinyImageNet datasets. The results demonstrate significant improvements over state-of-the-art methods. We have made the implementation code of our approach available for reference.

Data Augmented Flatness-aware Gradient Projection for Continual Learning

UniGrad-FS: Unified Gradient Projection with Flatter Sharpness for Continual Learning

Progressive Learning without Forgetting

Class Gradient Projection for Continual Learning

Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning

Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning

Introducing Common Null Space of Gradients for Gradient Projection Methods in Continual Learning

Make Continual Learning Stronger via C-Flat

Restricted Orthogonal Gradient Projection for Continual Learning

CODE-CL: COnceptor-Based Gradient Projection for DEep Continual Learning

Iterative Relaxing Gradient Projection for Continual Learning

TRGP: Trust Region Gradient Projection for Continual Learning

TARGET: Federated Class-Continual Learning Via Exemplar-Free Distillation

Measuring Asymmetric Gradient Discrepancy in Parallel Continual Learning

PNSP: Overcoming Catastrophic Forgetting Using Primary Null Space Projection in Continual Learning

Orthogonal Gradient Descent for Continual Learning

Rethinking Gradient Projection Continual Learning: Stability / Plasticity Feature Space Decoupling

An Effective Dynamic Gradient Calibration Method for Continual Learning

Learning to Predict Gradients for Semi-Supervised Continual Learning

Elastic Multi-Gradient Descent for Parallel Continual Learning

Gradient Regularized Contrastive Learning for Continual Domain Adaptation