Abstract:Continual Learning (CL) plays a crucial role in enhancing learning performance for both new and previous tasks in continuous data streams, thus contributing to the advancement of cognitive computing. However, CL faces a fundamental challenge known as the stability-plasticity quandary. In this research, we present an innovative and effective CL algorithm called Primary Null Space Projection (PNSP) to strike a balance between network plasticity and stability. PNSP consists of three main components. Firstly, it leverages the NSP-LRA algorithm to project the gradient of network parameters from previous tasks into a meticulously designed null space. NSP-LRA harnesses high-dimensional geometric information extracted from the feature covariance matrix through low-rank approximation algorithm to obtain the basis of null space dynamically. This process constructs an innovation null space and ensures the continuous updating of orthonormal bases to accommodate changes in the input data. Secondly, we propose a Consistency-guided Task-specific Feature Learning (CTFL) mechanism to tackle the issue of catastrophic forgetting and facilitate continual learning. CTFL achieves this by aligning feature vectors and maintaining consistent feature learning directions, thereby preventing the loss of previously acquired knowledge. Lastly, we introduce Label Guided Self-Distillation (LGSD), a technique that utilizes true labels to guide the distillation process and incorporates a dynamic temperature mechanism to enhance performance. To evaluate the effectiveness of our proposed method, we conduct experiments on the CIFAR100 and TinyImageNet datasets. The results demonstrate significant improvements over state-of-the-art methods. We have made the implementation code of our approach available for reference.

Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning

Progressive Learning without Forgetting

Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks

Investigating the Impact of Weight Sharing Decisions on Knowledge Transfer in Continual Learning

Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning

Defeating Catastrophic Forgetting via Enhanced Orthogonal Weights Modification

BNS: Building Network Structures Dynamically for Continual Learning

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

Sparse Orthogonal Parameters Tuning for Continual Learning

Forget-free Continual Learning with Soft-Winning SubNetworks

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One

Adaptive Progressive Continual Learning.

Adaptive Orthogonal Projection for Continual Learning

Training Networks in Null Space of Feature Covariance for Continual Learning

Revisiting Neural Networks for Continual Learning: An Architectural Perspective

PNSP: Overcoming Catastrophic Forgetting Using Primary Null Space Projection in Continual Learning

Adaptive online continual multi-view learning

Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning

Similarity-based context aware continual learning for spiking neural networks