Abstract:Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. However, these methods lack a unified framework and common terminology for describing their approaches. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies. Notably, this new framework is capable of encompassing established CL approaches as special instances within a unified and general optimization objective. An intriguing finding is that despite their diverse origins, these methods share common mathematical structures. This observation highlights the compatibility of these seemingly distinct techniques, revealing their interconnectedness through a shared underlying optimization objective. Moreover, the proposed general framework introduces an innovative concept called refresh learning, specifically designed to enhance the CL performance. This novel approach draws inspiration from neuroscience, where the human brain often sheds outdated information to improve the retention of crucial knowledge and facilitate the acquisition of new information. In essence, refresh learning operates by initially unlearning current data and subsequently relearning it. It serves as a versatile plug-in that seamlessly integrates with existing CL methods, offering an adaptable and effective enhancement to the learning process. Extensive experiments on CL benchmarks and theoretical analysis demonstrate the effectiveness of the proposed refresh learning. Code is available at \url{

Make Continual Learning Stronger via C-Flat

UniGrad-FS: Unified Gradient Projection with Flatter Sharpness for Continual Learning

Progressive Learning without Forgetting

Create and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning

Data Augmented Flatness-aware Gradient Projection for Continual Learning

Efficient Loss Landscape Reshaping for Convolutional Neural Networks

Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes

FAM: Relative Flatness Aware Minimization

Improving Data-aware and Parameter-aware Robustness for Continual Learning

Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning

A Unified and General Framework for Continual Learning

Meta Continual Learning Revisited: Implicitly Enhancing Online Hessian Approximation Via Variance Reduction.

Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization

Dynamic Consolidation for Continual Learning

An Effective Dynamic Gradient Calibration Method for Continual Learning

Regularizing Scale-Adaptive Central Moment Sharpness for Neural Networks

Density Distribution-based Learning Framework for Addressing Online Continual Learning Challenges

Generalized Federated Learning via Gradient Norm-Aware Minimization and Control Variables

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

Class-Conditional Sharpness-Aware Minimization for Deep Long-Tailed Recognition

Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regularization Favoring Flat Solutions