Abstract:The ability to learn sequentially from different data sites is crucial for a deep network in solving practical medical image diagnosis problems due to privacy restrictions and storage limitations. However, adapting on incoming site leads to catastrophic forgetting on past sites and decreases generalizablity on unseen sites. Existing Continual Learning (CL) and Domain Generalization (DG) methods have been proposed to solve these two challenges respectively, but none of them can address both simultaneously. Recognizing this limitation, this paper proposes a novel training paradigm, learning towards Synchronous Memorizability and Generalizability (SMG-Learning). To achieve this, we create the orientational gradient alignment to ensure memorizability on previous sites, and arbitrary gradient alignment to enhance generalizability on unseen sites. This approach is named as Parallel Gradient Alignment (PGA). Furthermore, we approximate the PGA as dual meta-objectives using the first-order Taylor expansion to reduce computational cost of aligning gradients. Considering that performing gradient alignments, especially for previous sites, is not feasible due to the privacy constraints, we design a Site-Modulated Diffusion (SMD) model to generate images with site-specific learnable prompts, replaying images have similar data distributions as previous sites. We evaluate our method on two medical image segmentation tasks, where data from different sites arrive sequentially. Experimental results show that our method efficiently enhances both memorizability and generalizablity better than other state-of-the-art methods, delivering satisfactory performance across all sites. Our code will be available at: <a class="link-external link-https" href="https://github.com/dyxu-cuhkcse/SMG-Learning" rel="external noopener nofollow">this https URL</a>.

Gradient-based Editing of Memory Examples for Online Task-free Continual Learning

Progressive Learning without Forgetting

Adaptive Memory Replay for Continual Learning

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Fine-Grained Gradient Restriction: A Simple Approach for Mitigating Catastrophic Forgetting

Online Continual Learning with Declarative Memory

Slowing Down Forgetting in Continual Learning

Elastic Multi-Gradient Descent for Parallel Continual Learning

MGSER-SAM: Memory-Guided Soft Experience Replay with Sharpness-Aware Optimization for Enhanced Continual Learning

Towards Synchronous Memorizability and Generalizability with Site-Modulated Diffusion Replay for Cross-Site Continual Segmentation

Graph Continual Learning with Debiased Lossless Memory Replay

Memory Enhanced Replay for Continual Learning

CORE: Mitigating Catastrophic Forgetting in Continual Learning through Cognitive Replay

Gradient Episodic Memory for Continual Learning

Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning

Adaptive online continual multi-view learning

Controlling Forgetting with Test-Time Data in Continual Learning

Principal Gradient Direction and Confidence Reservoir Sampling for Continual Learning

AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning

Squeezing More Past Knowledge for Online Class-Incremental Continual Learning

Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting