Abstract:Endoscopy is a widely used technique for the early detection of diseases or robotic-assisted minimally invasive surgery (RMIS). Numerous deep learning (DL)-based research works have been developed for automated diagnosis or processing of endoscopic view. However, existing DL models may suffer from catastrophic forgetting. When new target classes are introduced over time or cross institutions, the performance of old classes may suffer severe degradation. More seriously, data privacy and storage issues may lead to the unavailability of old data when updating the model. Therefore, it is necessary to develop a continual learning (CL) methodology to solve the problem of catastrophic forgetting in endoscopic image segmentation. To tackle this, we propose a Endoscopy Continual Semantic Segmentation (EndoCSS) framework that does not involve the storage and privacy issues of exemplar data. The framework includes a mini-batch pseudo-replay (MB-PR) mechanism and a self-adaptive noisy cross-entropy (SAN-CE) loss. The MB-PR strategy circumvents privacy and storage issues by generating pseudo-replay images through a generative model. Meanwhile, the MB-PR strategy can also correct the model deviation to the replay data and current training data, which is aroused by the significant difference in the amount of current and replay images. Therefore, the model can perform effective representation learning on both new and old tasks. SAN-CE loss can help model fitting by adjusting the model's output logits, and also improve the robustness of training. Extensive continual semantic segmentation (CSS) experiments on public datasets demonstrate that our method can robustly and effectively address the catastrophic forgetting brought by class increment in endoscopy scenes. The results show that our framework holds excellent potential for real-world deployment in a streaming learning manner.

Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting

DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation

Towards Synchronous Memorizability and Generalizability with Site-Modulated Diffusion Replay for Cross-Site Continual Segmentation

DDGR: Continual Learning with Deep Diffusion-based Generative Replay.

Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay

SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection

Autonomous Generative Feature Replay for Non-Exemplar Class-Incremental Learning

Continual Learning with Deep Generative Replay

Generative appearance replay for continual unsupervised domain adaptation

Learning Towards Synchronous Network Memorizability and Generalizability for Continual Segmentation Across Multiple Sites

Adaptive Prototype Replay for Class Incremental Semantic Segmentation

Memory Enhanced Replay for Continual Learning

Memory-Free Generative Replay For Class-Incremental Learning

RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation

Saliency-Guided Hidden Associative Replay for Continual Learning

Adaptive Visual Scene Understanding: Incremental Scene Graph Generation

Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning

Memory Replay GANs: Learning to Generate Images from New Categories Without Forgetting

Generative Feature Replay For Class-Incremental Learning

M EMORY R EPLAY WITH D ATA C OMPRESSION FOR C ONTINUAL L EARNING