Abstract:Learning from a sequence of tasks for a lifetime is essential for an agent toward artificial general intelligence. Despite the explosion of this research field in recent years, most work focuses on the well-known catastrophic forgetting issue. In contrast, this work aims to explore knowledge-transferable lifelong learning without storing historical data and significant additional computational overhead. We demonstrate that existing data-free frameworks, including regularization-based single-network and structure-based multinetwork frameworks, face a fundamental issue of lifelong learning, named anterograde forgetting, i.e., preserving and transferring memory may inhibit the learning of new knowledge. We attribute it to the fact that the learning network capacity decreases while memorizing historical knowledge and conceptual confusion between the irrelevant old knowledge and the current task. Inspired by the complementary learning theory in neuroscience, we endow artificial neural networks with the ability to continuously learn without forgetting while recalling historical knowledge to facilitate learning new knowledge. Specifically, this work proposes a general framework named cycle memory networks (CMNs). The CMN consists of two individual memory networks to store short-and long-term memories separately to avoid capacity shrinkage and a transfer cell between them. It enables knowledge transfer from the long-term to the short-term memory network to mitigate conceptual confusion. In addition, the memory consolidation mechanism integrates short-term knowledge into the long-term memory network for knowledge accumulation. We demonstrate that the CMN can effectively address the anterograde forgetting on several task-related, task-conflict, class-incremental, and cross-domain benchmarks. Furthermore, we provide extensive ablation studies to verify each framework component. The source codes are available at: https://github.com/GeoX-Lab/CMN.

Better Memorization, Better Recall: A Lifelong Learning Framework for Remote Sensing Image Scene Classification

Asymmetric collaborative network: transferable lifelong learning for remote sensing images

A lightweight and stochastic depth residual attention network for remote sensing scene classification

A Multiscale Incremental Learning Network for Remote Sensing Scene Classification

Continual Learning for Remote Sensing Image Scene Classification With Prompt Learning

Remote Sensing Scene Classification Using Sparse Representation-Based Framework with Deep Feature Fusion

Deep Metric Learning Based on Scalable Neighborhood Components for Remote Sensing Scene Characterization

Superpixel-Based Long-Range Dependent Network for High-Resolution Remote-Sensing Image Classification

Lifelong Learning for Scene Recognition in Remote Sensing Images

Incremental Learning With Open-Set Recognition for Remote Sensing Image Scene Classification

Self-Training and Curriculum Learning Guided Dynamic Refined Network for Remote Sensing Class-Incremental Semantic Segmentation

A novel self-taught learning framework using spatial pyramid matching for scene classification

Transferring CNN With Adaptive Learning for Remote Sensing Scene Classification

MiSSNet: Memory-Inspired Semantic Segmentation Augmentation Network for Class-Incremental Learning in Remote Sensing Images

A Lightweight and Multiscale Network for Remote Sensing Image Scene Classification

CLRS: Continual Learning Benchmark for Remote Sensing Image Scene Classification

Lifelong Learning With Cycle Memory Networks

Instance-Level Scaling and Dynamic Margin-Alignment Knowledge Distillation for Remote Sensing Image Scene Classification

Evolutionary Multitasking Collaborative Neural Architecture Search for Scene Classification.

RSSM-Net: Remote Sensing Image Scene Classification Based on Multi-Objective Neural Architecture Search

Remote Sensing Image Classification using Transfer Learning and Attention Based Deep Neural Network