Abstract:Deep neural networks (DNNs) have gained great success in information fusion. However, recent studies report that DNNs are suffering from catastrophic forgetting, i.e., DNNs would forget the knowledge learned from previous tasks when training on the current task. To address this issue, continual learning is proposed to enhance long-term memories for DNNs. Since continual learning is very challenging, existing work simplifies the setting to simulate the sequentially online multi-task learning paradigm. Specifically, existing works commonly split one dataset into multiple disjoint categories to get multiple tasks that follow the same marginal distribution. We argue that this setting is too simple to approximate the real-world applications. In real-world scenarios, the data distributions of sequentially arrived tasks would change significantly from time to time, e.g., the lighting from day to night, and the background from spring to winter. Thus, the real-world applications are in a multi-view manner, yet existing methods ignore this challenge. To tame this, we propose Adaptive Online Continual Multi-view Learning (AOCML) to align distributions and reduce catastrophic forgetting as new tasks arrive. AOCML integrates experience replay and adversarial learning in an end-to-end framework, which stores samples in a memory buffer to replay previous tasks, while leveraging a discriminator to adaptively align distributions across views on-the-fly. In addition to common replay buffer, we also incorporate a soft label-based replay and an entropy-based reweighting to further prevent forgetting. Extensive experiments on four datasets verify that our method is able to significantly outperform previous CL methods and our method pushes CL one step forward towards practical multi-view orientation.

SOLA: Continual Learning with Second-Order Loss Approximation

Progressive Learning without Forgetting

Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint

Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation

Efficient Meta-Learning for Continual Learning with Taylor Expansion Approximation

Orthogonal Gradient Descent for Continual Learning

Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting

On Sequential Loss Approximation for Continual Learning

Memory Efficient Data-Free Distillation for Continual Learning.

Overcoming Catastrophic Forgetting for Continual Learning Via Model Adaptation

Adaptive Progressive Continual Learning.

On the Convergence of Continual Learning with Adaptive Methods

Challenging Common Assumptions about Catastrophic Forgetting

Reinforced Continual Learning

Continual Learning by Modeling Intra-Class Variation

Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

Deep Generative Dual Memory Network for Continual Learning

Learning to Continually Learn Rapidly from Few and Noisy Data

Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations

Layerwise Optimization by Gradient Decomposition for Continual Learning

Adaptive online continual multi-view learning