Abstract:Multi-class supervised learning systems require the knowledge of the entire range of labels they predict. Often when learnt incrementally, they suffer from catastrophic forgetting. To avoid this, generous leeways have to be made to the philosophy of incremental learning that either forces a part of the machine to not learn, or to retrain the machine again with a selection of the historic data. While these hacks work to various degrees, they do not adhere to the spirit of incremental learning. In this article, we redefine incremental learning with stringent conditions that do not allow for any undesirable relaxations and assumptions. We design a strategy involving generative models and the distillation of dark knowledge as a means of hallucinating data along with appropriate targets from past distributions. We call this technique, phantom <a class="link-external link-http" href="http://sampling.We" rel="external noopener nofollow">this http URL</a> show that phantom sampling helps avoid catastrophic forgetting during incremental learning. Using an implementation based on deep neural networks, we demonstrate that phantom sampling dramatically avoids catastrophic forgetting. We apply these strategies to competitive multi-class incremental learning of deep neural networks. Using various benchmark datasets and through our strategy, we demonstrate that strict incremental learning could be achieved. We further put our strategy to test on challenging cases, including cross-domain increments and incrementing on a novel label space. We also propose a trivial extension to unbounded-continual learning and identify potential for future development.

Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence

Learning with Recoverable Forgetting

A Strategy for an Uncompromising Incremental Learner

An Efficient Strategy for Catastrophic Forgetting Reduction in Incremental Learning

Class Incremental Learning with Less Forgetting Direction and Equilibrium Point

On the Exploration of Incremental Learning for Fine-grained Image Retrieval

SRIL: Selective Regularization for Class-Incremental Learning

Discrimination Correction and Balance for Class-Incremental Learning

Reducing Catastrophic Forgetting in Online Class Incremental Learning Using Self-Distillation

Distilled Meta-learning for Multi-Class Incremental Learning

A Robust and Anti-Forgettiable Model for Class-Incremental Learning

Class incremental learning with probability dampening and cascaded gated classifier

Incremental Scene Classification Using Dual Knowledge Distillation and Classifier Discrepancy on Natural and Remote Sensing Images

Model Behavior Preserving for Class-Incremental Learning

Max-margin Class Incremental Learning with Mixup Augmentation

Incremental Learning of Object Detectors without Catastrophic Forgetting

Overcoming Catastrophic Forgetting by Incremental Moment Matching

Incremental Learning Using Conditional Adversarial Networks

Online Class-incremental Continual Learning with Maximum Entropy Memory Update

Balanced softmax cross-entropy for incremental learning with and without memory

More Classifiers, Less Forgetting: A Generic Multi-classifier Paradigm for Incremental Learning