Abstract:Humans and other animals navigate different environments effortlessly, their brains rapidly and accurately generalizing across contexts. Despite recent progress in deep learning, this flexibility remains a challenge for many artificial systems. Here, we show how a bio-inspired network motif can explicitly address this issue. We do this using a dataset of MNIST digits of varying transparency, set on one of two backgrounds of different statistics that define two contexts: a pixel-wise noise or a more naturalistic background from the CIFAR-10 dataset. After learning digit classification when both contexts are shown sequentially, we find that both shallow and deep networks have sharply decreased performance when returning to the first background - an instance of the catastrophic forgetting phenomenon known from continual learning. To overcome this, we propose the bottleneck-switching network or switching network for short. This is a bio-inspired architecture analogous to a well-studied network motif in the visual cortex, with additional "switching" units that are activated in the presence of a new background, assuming a priori a contextual signal to turn these units on or off. Intriguingly, only a few of these switching units are sufficient to enable the network to learn the new context without catastrophic forgetting through inhibition of redundant background features. Further, the bottleneck-switching network can generalize to novel contexts similar to contexts it has learned. Importantly, we find that - again as in the underlying biological network motif, recurrently connecting the switching units to network layers is advantageous for context generalization.

A Bio-Inspired Incremental Learning Architecture for Applied Perceptual Problems

Incremental Learning with Differentiable Architecture and Forgetting Search

Self-Organizing Pathway Expansion for Non-Exemplar Class-Incremental Learning

Incremental Learning Without Looking Back: a Neural Connection Relocation Approach

Incremental Learning of Object Detectors without Catastrophic Forgetting

Incremental learning with neural networks for computer vision: a survey

Towards Class-incremental Object Detection with Nearest Mean of Exemplars

A Strategy for an Uncompromising Incremental Learner

SynapNet: A Complementary Learning System Inspired Algorithm With Real-Time Application in Multimodal Perception

A biologically inspired architecture with switching units can learn to generalize across backgrounds

Incremental Concept Formation over Visual Images Without Catastrophic Forgetting

Incremental Learning Via Rate Reduction

Class-Incremental Learning: Survey and Performance Evaluation on Image Classification

Prototype Representation Expansion in Incremental Learning

Incremental Learning In Online Scenario

Neural Architecture Search for Class-incremental Learning

Large Scale Incremental Learning

The dynamics of perceptual learning: an incremental reweighting model.

iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning

FearNet: Brain-Inspired Model for Incremental Learning

Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning