Abstract:Our brains extract durable, generalizable knowledge from transient experiences of the world. Artificial neural networks come nowhere close to this ability. When tasked with learning to classify objects by training on non-repeating video frames in temporal order (online stream learning), models that learn well from shuffled datasets catastrophically forget old knowledge upon learning new stimuli. We propose a new continual learning algorithm, Compositional Replay Using Memory Blocks (CRUMB), which mitigates forgetting by replaying feature maps reconstructed by combining generic parts. CRUMB concatenates trainable and re-usable "memory block" vectors to compositionally reconstruct feature map tensors in convolutional neural networks. Storing the indices of memory blocks used to reconstruct new stimuli enables memories of the stimuli to be replayed during later tasks. This reconstruction mechanism also primes the neural network to minimize catastrophic forgetting by biasing it towards attending to information about object shapes more than information about image textures, and stabilizes the network during stream learning by providing a shared feature-level basis for all training examples. These properties allow CRUMB to outperform an otherwise identical algorithm that stores and replays raw images, while occupying only 3.6% as much memory. We stress-tested CRUMB alongside 13 competing methods on 7 challenging datasets. To address the limited number of existing online stream learning datasets, we introduce 2 new benchmarks by adapting existing datasets for stream learning. With only 3.7-4.1% as much memory and 15-43% as much runtime, CRUMB mitigates catastrophic forgetting more effectively than the state-of-the-art. Our code is available at <a class="link-external link-https" href="https://github.com/MorganBDT/crumb.git" rel="external noopener nofollow">this https URL</a>.

Memory-efficient training with streaming dimensionality reduction

Low-Rank Gradient Descent for Memory-Efficient Training of Deep In-Memory Arrays

Streaming Batch Gradient Tracking for Neural Network Training (student Abstract).

A Parallel Framework for Streaming Dimensionality Reduction

A scalable supervised algorithm for dimensionality reduction on streaming data

Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis

Streaming Batch Eigenupdates for Hardware Neural Networks.

Memory Efficient Experience Replay for Streaming Learning

An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data

ON TRAINING DEEP NEURAL NETWORKS USING A STREAMING APPROACH

Edge-Cloud Collaborative Streaming Video Analytics with Multi-agent Deep Reinforcement Learning

Efficient Unsupervised Dimension Reduction for Streaming Multiview Data.

Memory Efficient On-Line Streaming for Multichannel Spike Train Analysis

Rivalry of Two Families of Algorithms for Memory-Restricted Streaming PCA

Efficient Principal Subspace Projection of Streaming Data Through Fast Similarity Matching

Summarizing Stream Data for Memory-Constrained Online Continual Learning

Streaming Probabilistic Deep Tensor Factorization

Batch Adaptative Streaming for Video Analytics

Dual Memory Architectures for Fast Deep Learning of Stream Data via an Online-Incremental-Transfer Strategy

Don't Think It Twice: Exploit Shift Invariance for Efficient Online Streaming Inference of CNNs

Tuned Compositional Feature Replays for Efficient Stream Learning