Abstract:Our brains extract durable, generalizable knowledge from transient experiences of the world. Artificial neural networks come nowhere close to this ability. When tasked with learning to classify objects by training on non-repeating video frames in temporal order (online stream learning), models that learn well from shuffled datasets catastrophically forget old knowledge upon learning new stimuli. We propose a new continual learning algorithm, Compositional Replay Using Memory Blocks (CRUMB), which mitigates forgetting by replaying feature maps reconstructed by combining generic parts. CRUMB concatenates trainable and re-usable "memory block" vectors to compositionally reconstruct feature map tensors in convolutional neural networks. Storing the indices of memory blocks used to reconstruct new stimuli enables memories of the stimuli to be replayed during later tasks. This reconstruction mechanism also primes the neural network to minimize catastrophic forgetting by biasing it towards attending to information about object shapes more than information about image textures, and stabilizes the network during stream learning by providing a shared feature-level basis for all training examples. These properties allow CRUMB to outperform an otherwise identical algorithm that stores and replays raw images, while occupying only 3.6% as much memory. We stress-tested CRUMB alongside 13 competing methods on 7 challenging datasets. To address the limited number of existing online stream learning datasets, we introduce 2 new benchmarks by adapting existing datasets for stream learning. With only 3.7-4.1% as much memory and 15-43% as much runtime, CRUMB mitigates catastrophic forgetting more effectively than the state-of-the-art. Our code is available at <a class="link-external link-https" href="https://github.com/MorganBDT/crumb.git" rel="external noopener nofollow">this https URL</a>.

Stacked Bin Convolutional Neural Networks based Sparse Low-Rank Regressor: Robust, Scalable and Novel Model for Memorability Prediction of Videos

Creating Memorable Video Summaries That Satisfy the User's Intention for Taking the Videos.

Fast Real-Time Video Object Segmentation with a Tangled Memory Network

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

Learning Computational Models of Video Memorability from Fmri Brain Imaging

Learning Low-Rank Sparse Representations With Robust Relationship Inference for Image Memorability Prediction

A novel spatio-temporal memory network for video anomaly detection

Predictive Coding Based Multiscale Network with Encoder-Decoder LSTM for Video Prediction

Tuned Compositional Feature Replays for Efficient Stream Learning

Deep Multi-Scale Pyramidal Features Network for Supervised Video Summarization.

Stacked residual blocks based encoder-decoder framework for human motion prediction

Stacked Multimodal Attention Network for Context-Aware Video Captioning

Memory-Efficient Network for Large-scale Video Compressive Sensing

Video SemNet: Memory-Augmented Video Semantic Network

Learning Video Object Segmentation with Visual Memory

Memorize, then Recall: A Generative Framework for Low Bit-Rate Surveillance Video Compression

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation

STAM: A SpatioTemporal Attention Based Memory for Video Prediction

Intelligent 3D Network Protocol for Multimedia Data Classification using Deep Learning

Multi-Memory Convolutional Neural Network for Video Super-Resolution.