Abstract:Intelligence necessitates memory. Without memory, humans fail to perform various nontrivial tasks such as reading novels, playing games or solving maths. As the ultimate goal of machine learning is to derive intelligent systems that learn and act automatically just like human, memory construction for machine is inevitable. Artificial neural networks model neurons and synapses in the brain by interconnecting computational units via weights, which is a typical class of machine learning algorithms that resembles memory structure. Their descendants with more complicated modeling techniques (a.k.a deep learning) have been successfully applied to many practical problems and demonstrated the importance of memory in the learning process of machinery systems. Recent progresses on modeling memory in deep learning have revolved around external memory constructions, which are highly inspired by computational Turing models and biological neuronal systems. Attention mechanisms are derived to support acquisition and retention operations on the external memory. Despite the lack of theoretical foundations, these approaches have shown promises to help machinery systems reach a higher level of intelligence. The aim of this thesis is to advance the understanding on memory and attention in deep learning. Its contributions include: (i) presenting a collection of taxonomies for memory, (ii) constructing new memory-augmented neural networks (MANNs) that support multiple control and memory units, (iii) introducing variability via memory in sequential generative models, (iv) searching for optimal writing operations to maximise the memorisation capacity in slot-based memory networks, and (v) simulating the Universal Turing Machine via Neural Stored-program Memory-a new kind of external memory for neural networks.

Learning to Generate with Memory

MemoryGAN: GAN Generator As Heterogeneous Memory for Compositional Image Synthesis

Sequence Modeling with Hierarchical Deep Generative Models with Dual Memory.

Generative Modeling with Explicit Memory

Generative Memory for Lifelong Learning

Memory Replay GANs: Learning to Generate Images from New Categories Without Forgetting

MFS: A Brain-Inspired Memory Formation System for GAN

A Geometric Framework for Understanding Memorization in Generative Models

Memory Augmented Generative Adversarial Networks for Anomaly Detection

GAN Memory with No Forgetting

Generative Memory for Lifelong Reinforcement Learning

Few-shot Generation Via Recalling Brain-Inspired Episodic-Semantic Memory.

Memory Enhanced Replay for Continual Learning

Learning Deep Generative Models with Doubly Stochastic Gradient MCMC

Lifelong Learning Process: Self-Memory Supervising and Dynamically Growing Networks

Embedding Space Selection for Detecting Memorization and Fingerprinting in Generative Models

Memory and attention in deep learning

Multi-objects Generation with Amortized Structural Regularization.

Learning to Learn Variational Semantic Memory.

Deep Generative Dual Memory Network for Continual Learning

Generative Model for Heterogeneous Inference.