Abstract:It is well known that canonical recurrent neural networks (RNNs) face limitations in learning long-term dependencies which have been addressed by memory structures in long short-term memory (LSTM) networks. Neural Turing machines (NTMs) are novel RNNs that implement the notion of programmable computers with neural network controllers that can learn simple algorithmic tasks. Matrix neural networks feature matrix representation which inherently preserves the spatial structure of data when compared to canonical neural networks that use vector-based representation. One may then argue that neural networks with matrix representations may have the potential to provide better memory capacity. In this paper, we define and study a probabilistic notion of memory capacity based on Fisher information for matrix-based RNNs. We find bounds on memory capacity for such networks under various hypotheses and compare them with their vector counterparts. In particular, we show that the memory capacity of such networks is bounded by $N^2$ for $N\times N$ state matrix which generalizes the one known for vector networks. We also show and analyze the increase in memory capacity for such networks which is introduced when one exhibits an external state memory, such as NTMs. Consequently, we construct NTMs with RNN controllers with matrix-based representation of external memory, leading us to introduce Matrix NTMs. We demonstrate the performance of this class of memory networks under certain algorithmic learning tasks such as copying and recall and compare it with Matrix RNNs. We find an improvement in the performance of Matrix NTMs by the addition of external memory, in comparison to Matrix RNNs.

Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes

Structured Memory for Neural Turing Machines

Dynamic neural network with memristive CIM and CAM for 2D and 3D vision

A Continuous Time Dynamical Turing Machine

Hybrid computing using a neural network with dynamic external memory

On the Tensor Representation and Algebraic Homomorphism of the Neural State Turing Machine

Semantic memory-based dynamic neural network using memristive ternary CIM and CAM for 2D and 3D vision

Reservoir memory machines

Dendritic Neuron Model with Effective Learning Algorithms for Classification, Approximation, and Prediction

Ask Me Even More: Dynamic Memory Tensor Networks (Extended Model)

Dynamic neural fields as a step toward cognitive neuromorphic architectures

Cortical Processing with Thermodynamic-RAM

Memory Capacity of Recurrent Neural Networks with Matrix Representation

TYURINGMACHINE AND ARTIFICIAL NEURAL NETWORKS

One-shot Learning with Memory-Augmented Neural Networks

DARDA: Domain-Aware Real-Time Dynamic Neural Network Adaptation

Towards Deep Machine Reasoning: a Prototype-based Deep Neural Network with Decision Tree Inference

Hybrid Neural State Machine for Neural Network

Continuous Timescale Long-Short Term Memory Neural Network for Human Intent Understanding

D-NAT: Data-Driven Non-Ideality Aware Training Framework for Fabricated Computing-In-Memory Macros

A Multi-Layer Memristive Recurrent Neural Network for Solving Static and Dynamic Image Associative Memory