Abstract:Recurrent neural networks (RNNs) have achieved state-of-the-art performances in many natural language processing tasks, such as language modeling and machine translation. However, when the vocabulary is large, the RNN model will become very big (e.g., possibly beyond the memory capacity of a GPU device) and its training will become very inefficient. In this work, we propose a novel technique to tackle this challenge. The key idea is to use 2-Component (2C) shared embedding for word representations. We allocate every word in the vocabulary into a table, each row of which is associated with a vector, and each column associated with another vector. Depending on its position in the table, a word is jointly represented by two components: a row vector and a column vector. Since the words in the same row share the row vector and the words in the same column share the column vector, we only need $2 \sqrt{|V|}$ vectors to represent a vocabulary of $|V|$ unique words, which are far less than the $|V|$ vectors required by existing approaches. Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets. The results show that our algorithm significantly reduces the model size and speeds up the training process, without sacrifice of accuracy (it achieves similar, if not better, perplexity as compared to state-of-the-art language models). Remarkably, on the One-Billion-Word benchmark Dataset, our algorithm achieves comparable perplexity to previous language models, whilst reducing the model size by a factor of 40-100, and speeding up the training process by a factor of 2. We name our proposed algorithm \emph{LightRNN} to reflect its very small model size and very high training speed.

Variable Computation in Recurrent Neural Networks

Residual Recurrent Neural Networks for Learning Sequential Representations.

Episodic Memory Theory for the Mechanistic Interpretation of Recurrent Neural Networks

Recurrent Segmentation for Variable Computational Budgets

Reversible Recurrent Neural Networks

Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels

Efficient recurrent architectures through activity sparsity and sparse back-propagation through time

Decomposing a Recurrent Neural Network into Modules for Enabling Reusability and Replacement

Machine Learning on Sequential Data Using a Recurrent Weighted Average

Restricted Recurrent Neural Networks

On the Provable Generalization of Recurrent Neural Networks

Parameter Compression of Recurrent Neural Networks and Degradation of Short-term Memory

A Critical Review of Recurrent Neural Networks for Sequence Learning

On the Long-Term Memory of Deep Recurrent Networks

Learning The Sequential Temporal Information with Recurrent Neural Networks

Saving RNN Computations with a Neuron-Level Fuzzy Memoization Scheme

LightRNN: Memory and Computation-Efficient Recurrent Neural Networks

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

Compressing Recurrent Neural Network with Tensor Train

Dual-module Inference for Efficient Recurrent Neural Networks