Abstract:Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

Feedforward Sequential Memory Networks Based Encoder-Decoder Model for Machine Translation

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation.

Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency.

Is Encoder-Decoder Redundant for Neural Machine Translation?

Feedforward Sequential Memory Neural Networks without Recurrent Feedback

Fast Structured Decoding for Sequence Models

Cache Friendly Parallelization of Neural Encoder-Decoder Models Without Padding on Multi-core Architecture.

Encoding Gated Translation Memory into Neural Machine Translation

Deconvolution-Based Global Decoding for Neural Machine Translation.

Multi-channel Encoder for Neural Machine Translation

Multi-Head Highly Parallelized LSTM Decoder for Neural Machine Translation

A Context-Aware Recurrent Encoder for Neural Machine Translation

Sequence to Sequence Learning with Neural Networks

Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation

Fast Decoding in Sequence Models using Discrete Latent Variables

Parallelizing and Optimizing Neural Encoder–Decoder Models Without Padding on Multi-Core Architecture

A Simple but Effective Way to Improve the Performance of RNN-Based Encoder in Neural Machine Translation Task

Recurrent Positional Embedding for Neural Machine Translation.

Recurrent Neural Machine Translation.

Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction