Abstract:Attention mechanism, which selectively focuses on source-side information to learn a context vector for generating target words, has been shown to be an effective method for neural machine translation (NMT). In fact, generating target words depends on not only the source-side information but also the target-side information. Although the vanilla NMT can acquire target-side information implicitly by recurrent neural networks (RNN), RNN cannot adequately capture the global relationship between target-side words. To solve this problem, this paper proposes a novel target-attention approach to capture this information, thus enhancing target word predictions in NMT. Specifically, we propose three variants of target-attention model to directly obtain the global relationship among target words: 1) a forward target-attention model that uses a target attention mechanism to incorporate previous historical target words into the prediction of the current target word; 2) a reverse target-attention model that adopts a reverse RNN model to obtain the entire reverse target words information, and then to combine with source context information to generate target sequence; 3) a bidirectional target-attention model that combines the forward target-attention model and reverse target-attention model together, which can make full use of target words to further improve the performance of NMT. Our methods can be integrated into both RNN based NMT and self-attention based NMT, and help NMT get global target-side information to improve translation performance. Experiments on the NIST Chinese-to-English and the WMT English-to-German translation tasks show that the proposed models achieve significant improvements over state-of-the-art baselines.

Neural Machine Translation with Attention Based on a New Syntactic Branch Distance

Syntax-Directed Attention for Neural Machine Translation

Effective Approaches to Attention-based Neural Machine Translation

Neural Machine Translation with Deep Attention

Syntax-aware Neural Machine Translation Directed by Syntactic Dependency Degree

Learning When to Attend for Neural Machine Translation

Neural Machine Translation with Recurrent Attention Modeling

Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder

Neural Machine Translation with Phrasal Attention.

Neural Machine Translation with Target-Attention Model.

Boosting Neural Machine Translation with Dependency-Scaled Self-Attention Network.

Neural Machine Translation with Supervised Attention

Attention-via-Attention Neural Machine Translation

Improving BERT with Syntax-aware Local Attention

Enhancing low-resource neural machine translation with syntax-graph guided self-attention

Towards Neural Machine Translation with Latent Tree Attention

Bilingual Attention Based Neural Machine Translation

Syntax-based Transformer for Neural Machine Translation

Syntax-Aware Complex-Valued Neural Machine Translation

Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings

Improving Neural Machine Translation with Neural Syntactic Distance