Abstract:Attention mechanism, which selectively focuses on source-side information to learn a context vector for generating target words, has been shown to be an effective method for neural machine translation (NMT). In fact, generating target words depends on not only the source-side information but also the target-side information. Although the vanilla NMT can acquire target-side information implicitly by recurrent neural networks (RNN), RNN cannot adequately capture the global relationship between target-side words. To solve this problem, this paper proposes a novel target-attention approach to capture this information, thus enhancing target word predictions in NMT. Specifically, we propose three variants of target-attention model to directly obtain the global relationship among target words: 1) a forward target-attention model that uses a target attention mechanism to incorporate previous historical target words into the prediction of the current target word; 2) a reverse target-attention model that adopts a reverse RNN model to obtain the entire reverse target words information, and then to combine with source context information to generate target sequence; 3) a bidirectional target-attention model that combines the forward target-attention model and reverse target-attention model together, which can make full use of target words to further improve the performance of NMT. Our methods can be integrated into both RNN based NMT and self-attention based NMT, and help NMT get global target-side information to improve translation performance. Experiments on the NIST Chinese-to-English and the WMT English-to-German translation tasks show that the proposed models achieve significant improvements over state-of-the-art baselines.

Look-ahead Attention for Generation in Neural Machine Translation

Neural Machine Translation with Target-Attention Model.

Learning When to Attend for Neural Machine Translation

Attention-via-Attention Neural Machine Translation

Optimizing Attention Mechanism for Neural Machine Transltion

Effective Approaches to Attention-based Neural Machine Translation

Sequence Generation with Target Attention.

Interactive Attention for Neural Machine Translation

Neural Machine Translation with Recurrent Attention Modeling

A GRU-Gated Attention Model for Neural Machine Translation

Neural Machine Translation with Supervised Attention

Syntax-Directed Attention for Neural Machine Translation

Modeling Coverage for Neural Machine Translation

Modeling Past and Future for Neural Machine Translation

Neural Machine Translation with Attention Based on a New Syntactic Branch Distance

Neural Machine Translation With GRU-Gated Attention Model

Coverage-based Neural Machine Translation.

Improving Neural Machine Translation Using Gated State Network and Focal Adaptive Attention Networtk

Temporal Attention Model for Neural Machine Translation

Autoregressive Modeling with Lookahead Attention

A Hierarchy-to-Sequence Attentional Neural Machine Translation Model.