Abstract:Attention mechanism, which selectively focuses on source-side information to learn a context vector for generating target words, has been shown to be an effective method for neural machine translation (NMT). In fact, generating target words depends on not only the source-side information but also the target-side information. Although the vanilla NMT can acquire target-side information implicitly by recurrent neural networks (RNN), RNN cannot adequately capture the global relationship between target-side words. To solve this problem, this paper proposes a novel target-attention approach to capture this information, thus enhancing target word predictions in NMT. Specifically, we propose three variants of target-attention model to directly obtain the global relationship among target words: 1) a forward target-attention model that uses a target attention mechanism to incorporate previous historical target words into the prediction of the current target word; 2) a reverse target-attention model that adopts a reverse RNN model to obtain the entire reverse target words information, and then to combine with source context information to generate target sequence; 3) a bidirectional target-attention model that combines the forward target-attention model and reverse target-attention model together, which can make full use of target words to further improve the performance of NMT. Our methods can be integrated into both RNN based NMT and self-attention based NMT, and help NMT get global target-side information to improve translation performance. Experiments on the NIST Chinese-to-English and the WMT English-to-German translation tasks show that the proposed models achieve significant improvements over state-of-the-art baselines.

Adaptation of Language Models for SMT Using Neural Networks with Topic Information.

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

An Investigation On Statistical Machine Translation With Neural Language Models

Translating with Bilingual Topic Knowledge for Neural Machine Translation.

Translation Model Adaptation for Statistical Machine Translation with Monolingual Topic Information.

Neural Machine Translation with Sentence-Level Topic Context

A Topic-Triggered Language Model for Statistical Machine Translation.

A Novel Neural Topic Model and Its Supervised Extension.

Neural Topic Modeling with Large Language Models in the Loop

Topic Model Based Adaptation Data Selection for Domain-Specific Machine Translation.

Learning Multilingual Topics with Neural Variational Inference

Neural Machine Translation Advised by Statistical Machine Translation

Additive Neural Networks for Statistical Machine Translation.

TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Linguistic Knowledge-Aware Neural Machine Translation

Language Model-Driven Unsupervised Neural Machine Translation

SAE-NTM: Sentence-Aware Encoder for Neural Topic Modeling

A Context-Aware Topic Model for Statistical Machine Translation.

Neural Topic Modeling with Deep Mutual Information Estimation

Neural Machine Translation with Target-Attention Model.

Incorporating Statistical Machine Translation Word Knowledge into Neural Machine Translation