Abstract:Directly training a document-to-document (Doc2Doc) neural machine translation (NMT) via Transformer from scratch, especially on small datasets, usually fails to converge. Our dedicated probing tasks show that 1) both the absolute position and relative position information gets gradually weakened or even vanished once it reaches the upper encoder layers, and 2) the vanishing of absolute position information in encoder output causes the training failure of Doc2Doc NMT. To alleviate this problem, we propose a position-aware Transformer (P-Transformer) to enhance both the absolute and relative position information in both self-attention and cross-attention. Specifically, we integrate absolute positional information, i.e., position embeddings, into the query-key pairs both in self-attention and cross-attention through a simple yet effective addition operation. Moreover, we also integrate relative position encoding in self-attention. The proposed P-Transformer utilizes sinusoidal position encoding and does not require any task-specified position embedding, segment embedding, or attention mechanism. Through the above methods, we build a Doc2Doc NMT model with P-Transformer, which ingests the source document and completely generates the target document in a sequence-to-sequence (seq2seq) way. In addition, P-Transformer can be applied to seq2seq-based document-to-sentence (Doc2Sent) and sentence-to-sentence (Sent2Sent) translations. Extensive experimental results of Doc2Doc NMT show that P-Transformer significantly outperforms strong baselines on the widely-used 9 document-level datasets in 7 language pairs, covering small-, middle-, and large-scales, and achieves a new state-of-the-art. Experimentation on discourse phenomena shows that our Doc2Doc NMT models improve the translation quality in both BLEU and discourse coherence. We make our code available on Github.

Rethinking Document-level Neural Machine Translation

Document-level Neural Machine Translation with Document Embeddings

Document-level Neural Machine Translation with Inter-Sentence Attention

Towards Making the Most of Context in Neural Machine Translation

Document-Level Language Models for Machine Translation

Improving the Transformer Translation Model with Document-Level Context

Document-level Neural Machine Translation with Associated Memory Network

A Survey on Document-level Neural Machine Translation: Methods and Evaluation

Exploring Discourse Structure in Document-level Machine Translation

Multi-Hop Transformer for Document-Level Machine Translation

Only 5\% Attention Is All You Need: Efficient Long-range Document-level Neural Machine Translation

Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation

Document-level Neural Machine Translation Using BERT As Context Encoder

Non-Autoregressive Document-Level Machine Translation

Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

G-Transformer for Document-level Machine Translation

Pretrained Language Models for Document-Level Neural Machine Translation

Context-Adaptive Document-Level Neural Machine Translation

Leveraging Discourse Rewards for Document-Level Neural Machine Translation

A Comparison of Approaches to Document-level Machine Translation

P-Transformer: Towards Better Document-to-Document Neural Machine Translation