Abstract:Existing Neural Machine Translation (NMT) systems are generally trained on a large amount of sentence-level parallel data, and during prediction sentences are independently translated, ignoring cross-sentence contextual information. This leads to inconsistency between translated sentences. In order to address this issue, context-aware models have been proposed. However, document-level parallel data constitutes only a small part of the parallel data available, and many approaches build context-aware models based on a pre-trained frozen sentence-level translation model in a two-step training manner. The computational cost of these approaches is usually high. In this paper, we propose to make the most of layers pre-trained on sentence-level data in contextual representation learning, reusing representations from the sentence-level Transformer and significantly reducing the cost of incorporating contexts in translation. We find that representations from shallow layers of a pre-trained sentence-level encoder play a vital role in source context encoding, and propose to perform source context encoding upon weighted combinations of pre-trained encoder layers' outputs. Instead of separately performing source context and input encoding, we propose to iteratively and jointly encode the source input and its contexts and to generate input-aware context representations with a cross-attention layer and a gating mechanism, which resets irrelevant information in context encoding. Our context-aware Transformer model outperforms the recent CADec [Voita et al., 2019c] on the English-Russian subtitle data and is about twice as fast in training and decoding.

Document-Level Machine Translation with Effective Batch-Level Context Representation

Towards Making the Most of Context in Neural Machine Translation

Document-level Neural Machine Translation with Document Embeddings

Document-level Neural Machine Translation Using BERT As Context Encoder

Diving Deep into Context-Aware Neural Machine Translation

Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation

Document-Level Neural Machine Translation with Hierarchical Modeling of Global Context

Efficient Context-Aware Neural Machine Translation with Layer-Wise Weighting and Input-Aware Gating.

Document Graph for Neural Machine Translation

Semantically Constrained Document-Level Chinese-Mongolian Neural Machine Translation

Context-Adaptive Document-Level Neural Machine Translation

Document-level Neural Machine Translation with Inter-Sentence Attention

Capturing document context inside sentence-level neural machine translation models with self-training

A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning

Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation

HanoiT: Enhancing Context-aware Translation via Selective Context

Multi-Hop Transformer for Document-Level Machine Translation

Improving the Transformer Translation Model with Document-Level Context

SMDT: Selective Memory-Augmented Neural Document Translation

Improving Context-Aware Neural Machine Translation with Source-side Monolingual Documents.

Exploiting Cross-Sentence Context for Neural Machine Translation