Abstract:As a new neural machine translation approach, Non-Autoregressive machine Translation (NAT) has attracted attention recently due to its high efficiency in inference. However, the high efficiency has come at the cost of not capturing the sequential dependency on the target side of translation, which causes NAT to suffer from two kinds of translation errors: 1) repeated translations (due to indistinguishable adjacent decoder hidden states), and 2) incomplete translations (due to incomplete transfer of source side information via the decoder hidden states). In this paper, we propose to address these two problems by improving the quality of decoder hidden representations via two auxiliary regularization terms in the training process of an NAT model. First, to make the hidden states more distinguishable, we regularize the similarity between consecutive hidden states based on the corresponding target tokens. Second, to force the hidden states to contain all the information in the source sentence, we leverage the dual nature of translation tasks (e.g., English to German and German to English) and minimize a backward reconstruction error to ensure that the hidden states of the NAT decoder are able to recover the source side sentence. Extensive experiments conducted on several benchmark datasets show that both regularization strategies are effective and can alleviate the issues of repeated translations and incomplete translations in NAT models. The accuracy of NAT models is therefore improved significantly over the state-of-the-art NAT models with even better efficiency for inference.

Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators

Hybrid-Regressive Paradigm for Accurate and Speed-Robust Neural Machine Translation

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Improving Non-Autoregressive Machine Translation Via Autoregressive Training

Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation

Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation

Improving Non-autoregressive Neural Machine Translation with Monolingual Data

Non-Autoregressive Machine Translation with Auxiliary Regularization

Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

Non-autoregressive Translation with Dependency-Aware Decoder

Reciprocal Supervised Learning Improves Neural Machine Translation

Character-Aware Low-Resource Neural Machine Translation With Weight Sharing And Pre-Training

Dynamic Mask Curriculum Learning for Non-Autoregressive Neural Machine Translation

Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints

Non-autoregressive neural machine translation with auxiliary representation fusion

Multi-Task Learning for Multiple Language Translation.

Reinforcement Learning for Edit-Based Non-Autoregressive Neural Machine Translation

Imitation Learning for Non-Autoregressive Neural Machine Translation.

Multi-agent Learning for Neural Machine Translation

Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC

Progressive Multi-Granularity Training for Non-Autoregressive Translation