Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning

Rongxiang Weng,Qiang Wang,Wensen Cheng,Changfeng Zhu,Min Zhang

2023-09-19

Abstract:Neural machine translation (NMT) has achieved remarkable success in producing high-quality translations. However, current NMT systems suffer from a lack of reliability, as their outputs that are often affected by lexical or syntactic changes in inputs, resulting in large variations in quality. This limitation hinders the practicality and trustworthiness of NMT. A contributing factor to this problem is that NMT models trained with the one-to-one paradigm struggle to handle the source diversity phenomenon, where inputs with the same meaning can be expressed differently. In this work, we treat this problem as a bilevel optimization problem and present a consistency-aware meta-learning (CAML) framework derived from the model-agnostic meta-learning (MAML) algorithm to address it. Specifically, the NMT model with CAML (named CoNMT) first learns a consistent meta representation of semantically equivalent sentences in the outer loop. Subsequently, a mapping from the meta representation to the output sentence is learned in the inner loop, allowing the NMT model to translate semantically equivalent sentences to the same target sentence. We conduct experiments on the NIST Chinese to English task, three WMT translation tasks, and the TED M2O task. The results demonstrate that CoNMT effectively improves overall translation quality and reliably handles diverse inputs.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the issue of reliability in Neural Machine Translation (NMT) systems, particularly focusing on the challenge posed by the source diversity phenomenon. The source diversity problem refers to the fact that inputs with the same meaning can be expressed differently, leading to variations in the quality of translations produced by NMT models. Current NMT systems struggle with this issue because they are typically trained using a one-to-one paradigm, which does not handle the many-to-many nature of natural language well.

To tackle this problem, the authors propose a Consistency-Aware Meta-Learning (CAML) framework. CAML is designed to improve the reliability of NMT by enabling the model to learn a consistent meta-representation from semantically equivalent sentences. The framework consists of two main components:

1. **Learning a Consistent Meta Representation**: In the outer loop, CAML trains the NMT model to learn a consistent meta-representation of semantically equivalent sentences. This is achieved through two training objectives:
- Sentence-Level Objective: Forces semantically equivalent sentences to generate each other through partial reconstruction.
- Word-Level Objective: Constrains the output distributions from semantically equivalent sentences to be similar at each decoding step.

2. **Mapping from Meta Representation to Output Sentence**: In the inner loop, the NMT model learns to map the meta-representation to the target sentence, allowing for more consistent and reliable translations.