Abstract:Traditionally, Machine Translation (MT) Evaluation has been treated as a regression problem -- producing an absolute translation-quality score. This approach has two limitations: i) the scores lack interpretability, and human annotators struggle with giving consistent scores; ii) most scoring methods are based on (reference, translation) pairs, limiting their applicability in real-world scenarios where references are absent. In practice, we often care about whether a new MT system is better or worse than some competitors. In addition, reference-free MT evaluation is increasingly practical and necessary. Unfortunately, these two practical considerations have yet to be jointly explored. In this work, we formulate the reference-free MT evaluation into a pairwise ranking problem. Given the source sentence and a pair of translations, our system predicts which translation is better. In addition to proposing this new formulation, we further show that this new paradigm can demonstrate superior correlation with human judgments by merely using indirect supervision from natural language inference and weak supervision from our synthetic data. In the context of reference-free evaluation, MT-Ranker, trained without any human annotations, achieves state-of-the-art results on the WMT Shared Metrics Task benchmarks DARR20, MQM20, and MQM21. On a more challenging benchmark, ACES, which contains fine-grained evaluation criteria such as addition, omission, and mistranslation errors, MT-Ranker marks state-of-the-art against reference-free as well as reference-based baselines.

Simulated Multiple Reference Training Improves Low-Resource Machine Translation

Multiple References with Meaningful Variations Improve Literary Machine Translation

Better Simultaneous Translation with Monotonic Knowledge Distillation.

Insight Into Multiple References In An Mt Evaluation Metric

MT-Ranker: Reference-free machine translation evaluation by inter-system ranking

Simultaneous Machine Translation with Tailored Reference

Meta-Learning for Low-Resource Neural Machine Translation.

Handling Syntactic Divergence in Low-resource Machine Translation

Improving data augmentation for low resource speech-to-text translation with diverse paraphrasing

To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages

Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation

Multi-Round Transfer Learning for Low-Resource NMT Using Multiple High-Resource Languages

Low-Resource Machine Translation Training Curriculum Fit for Low-Resource Languages

Combining Multiple Resources to Improve SMT-based Paraphrasing Model

Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages

Translation of Multifaceted Data without Re-Training of Machine Translation Systems

Rephrasing the Reference for Non-Autoregressive Machine Translation

An Efficient Method for Generating Synthetic Data for Low-Resource Machine Translation

BitextEdit: Automatic Bitext Editing for Improved Low-Resource Machine Translation

Enhancing Neural Machine Translation of Low-Resource Languages: Corpus Development, Human Evaluation and Explainable AI Architectures

Self-supervised and Supervised Joint Training for Resource-rich Machine Translation