Abstract:Recent machine translation (MT) metrics calibrate their effectiveness by correlating with human judgement but without any insights about their behaviour across different error types. Challenge sets are used to probe specific dimensions of metric behaviour but there are very few such datasets and they either focus on a limited number of phenomena or a limited number of language pairs. We introduce ACES, a contrastive challenge set spanning 146 language pairs, aimed at discovering whether metrics can identify 68 translation accuracy errors. These phenomena range from simple alterations at the word/character level to more complex errors based on discourse and real-world knowledge. We conduct a large-scale study by benchmarking ACES on 50 metrics submitted to the WMT 2022 and 2023 metrics shared tasks. We benchmark metric performance, assess their incremental performance over successive campaigns, and measure their sensitivity to a range of linguistic phenomena. We also investigate claims that Large Language Models (LLMs) are effective as MT evaluators by evaluating on ACES. Our results demonstrate that different metric families struggle with different phenomena and that LLM-based methods fail to demonstrate reliable performance. Our analyses indicate that most metrics ignore the source sentence, tend to prefer surface-level overlap and end up incorporating properties of base models which are not always beneficial. We expand ACES to include error span annotations, denoted as SPAN-ACES and we use this dataset to evaluate span-based error metrics showing these metrics also need considerable improvement. Finally, we provide a set of recommendations for building better MT metrics, including focusing on error labels instead of scores, ensembling, designing strategies to explicitly focus on the source sentence, focusing on semantic content and choosing the right base model for representations.

MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Task

ACES: Translation Accuracy Challenge Sets at WMT 2023

MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration

xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics

Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!

The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control

Alibaba-Translate China's Submission for WMT 2022 Quality Estimation Shared Task

Empowering a Metric with LLM-assisted Named Entity Annotation: HW-TSC's Submission to the WMT23 Metrics Shared Task.

A Description of Tunable Machine Translation Evaluation Systems in WMT13 Metrics Task

Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets

Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task

Microsoft's Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data

Consistent Human Evaluation of Machine Translation across Language Pairs

Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies

Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation

HW-TSC's Participation at WMT 2021 Quality Estimation Shared Task.

Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation.

Quality and Quantity of Machine Translation References for Automatic Metrics

ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics

Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task