Abstract:Since the 1950s, machine translation (MT) has become one of the important tasks of AI and development, and has experienced several different periods and stages of development, including rule-based methods, statistical methods, and recently proposed neural network-based learning methods. Accompanying these staged leaps is the evaluation research and development of MT, especially the important role of evaluation methods in statistical translation and neural translation research. The evaluation task of MT is not only to evaluate the quality of machine translation, but also to give timely feedback to machine translation researchers on the problems existing in machine translation itself, how to improve and how to optimise. In some practical application fields, such as in the absence of reference translations, the quality estimation of machine translation plays an important role as an indicator to reveal the credibility of automatically translated target languages. This report mainly includes the following contents: a brief history of machine translation evaluation (MTE), the classification of research methods on MTE, and the the cutting-edge progress, including human evaluation, automatic evaluation, and evaluation of evaluation methods (meta-evaluation). Manual evaluation and automatic evaluation include reference-translation based and reference-translation independent participation; automatic evaluation methods include traditional n-gram string matching, models applying syntax and semantics, and deep learning models; evaluation of evaluation methods includes estimating the credibility of human evaluations, the reliability of the automatic evaluation, the reliability of the test set, etc. Advances in cutting-edge evaluation methods include task-based evaluation, using pre-trained language models based on big data, and lightweight optimisation models using distillation techniques.

CCMT 2019 Machine Translation Evaluation Report

Description and Findings of OPPO's Machine Translation Systems for CCMT 2020

HW-TSC's Submission to the CCMT 2024 Machine Translation Tasks

NJUNLP’s Machine Translation System for CCMT-2020 Uighur → Chinese Translation Task

NJUNLP’s Machine Translation System for CCMT-2020 Uighur $$\rightarrow $$ Chinese Translation Task

Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Task

A Description of Tunable Machine Translation Evaluation Systems in WMT13 Metrics Task

The ICT Statistical Machine Translation Systems for IWSLT 2007.

Alibaba-Translate China's Submission for WMT 2022 Quality Estimation Shared Task

Tsinghua University Neural Machine Translation Systems for CCMT 2020

Difficulty-Aware Machine Translation Evaluation

Fine-grained linguistic evaluation for state-of-the-art Machine Translation

DiDi's Machine Translation System for WMT2020

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task

WeChat Neural Machine Translation Systems for WMT20

An Overview on Machine Translation Evaluation

The ICT System Description for IWSLT 2008

SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task

A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content

Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese

Evaluation of Chinese-English Machine Translation of Emotion-Loaded Microblog Texts: A Human Annotated Dataset for the Quality Assessment of Emotion Translation