Abstract:Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicate and apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks. In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation. Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using part-of-speech (POS) to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages. In addition, we also present some novel work on quality estimation of MT without using reference translations including the usage of probability models of Naïve Bayes (NB), support vector machine (SVM) classification algorithms, and CRFs.

Meteor++: Incorporating Copy Knowledge into Machine Translation Evaluation

Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation.

Better Simultaneous Translation with Monotonic Knowledge Distillation.

A Description of Tunable Machine Translation Evaluation Systems in WMT13 Metrics Task

Improve the Evaluation of Fluency Using Entropy for Machine Translation Evaluation Metrics

Exploring Context-Aware Evaluation Metrics for Machine Translation

xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics

Difficulty-Aware Machine Translation Evaluation

BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation

MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language

The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics

LEPOR: An Augmented Machine Translation Evaluation Metric

MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibration

Document-Level Machine Translation Evaluation with Gist Consistency and Text Cohesion.

ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics

Evaluating Terminology Translation in Machine Translation Systems Via Metamorphic Testing

Learning to Copy for Automatic Post-Editing

Uncertainty-Aware Machine Translation Evaluation

xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection

Towards Explainable Evaluation Metrics for Machine Translation

Pitfalls and Outlooks in Using COMET