Abstract:Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicate and apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks. In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation. Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using part-of-speech (POS) to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages. In addition, we also present some novel work on quality estimation of MT without using reference translations including the usage of probability models of Naïve Bayes (NB), support vector machine (SVM) classification algorithms, and CRFs.

Document-Level Machine Translation Evaluation Metrics Enhanced with Simplified Lexical Chain.

Document-Level Machine Translation Evaluation with Gist Consistency and Text Cohesion.

Fusion of Word and Letter Based Metrics for Automatic MT Evaluation

Lexical Chain Based Cohesion Models for Document-Level Statistical Machine Translation.

LEPOR: An Augmented Machine Translation Evaluation Metric

Difficulty-Aware Machine Translation Evaluation

Evaluating MT quality by letter-based calculation

A Comparison of Approaches to Document-level Machine Translation

Evaluating the Efficacy of Length-Controllable Machine Translation

Modeling Lexical Cohesion for Document-Level Machine Translation

Assessing Crosslingual Discourse Relations in Machine Translation

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

Improvement of Machine Translation Evaluation by Simple Linguistically Motivated Features

Evaluation of Machine Translation Based on Semantic Dependencies and Keywords

MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language

All in Strings: a Powerful String-Based Automatic MT Evaluation Metric with Multiple Granularities

BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation

Extrinsic Evaluation of Machine Translation Metrics

A Comprehensive Survey on Various Fully Automatic Machine Translation Evaluation Metrics

Novel Document Level Features for Statistical Machine Translation

Document-Level Machine Translation with Large Language Models