Abstract:Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicate and apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks. In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation. Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using part-of-speech (POS) to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages. In addition, we also present some novel work on quality estimation of MT without using reference translations including the usage of probability models of Naïve Bayes (NB), support vector machine (SVM) classification algorithms, and CRFs.

A SVM Regression Based Skip-Ngram Approach to MT Evaluation

A skip-ngram approach of automatic evaluation for machine translation using SVM regression

A Linguistically Motivated MT Evaluation System Based on SVM Regression

All in Strings: a Powerful String-Based Automatic MT Evaluation Metric with Multiple Granularities

Fusion of Word and Letter Based Metrics for Automatic MT Evaluation

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

Improvement of Machine Translation Evaluation by Simple Linguistically Motivated Features

LEPOR: An Augmented Machine Translation Evaluation Metric

A Study on Automatic Scoring for Machine Translation Systems

Evaluating MT quality by letter-based calculation

An Investigation On Statistical Machine Translation With Neural Language Models

A Description of Tunable Machine Translation Evaluation Systems in WMT13 Metrics Task

Automatic evaluation of sentence fluency

Application and Analysis of Sentence Similarity Based Machine Translation Evaluation

Improved Neural Machine Translation With Smt Features

A Survey on Evaluation Metrics for Machine Translation

Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese

MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

An automatic evaluation method for localization oriented lexicalised EBMT system

Document-Level Machine Translation Evaluation with Gist Consistency and Text Cohesion.

Trained MT Metrics Learn to Cope with Machine-translated References