Abstract:Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicate and apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks. In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation. Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using part-of-speech (POS) to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages. In addition, we also present some novel work on quality estimation of MT without using reference translations including the usage of probability models of Naïve Bayes (NB), support vector machine (SVM) classification algorithms, and CRFs.

Evaluating Natural Language Generation Via Unbalanced Optimal Transport.

Toward Human-Like Evaluation for Natural Language Generation with Error Analysis

Evaluation Metrics of Language Generation Models for Synthetic Traffic Generation Tasks

Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

LEPOR: An Augmented Machine Translation Evaluation Metric

Test-time Assessment of a Model's Performance on Unseen Domains via Optimal Transport

NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models

Unbalancedness in Neural Monge Maps Improves Unpaired Domain Translation

Difficulty-Aware Machine Translation Evaluation

Reward Optimization for Neural Machine Translation with Learned Metrics

Scalable Unbalanced Optimal Transport using Generative Adversarial Networks

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory

Optimizing Non-Decomposable Evaluation Metrics for Neural Machine Translation

On the potential of Optimal Transport in Geospatial Data Science

Beyond Words: Evaluating Large Language Models in Transportation Planning

Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models

Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

Unbalanced Optimal Transport for Unbalanced Word Alignment