Evaluating Machine Translation Performance on Chinese Idioms with a Blacklist Method

Yutong Shao,Rico Sennrich,Bonnie Webber,Federico Fancellu
DOI: https://doi.org/10.48550/arXiv.1711.07646
2017-11-21
Computation and Language
Abstract:Idiom translation is a challenging problem in machine translation because the meaning of idioms is non-compositional, and a literal (word-by-word) translation is likely to be wrong. In this paper, we focus on evaluating the quality of idiom translation of MT systems. We introduce a new evaluation method based on an idiom-specific blacklist of literal translations, based on the insight that the occurrence of any blacklisted words in the translation output indicates a likely translation error. We introduce a dataset, CIBB (Chinese Idioms Blacklists Bank), and perform an evaluation of a state-of-the-art Chinese-English neural MT system. Our evaluation confirms that a sizable number of idioms in our test set are mistranslated (46.1%), that literal translation error is a common error type, and that our blacklist method is effective at identifying literal translation errors.
What problem does this paper attempt to address?