Abstract:Machine translation has wide applications in daily life. In mission-critical applications such as translating official documents, incorrect translation can have unpleasant or sometimes catastrophic consequences. This motivates recent research on testing methodologies for machine translation systems. Existing methodologies mostly rely on metamorphic relations designed at the textual level (e.g., Levenshtein distance) or syntactic level (e.g., the distance between grammar structures) to determine the correctness of translation results. However, these metamorphic relations do not consider whether the original and translated sentences have the same meaning (i.e., Semantic similarity). Therefore, in this paper, we propose SemMT, an automatic testing approach for machine translation systems based on semantic similarity checking. SemMT applies round-trip translation and measures the semantic similarity between the original and translated sentences. Our insight is that the semantics expressed by the logic and numeric constraint in sentences can be captured using regular expressions (or deterministic finite automata) where efficient equivalence/similarity checking algorithms are available. Leveraging the insight, we propose three semantic similarity metrics and implement them in SemMT. The experiment result reveals SemMT can achieve higher effectiveness compared with state-of-the-art works, achieving an increase of 21% and 23% on accuracy and F-Score, respectively. We also explore potential improvements that can be achieved when proper combinations of metrics are adopted. Finally, we discuss a solution to locate the suspicious trip in round-trip translation, which may shed lights on further exploration.

A Semantic Context-Aware Automatic Quality Scoring Method for Machine Translation Based on Pretraining Language Model

Quality Evaluation Model of Automatic Machine Translation Based on Deep Learning Algorithm

SemMT: A Semantic-based Testing Approach for Machine Translation Systems

Unsupervised quality estimation model for English to German translation and its application in extensive supervised evaluation.

Semantic Analysis and Evaluation of Translation Based on Abstract Meaning Representation

Evaluation of Machine Translation Based on Semantic Dependencies and Keywords

An Investigation On Statistical Machine Translation With Neural Language Models

Quality Estimation for Machine Translation Using the Joint Method of Evaluation Criteria and Statistical Modeling.

The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

Integrating Pre-trained Language Model into Neural Machine Translation

SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation

Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis

Exploring Human-Like Translation Strategy with Large Language Models

MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators

A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content

Sentiment-aware Multimodal Pre-Training for Multimodal Sentiment Analysis

Explicit Cross-lingual Pre-training for Unsupervised Machine Translation

SGPT: Semantic Graphs Based Pre-training for Aspect-based Sentiment Analysis

A Survey of Sentiment Analysis Based on Pretrained Language Models.

Exploring the Correlation between Human and Machine Evaluation of Simultaneous Speech Translation

Computer Assisted Translation with Neural Quality Estimation and Automatic Post-Editing