Abstract:Machine translation systems have been widely adopted in our daily life, making life easier and more convenient. Unfortunately, erroneous translations may result in severe consequences, such as financial losses. This requires to improve the accuracy and the reliability of machine translation systems. However, it is challenging to test machine translation systems because of the complexity and intractability of the underlying neural models. To tackle these challenges, we propose a novel metamorphic testing approach by syntactic tree pruning (STP) to validate machine translation systems. Our key insight is that a pruned sentence should have similar crucial semantics compared with the original sentence. Specifically, STP (1) proposes a core semantics-preserving pruning strategy by basic sentence structure and dependency relations on the level of syntactic tree representation; (2) generates source sentence pairs based on the metamorphic relation; (3) reports suspicious issues whose translations break the consistency property by a bag-of-words model. We further evaluate STP on two state-of-the-art machine translation systems (i.e., Google Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs. The results show that STP can accurately find 5,073 unique erroneous translations in Google Translate and 5,100 unique erroneous translations in Bing Microsoft Translator (400% more than state-of-the-art techniques), with 64.5% and 65.4% precision, respectively. The reported erroneous translations vary in types and more than 90% of them cannot be found by state-of-the-art techniques. There are 9,393 erroneous translations unique to STP, which is 711.9% more than state-of-the-art techniques. Moreover, STP is quite effective to detect translation errors for the original sentences with a recall reaching 74.0%, improving state-of-the-art techniques by 55.1% on average.

Language Modeling for Syntax-Based Machine Translation Using Tree Substitution Grammars

A Tree-to-tree Alignment-Based Model for Statistical Machine Translation

A Statistical Machine Translation Model Based on a Synthetic Synchronous Grammar.

Grammar Comparison Study for Translational Equivalence Modeling and Statistical Machine Translation.

Subgraph based multitext grammar for statistical machine translation

N-gram-based Tense Models for Statistical Machine Translation

Forest-based Tree Sequence to String Translation Model

A SP synchronous generation model for statistical machine translation 1

Prior Derivation Models For Formally Syntax-Based Translation Using Linguistically Syntactic Parsing and Tree Kernels.

Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations

Statistical Translation Model Based On Source Syntax Structure

Meta-Structure Transformation Model for Statistical Machine Translation.

Syntax Encapsulated Phrase Model for Statistical Machine Translation

An Improved Hierarchical Phrase Based Machine Translation Model

Syntax-Based Context Representation for Statistical Machine Translation.

A Source Dependency Model for Statistical Machine Translation.

Syntax-Aware Complex-Valued Neural Machine Translation

A Tree Sequence Alignment-based Tree-to-Tree Translation Model.

Towards Integrated Machine Translation Using Structural Alignment From Syntax-Augmented Synchronous Parsing

Machine Translation Testing via Syntactic Tree Pruning

Improving Tree-to-Tree Translation with Packed Forests.