Abstract:In recent years, neural code translation has gained increasing attention. While most of the research focuses on improving model architectures and training processes, we notice that the evaluation process and benchmark for code translation models are severely limited: they primarily treat source code as natural languages and provide a holistic accuracy score while disregarding the full spectrum of model capabilities across different translation types and complexity. In this paper, we present a comprehensive investigation of four state-of-the-art models and analyze in-depth the advantages and limitations of three existing benchmarks. Based on the empirical results, we develop a taxonomy that categorizes code translation tasks into four primary types according to their complexity and knowledge dependence: token level (type 1), syntactic level (type 2), library level (type 3), and algorithm level (type 4). We then conduct a thorough analysis of how existing approaches perform across these four categories. Our findings indicate that while state-of-the-art code translation models excel in type-1 and type-2 translations, they struggle with knowledge-dependent ones such as type-3 and type-4. Existing benchmarks are biased towards trivial translations, such as keyword mapping. To overcome these limitations, we construct G-TransEval, a new benchmark by manually curating type-3 and type-4 translation pairs and unit test cases. Results on our new benchmark suggest that G-TransEval can exhibit more comprehensive and finer-grained capability of code translation models and thus provide a more rigorous evaluation. Our studies also provide more insightful findings and suggestions for future research, such as building type-3 and type-4 training data and ensembling multiple pretraining approaches.

Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite

A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation

Translation of Pronominal Anaphora between English and Spanish: Discrepancies and Evaluation

Quality Estimation based Feedback Training for Improving Pronoun Translation

Machine Translation: Phrase-Based, Rule-Based and Neural Approaches with Linguistic Evaluation

Context-Aware Neural Machine Translation Learns Anaphora Resolution

A sort Approach for Anaphora Resolution of Chinese Personal Pronoun Based on Machine Learning Method

What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns

A Test Suite for Evaluating Discourse Phenomena in Document-level Neural Machine Translation

Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation

Mention Attention for Pronoun Translation

Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatian

Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

A Comprehensive Survey on Various Fully Automatic Machine Translation Evaluation Metrics

On the Evaluation of Neural Code Translation: Taxonomy and Benchmark

A Set of Recommendations for Assessing Human-Machine Parity in Language Translation

Translationese in Machine Translation Evaluation

On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?

The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics

A Quantitative Analysis of Linguistic Factors in Human Translation Evaluation

On Measuring Gender Bias in Translation of Gender-neutral Pronouns