Abstract:This study aims to analyse how translation experts from the German department of the European Commission’s Directorate-General for Translation (DGT) identify and correct different error categories in neural machine translated texts (NMT) and their post-edited versions (NMTPE). The term translation expert encompasses translator, post-editor as well as revisor. Even though we focus on neural machine-translated segments, translator and post-editor are used synonymously because of the combined workflow using CAT-Tools as well as machine translation. Only the distinction between post-editor, which refers to a DGT translation expert correcting the neural machine translation output, and revisor, which refers to a DGT translation expert correcting the post-edited version of the neural machine translation output, is important and made clear whenever relevant. Using an automatic error annotation tool and the more fine-grained manual error annotation framework to identify characteristic error categories in the DGT texts, a corpus analysis revealed that quality assurance measures by post-editors and revisors of the DGT are most often necessary for lexical errors. More specifically, the corpus analysis showed that, if post-editors correct mistranslations, terminology or stylistic errors in an NMT sentence, revisors are likely to correct the same error type in the same post-edited sentence, suggesting that the DGT experts were being primed by the NMT output. Subsequently, we designed a controlled eye-tracking and key-logging experiment to compare participants’ eye movements for test sentences containing the three identified error categories (mistranslations, terminology or stylistic errors) and for control sentences without errors. We examined the three error types’ effect on early (first fixation durations, first pass durations) and late eye movement measures (e.g., total reading time and regression path durations). Linear mixed-effects regression models predict what kind of behaviour of the DGT experts is associated with the correction of different error types during the post-editing process.

An Analysis of Source-Side Grammatical Errors in NMT

Translation Quality and Error Recognition in Professional Neural Machine Translation Post-Editing

The Unbearable Weight of Generating Artificial Errors for Grammatical Error Correction

An Automatic Error Detection Method for Machine Translation Results Via Deep Learning

Exploiting Monolingual Data at Scale for Neural Machine Translation.

Did Translation Models Get More Robust Without Anyone Even Noticing?

Improving Robustness of Machine Translation with Synthetic Noise

The use of residual analysis to improve the error rate accuracy of machine translation

Grammatical Error Generation Based on Translated Fragments

MTNT: A Testbed for Machine Translation of Noisy Text

Assessing the Tolerance of Neural Machine Translation Systems Against Speech Recognition Errors

Synthetic and Natural Noise Both Break Neural Machine Translation

Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation

Predicting Target Language CCG Supertags Improves Neural Machine Translation

Computer Assisted Translation with Neural Quality Estimation and Automatic Post-Editing

A linguistically motivated taxonomy for Machine Translation error analysis

On the Integration of LinguisticFeatures into Statistical and Neural Machine Translation

Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Space

Detecting Failures of Neural Machine Translation in the Absence of Reference Translations

Automatic Evaluation and Analysis of Idioms in Neural Machine Translation