Abstract:This study explores the interaction effect between source text (ST) complexity and machine translation (MT) quality on the task difficulty of neural machine translation (NMT) post-editing from English to Chinese. When investigating human effort exerted in post-editing, existing studies have seldom taken both ST complexity and MT quality levels into account, and have mainly focused on MT systems used before the emergence of NMT. Drawing on process and product data of post-editing from 60 trainee translators, this study adopted a multi-method approach to measure post-editing task difficulty, including eye-tracking, keystroke logging, quality evaluation, subjective rating, and retrospective written protocols. The results show that: 1) ST complexity and MT quality present a significant interaction effect on task difficulty of NMT post-editing; 2) ST complexity level has a positive impact on post-editing low-quality NMT (i.e., post-editing task becomes less difficult when ST complexity decreases); while for post-editing high-quality NMT, it has a positive impact only on the subjective ratings received from participants; and 3) NMT quality has a negative impact on its post-editing task difficulty (i.e., the post-editing task becomes less difficult when MT quality goes higher), and this impact becomes stronger when ST complexity increases. This paper concludes that both ST complexity and MT quality should be considered when testing post-editing difficulty, designing tasks for post-editor training, and setting fair post-editing pricing schemes.

Measuring Translation Difficulty an Empirical Study

Measuring Translation Difficulty: Theoretical and Methodological Considerations

Difficulty-Aware Machine Translation Evaluation

Comparing the Effect of Product-Based Metrics on the Translation Process

Man or machine? Comparing the difficulty of human translation versus neural machine translation post-editing

The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation

Dictating Translations with Automatic Speech Recognition: Effects on Translators' Performance.

It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information

Effects Of Thinking Aloud On Cognitive Effort In Translation

A Comparative Study on the Effort of Human Translation and Post-Editing in Relation to Text Types: An Eye-Tracking and Key-Logging Experiment

Predicting Human Translation Difficulty with Neural Machine Translation

Rater Reliability of Translation Testing

Measuring the User Experience of Computer-Aided Translation Systems: A Comparative Study

On difficulty and closeness in translation of English for science and technology

On the Inter-rater Reliability of Translation Tests of Above-medium Length for College Students

Text Readability and Processing Effort in Second Language Reading: A Computational and Eye‐Tracking Investigation

The interaction effect between source text complexity and machine translation quality on the task difficulty of NMT post-editing from English to Chinese: A multi-method study

Question Difficulty Ranking for Multiple-Choice Reading Comprehension

Cognitive load and cognitive effort

Traditional Readability Formulas Compared for English

Item-based assessment of translation competence: Chimera of objectivity versus prospect of reliable measurement