Abstract:Word-level Quality Estimation (QE) of Machine Translation (MT) aims to find out potential translation errors in the translated sentence without reference. Typically, conventional works on word-level QE are designed to predict the translation quality in terms of the post-editing effort, where the word labels ("OK" and "BAD") are automatically generated by comparing words between MT sentences and the post-edited sentences through a Translation Error Rate (TER) toolkit. While the post-editing effort can be used to measure the translation quality to some extent, we find it usually conflicts with the human judgement on whether the word is well or poorly translated. To overcome the limitation, we first create a golden benchmark dataset, namely \emph{HJQE} (Human Judgement on Quality Estimation), where the expert translators directly annotate the poorly translated words on their judgements. Additionally, to further make use of the parallel corpus, we propose the self-supervised pre-training with two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to \emph{HJQE}. We conduct substantial experiments based on the publicly available WMT En-De and En-Zh corpora. The results not only show our proposed dataset is more consistent with human judgment but also confirm the effectiveness of the proposed tag correcting strategies.\footnote{The data can be found at \url{https://github.com/ZhenYangIACAS/HJQE}.}

HW-TSC's Participation at WMT 2020 Quality Estimation Shared Task.

HW-TSC's Participation at WMT 2021 Quality Estimation Shared Task.

HW-TSC's Participation in the WMT 2023 Automatic Post Editing Shared Task.

HW-TSC 2023 Submission for the Quality Estimation Shared Task.

HW-TSC's Participation in the WMT 2020 News Translation Shared Task.

NJU's submission to the WMT20 QE Shared Task.

Submissions for the WMT 19 Quality Estimation Shared Task

HW-TSC's Participation in the WMT 2021 News Translation Shared Task.

HW-TSC's Participation in the WMT 2021 Triangular MT Shared Task.

Alibaba-Translate China's Submission for WMT 2022 Quality Estimation Shared Task

Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task

NJU Submissions for the WMT19 Quality Estimation Shared Task

Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

WeChat Neural Machine Translation Systems for WMT20

DiDi's Machine Translation System for WMT2020

The NiuTrans System for the WMT20 Quality Estimation Shared Task.

HW-TSC’s Participation in the WMT 2021 Large-Scale Multilingual Translation Task

An Improved Multi-task Approach to Pre-trained Model Based MT Quality Estimation.

Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement

Ensemble-based Transfer Learning for Low-resource Machine Translation Quality Estimation

NJUNLP's Submission for CCMT20 Quality Estimation Task.