Abstract:Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment, providing both scores and fine-grained feedback. Although approaches such as GEMBA-MQM has shown SOTA performance on reference-free evaluation, the predicted errors do not align well with those annotated by human, limiting their interpretability as feedback signals. To enhance the quality of error annotations predicted by LLM evaluators, we introduce a universal and training-free framework, $\textbf{MQM-APE}$, based on the idea of filtering out non-impactful errors by Automatically Post-Editing (APE) the original translation based on each error, leaving only those errors that contribute to quality improvement. Specifically, we prompt the LLM to act as 1) $\textit{evaluator}$ to provide error annotations, 2) $\textit{post-editor}$ to determine whether errors impact quality improvement and 3) $\textit{pairwise quality verifier}$ as the error filter. Experiments show that our approach consistently improves both the reliability and quality of error spans against GEMBA-MQM, across eight LLMs in both high- and low-resource languages. Orthogonal to trained approaches, MQM-APE complements translation-specific evaluators such as Tower, highlighting its broad applicability. Further analysis confirm the effectiveness of each module and offer valuable insights into evaluator design and LLMs selection. The code will be released to facilitate the community.

Expected Error Minimization with Ultraconservative Update for SMT.

Transductive Minimum Error Rate Training for Statistical Machine Translation.

Locally Training the Log-Linear Model for SMT.

Discriminative Training for Log-Linear Based SMT: Global or Local Methods.

Softmax-margin Training for Statistical Machine Translation

Coordinate System Selection for Minimum Error Rate Training in Statistical Machine Translation

MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs

Optimizing Non-Decomposable Evaluation Metrics for Neural Machine Translation

Improving Translation Quality Estimation with Bias Mitigation

Addressing Exposure Bias With Document Minimum Risk Training: Cambridge at the WMT20 Biomedical Translation Task

Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization

Theoretical Investigations and Practical Enhancements on Tail Task Risk Minimization in Meta Learning

BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training

Bridging the Gap Between Training and Inference: Multi-Candidate Optimization for Diverse Neural Machine Translation.

Discriminative Language Model Adaptation for Mandarin Broadcast Speech Transcription and Translation

MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators

Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation

Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translation

Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation

Multi-Granularity Optimization for Non-Autoregressive Translation.

Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization