Abstract:From both human translators (HT) and machine translation (MT) researchers' point of view, translation quality evaluation (TQE) is an essential task. Translation service providers (TSPs) have to deliver large volumes of translations which meet customer specifications with harsh constraints of required quality level in tight time-frames and costs. MT researchers strive to make their models better, which also requires reliable quality evaluation. While automatic machine translation evaluation (MTE) metrics and quality estimation (QE) tools are widely available and easy to access, existing automated tools are not good enough, and human assessment from professional translators (HAP) are often chosen as the golden standard \cite{han-etal-2021-TQA}. Human evaluations, however, are often accused of having low reliability and agreement. Is this caused by subjectivity or statistics is at play? How to avoid the entire text to be checked and be more efficient with TQE from cost and efficiency perspectives, and what is the optimal sample size of the translated text, so as to reliably estimate the translation quality of the entire material? This work carries out such motivated research to correctly estimate the confidence intervals \cite{Brown_etal2001Interval} depending on the sample size of the translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA).

Improving the Confidence of Machine Translation Quality Estimates.

Coh-Metrix Model-Based Automatic Assessment of Interpreting Quality

Exploring Prediction Uncertainty in Machine Translation Quality Estimation

Uncertainty-Aware Machine Translation Evaluation

Practical Perspectives on Quality Estimation for Machine Translation

Evaluating Machine Translation Quality with Conformal Predictive Distributions

Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation

MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs

Measuring Uncertainty in Translation Quality Evaluation (TQE)

Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation

Can Automatic Metrics Assess High-Quality Translations?

Improving Back-Translation with Uncertainty-based Confidence Estimation

Quality Estimation & Interpretability for Code Translation

Unsupervised Quality Estimation for Neural Machine Translation

Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement

Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation

Quality Estimation of Machine Translated Texts based on Direct Evidence from Training Data

Self-Supervised Quality Estimation for Machine Translation.

Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation

Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model

Translation Error Detection as Rationale Extraction