Abstract:Text generation commonly relies on greedy and beam decoding that limit the search space and degrade output quality. Minimum Bayes Risk (MBR) decoding can mitigate this problem by utilizing automatic evaluation metrics and model-generated pseudo-references. Previous studies have conducted empirical analyses to reveal the improvement by MBR decoding, and reported various observations. However, despite these observations, the theoretical relationship between them remains uncertain. To address this, we present a novel theoretical interpretation of MBR decoding from the perspective of bias-diversity decomposition. We decompose errors in the estimated quality of generated hypotheses in MBR decoding into two key factors: bias, which reflects the closeness between utility functions and human evaluations, and diversity, which represents the variation in the estimated quality of utility functions. Our theoretical analysis reveals the difficulty in simultaneously improving both bias and diversity, and highlights the effectiveness of increasing diversity to enhance MBR decoding performance. This analysis verifies the alignment between our theoretical insights and the empirical results reported in previous work. Furthermore, to support our theoretical findings, we propose a new metric, pseudo-bias, which approximates the bias term using gold references. We also introduce a new MBR approach, Metric-augmented MBR (MAMBR), which increases diversity by adjusting the behavior of utility functions without altering the pseudo-references. Experimental results across multiple NLP tasks show that the decomposed terms in the bias-diversity decomposition correlate well with performance, and that MAMBR improves text generation quality by modifying utility function behavior. Our code will be available at <a class="link-external link-https" href="https://github.com/naist-nlp/mbr-bias-diversity" rel="external noopener nofollow">this https URL</a>.

Decoding and Diversity in Machine Translation

A Simple, Fast Diverse Decoding Algorithm for Neural Generation

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Lexical Diversity in Statistical and Neural Machine Translation

Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation

Diversity by Phonetics and its Application in Neural Machine Translation

Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation

Exploring Diversity in Back Translation for Low-Resource Machine Translation

Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation

Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem

Diverse Machine Translation with Translation Memory

A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation

Generating Gender Alternatives in Machine Translation

Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation

Bridging the Gap Between Training and Inference: Multi-Candidate Optimization for Diverse Neural Machine Translation.

Gender Bias in Machine Translation

Good, but not always Fair: An Evaluation of Gender Bias for three Commercial Machine Translation Systems

Theoretical Aspects of Bias and Diversity in Minimum Bayes Risk Decoding

Getting Gender Right in Neural Machine Translation

Data Diversification: A Simple Strategy For Neural Machine Translation

What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns