Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translation

Zheng Wei Lim,Ekaterina Vylomova,Trevor Cohn,Charles Kemp

2024-06-10

Abstract:A good translation should be faithful to the source and should respect the norms of the target language. We address a theoretical puzzle about the relationship between these objectives. On one hand, intuition and some prior work suggest that accuracy and fluency should trade off against each other, and that capturing every detail of the source can only be achieved at the cost of fluency. On the other hand, quality assessment researchers often suggest that accuracy and fluency are highly correlated and difficult for human raters to distinguish (Callison-Burch et al., 2007). We show that the tension between these views is an instance of Simpson's paradox, and that accuracy and fluency are positively correlated at the level of the corpus but trade off at the level of individual source segments. We further suggest that the relationship between accuracy and fluency is best evaluated at the segment (or sentence) level, and that the trade off between these dimensions has implications both for assessing translation quality and developing improved MT systems.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the relationship between accuracy and fluency in the translation process and their trade - off. Specifically, the author explores the following points:

1. **Trade - off between accuracy and fluency**: On the one hand, intuition and some previous studies suggest that there is a trade - off between accuracy and fluency, that is, when pursuing the complete retention of source - language information, the fluency of the target language may be sacrificed. On the other hand, quality assessment researchers often believe that accuracy and fluency are highly correlated and difficult to distinguish (Callison - Burch et al., 2007).

2. **Application of Simpson’s Paradox**: The author points out that the conflict between these two views is actually an instance of Simpson’s Paradox. Simpson’s Paradox refers to the situation where, in some cases, when data is analyzed at different levels, opposite correlations may occur. Specifically, at the corpus level, accuracy and fluency are positively correlated; while at the sentence or segment level, they are negatively correlated.

3. **Implications for evaluating translation quality and improving machine translation systems**: Based on the above findings, the author suggests that when evaluating translation quality and developing improved machine translation systems, more attention should be paid to the trade - off relationship between accuracy and fluency at the sentence or segment level.

### Main conclusions

- Accuracy and fluency are negatively correlated at the sentence or segment level, but positively correlated at the corpus level.
- This phenomenon can be explained by Simpson’s Paradox.
- In practical applications, especially when evaluating translation quality and improving machine translation systems, more attention should be paid to the trade - off between accuracy and fluency at the segment level.

### Formula representation

The probability models involved in the article can be represented by the following formulas:

- \(p(x|y)\) represents accuracy, that is, the conditional probability of the source language \(x\) given the target - language translation \(y\).
- \(p(y)\) represents fluency, that is, the prior probability of the target - language translation \(y\).
- \(p(y|x)\propto p(x|y)p(y)\) represents the joint probability, which is used to generate the translation \(y\).

These formulas are used in the article to simulate and verify the trade - off relationship between accuracy and fluency.