Abstract:Recently, it was shown that most popular IR measures are not interval-scaled, implying that decades of experimental IR research used potentially improper methods, which may have produced questionable results. However, it was unclear if and to what extent these findings apply to actual evaluations and this opened a debate in the community with researchers standing on opposite positions about whether this should be considered an issue (or not) and to what extent. In this paper, we first give an introduction to the representational measurement theory explaining why certain operations and significance tests are permissible only with scales of a certain level. For that, we introduce the notion of meaningfulness specifying the conditions under which the truth (or falsity) of a statement is invariant under permissible transformations of a scale. Furthermore, we show how the recall base and the length of the run may make comparison and aggregation across topics problematic. Then we propose a straightforward and powerful approach for turning an evaluation measure into an interval scale, and describe an experimental evaluation of the differences between using the original measures and the interval-scaled ones. For all the regarded measures - namely Precision, Recall, Average Precision, (Normalized) Discounted Cumulative Gain, Rank-Biased Precision and Reciprocal Rank - we observe substantial effects, both on the order of average values and on the outcome of significance tests. For the latter, previously significant differences turn out to be insignificant, while insignificant ones become significant. The effect varies remarkably between the tests considered but overall, on average, we observed a 25% change in the decision about which systems are significantly different and which are not.

Short Communication: Utility-Based Acceptability Indices

Utility-based acceptability indices

Dynamic Coherent Acceptability Indices and their Applications to Finance

Acceptability maximization

Star-shaped acceptability indexes

Generalized Optimized Certainty Equivalent with Applications in the Rank-Dependent Utility Model

Dynamic Assessment Indices

A new view on risk measures associated with acceptance sets

Efficiency in Pure-Exchange Economies with Risk-Averse Monetary Utilities

Utility Maximization and Indifference Value under Risk and Information Constraints for a Market with a Change Point

Utility Indifference Pricing: A Time Consistent Approach

Analytical-solution-based interval fuzzy utility acquisition and decision making with interval-valued fuzzy preference relations based on additive consistency

Inducing a probability distribution in Stochastic Multicriteria Acceptability Analysis

Dynamic quasi-concave performance measures

A risk index for ordinal variables and its statistical properties: A priority of intervention indicator in quality control framework

Parameterized utility functions on interval-valued intuitionistic fuzzy numbers with two kinds of entropy and their application in multi-criteria decision making

Risk Measures beyond Frictionless Markets

Computing strategies for achieving acceptability

Equivalence of inequality indices: Three dimensions of impact revisited

Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales