Abstract:The effectiveness of automatic evaluation of generative models is typically measured by comparing it to human evaluation using correlation metrics. However, metrics like Krippendorff's $\alpha$ and Randolph's $\kappa$, originally designed to measure the reliability of human labeling, make assumptions about human behavior and the labeling process. In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human behavior and automatic evaluation methods, including LLM-as-a-Judge. Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human labels (gathered during human evaluation) is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to human-to-human (HH) correlation. This can create the misleading impression that automatic evaluation is accurate enough to approximate the human majority label. However, as the proportion of samples with consistent human labels increases, the correlation between machine labels and human majority labels declines, falling below HH correlation. Based on these findings, we first propose stratifying results by human label uncertainty to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric - *binned Jensen-Shannon Divergence for perception* for such scenarios to better measure the effectiveness of automatic evaluations. Third, we present visualization techniques -- *perception charts*, to compare the strengths and limitations of automatic evaluation and to contextualize correlation measures appropriately

On the significance of the measurement evaluation method in decision making: A synthesis of JCGM 101 and JCGM 106 applied to gross alpha and gross beta measurements using LSC

Uncertainty analysis for decision-making in river basin management

On a Monte Carlo method for measurement uncertainty evaluation and its implementation

Getting started with uncertainty evaluation using the Monte Carlo method in R

Application of Monte Carlo Method in the Measurement Process Design According to Next Generation of Gps

A self-adaptive method for the assessment of dynamic measurement uncertainty

Modeling the systematic behavior at the micro and nano length scale

Understanding the implications of low knowledge and high uncertainty in risk studies

A study on mutative scale straightness measurement based on uncertainty analysis

Fundamentals in Measurement: The Role of Measurement Uncertainty in Conformity Assessment: Some Examples

The use of a Monte Carlo method for uncertainty calculation, with an application to the measurement of neutron ambient dose equivalent rate

Mathematical modeling applied to the uncertainty analysis of a tank prover calibration: Understanding the influence of calibration conditions on the GUM validation using the Monte Carlo method

Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge

Comparison of MCM and GUM Method for Evaluating Measurement Uncertainty of Wind Speed by Pitot Tube

Uncertainty-based measurement quality control

Uncertainty Evaluation on Temperature Detection of Middle Atmosphere by Rayleigh Lidar

Monte Carlo uncertainty estimation for wavelength calibration

Estimation and usefulness of measurement uncertainty from sampling at different spatial scales: microns to kilometres

A brief guide to measurement uncertainty (IUPAC Technical Report)

Application of the adaptive Monte Carlo method for uncertainty evaluation in the determination of total testosterone in human serum by triple isotope dilution mass spectrometry

Conformity assessment of processes and lots in the framework of JCGM 106:2012