Abstract:The agreement intra-class correlation coefficient (ICCa) is a suitable statistical index for inter-rater reliability studies. With balanced Gaussian data, we prove the explicit form of ICCa asymptotic normality (ASN), valid both with analysis of variance (ANOVA), maximum likelihood (ML), or restricted ML (REML) estimates. An asymptotic confidence interval is then derived and its performances are examined by simulation compared to the most commonly used methods, under small, moderate and large sample size designs. Then, we deduce sample size calculation formulas, for the number of subjects and observers needed, to achieve a desired confidence interval width or an acceptable ICCa value test power and give concrete examples of their use. Finally, we propose a likelihood ratio test (LRT) to compare two ICCa's from two distinct subpopulations of patients (or raters) and study by simulation its first order risk and power properties. These methods are illustrated using data from two inter-rater reliability studies, one in physiotherapy with 42 patients and 10 raters and the second in neonatology with 80 subjects and 14 raters. In conclusion, we made recommendations to employ the proposed confidence interval for medium to large samples combined with the quantification of the minimal required sample size at the planning step, or the posterior-power at the analysis step, using simple dedicated formulas. Furthermore, with sufficient sizes, the proposed LRT seems suitable to compare inter-rater reliability between two patient subpopulations. Used wisely, this proposed methods toolbox can remedy common current issues in inter-rater reliability studies.

Computing Information Agreement

High Agreement and High Prevalence: The Paradox of Cohen’s Kappa

Measuring Inter-group Agreement on zSlice Based General Type-2 Fuzzy Sets

Min-Mid-Max Scaling, Limits of Agreement, and Agreement Score

Measures of interrater agreement for quantitative data

Measuring agreement on linguistic expressions in medical treatment scenarios

Asymptotic Confidence Interval, Sample Size Formulas and Comparison Test for the Agreement Intra-Class Correlation Coefficient in Inter-Rater Reliability Studies

Fuzzy Kappa for the Agreement Measure of Fuzzy Classifications

Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

Do they agree? Bibliometric evaluation vs informed peer review in the Italian research assessment exercise

Interrater agreement statistics under the two-rater dichotomous-response case with correlated decisions

Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements

Assessing agreement on classification tasks: the kappa statistic

Measuring intergroup agreement and disagreement

Assessing agreement between two measurement systems: An alternative to the limits of agreement approach

Choice of agreement indices for assessing and improving measurement reproducibility in a core laboratory setting

A Unified Information Criterion For Evaluating Probe And Test Selection

Measuring Error Alignment for Decision-Making Systems

Unique Information and Secret Key Agreement

Similarity Measure for Aggregated Fuzzy Numbers from Interval-Valued Data

Statistical inference for agreement between multiple raters on a binary scale