Abstract:<p>Method comparison studies are essential for development in medical and clinical fields. These studies often compare a cheaper, faster, or less invasive measuring method with a widely used one to see if they have sufficient agreement for interchangeable use. Moreover, unlike simply reading measurements from devices, eg, reading body temperature from a thermometer, the response measurement in many clinical and medical assessments is impacted not only by the measuring device but also by the rater. For example, widespread inconsistencies are commonly observed among raters in psychological or cognitive assessment studies due to different characteristics such as rater training and experience, especially in large‐scale assessment studies when many raters are employed. This paper proposes a model‐based approach to assess agreement of two measuring methods for paired repeated binary measurements under the scenario where the agreement between two measuring methods and the agreement among raters are required to be studied simultaneously. Based upon the generalized linear mixed models (GLMMs), the decision on the adequacy of interchangeable use is made by testing the equality of fixed effects of methods. Approaches for assessing method agreement, such as the Bland‐Altman diagram and Cohen's kappa, are also developed for repeated binary measurements based upon the latent variables in GLMMs. We assess our novel model‐based approach by simulation studies and a real clinical application, in which patients are evaluated repeatedly for delirium with two validated screening methods. Both the simulation studies and the real data analyses demonstrate that our proposed approach can effectively assess method agreement.</p>

Assessing Interrater Agreement via the Average Deviation Index Given a Variety of Theoretical and Methodological Problems

Measures of interrater agreement for quantitative data

Interrater agreement statistics under the two-rater dichotomous-response case with correlated decisions

Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

Choice of agreement indices for assessing and improving measurement reproducibility in a core laboratory setting

An ordinal measure of interrater absolute agreement

Assessing method agreement for paired repeated binary measurements administered by multiple raters

Interrater reliability for multilevel data: A generalizability theory approach.

High Agreement and High Prevalence: The Paradox of Cohen’s Kappa

Liberal-Conservative Hierarchies of Intercoder Reliability Estimators

Theory and application of absolute discrepancy in experimental designs

Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements

Measuring intergroup agreement and disagreement

Measures of Agreement with Multiple Raters: Fréchet Variances and Inference

Answering the Call for a Standard Reliability Measure for Coding Data

Assessing the point at which averages are stable: A tool illustrated in the context of person perception

Statistical models for assessing agreement for quantitative data with heterogeneous random raters and replicate measurements

Interrater agreement on employees' job performance: review and directions

Adjusting for Chance Clustering Comparison Measures