Interrater reliability: the kappa statistic

Marry L. McHugh

DOI: https://doi.org/10.11613/BM.2012.031

2019-10-07

Biochemia Medica

Abstract:Lessons in biostatistics by Marry L. McHugh

medical laboratory technology

What problem does this paper attempt to address?

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

Ming Li,Qian Gao,Tianfei Yu

DOI: https://doi.org/10.1186/s12885-023-11325-z

IF: 4.638

2023-08-25

BMC Cancer

Abstract:Background: In research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses' statistical power for hypothesis testing. Methods: This article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved. Results: The Cohen's Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure. Conclusion: Despite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research.
Assessing agreement on classification tasks: the kappa statistic

Jean Carletta

DOI: https://doi.org/10.48550/arXiv.cmp-lg/9602004

1996-02-27

Abstract:Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa statistic. We discuss what is wrong with reliability measures as they are currently used for discourse and dialogue work in computational linguistics and cognitive science, and argue that we would be better off as a field adopting techniques from content analysis.

Computation and Language
Measuring agreement among several raters classifying subjects into one-or-more (hierarchical) nominal categories. A generalisation of Fleiss' kappa

Filip Moons,Ellen Vandervieren

2023-03-22

Abstract:Cohen's and Fleiss' kappa are well-known measures for inter-rater reliability. However, they only allow a rater to select exactly one category for each subject. This is a severe limitation in some research contexts: for example, measuring the inter-rater reliability of a group of psychiatrists diagnosing patients into multiple disorders is impossible with these measures. This paper proposes a generalisation of the Fleiss' kappa coefficient that lifts this limitation. Specifically, the proposed $\kappa$ statistic measures inter-rater reliability between multiple raters classifying subjects into one-or-more nominal categories. These categories can be weighted according to their importance, and the measure can take into account the category hierarchy (e.g., categories consisting of subcategories that are only available when choosing the main category like a primary psychiatric disorder and sub-disorders; but much more complex dependencies between categories are possible as well). The proposed $\kappa$ statistic can handle missing data and a varying number of raters for subjects or categories. The paper briefly overviews existing methods allowing raters to classify subjects into multiple categories. Next, we derive our proposed measure step-by-step and prove that the proposed measure equals Fleiss' kappa when a fixed number of raters chose one category for each subject. The measure was developed to investigate the reliability of a new mathematics assessment method, of which an example is elaborated. The paper concludes with the worked-out example of psychiatrists diagnosing patients into multiple disorders.

Methodology,Statistics Theory
Interrater reliability for multilevel data: A generalizability theory approach.

Debby ten Hove,Terrence D. Jorgensen,L. Andries van der Ark

DOI: https://doi.org/10.1037/met0000391

IF: 10.929

2021-04-05

Psychological Methods

Abstract:Current interrater reliability (IRR) coefficients ignore the nested structure of multilevel observational data, resulting in biased estimates of both subject- and cluster-level IRR. We used generalizability theory to provide a conceptualization and estimation method for IRR of continuous multilevel observational data. We explain how generalizability theory decomposes the variance of multilevel observational data into subject-, cluster-, and rater-related components, which can be estimated using Markov chain Monte Carlo (MCMC) estimation. We explain how IRR coefficients for each level can be derived from these variance components, and how they can be estimated as intraclass correlation coefficients (ICC). We assessed the quality of MCMC point and interval estimates with a simulation study, and showed that small numbers of raters were the main source of bias and inefficiency of the ICCs. In a follow-up simulation, we showed that a planned missing data design can diminish most estimation difficulties in these conditions, yielding a useful approach to estimating multilevel interrater reliability for most social and behavioral research. We illustrated the method using data on student–teacher relationships. All software code and data used for this article is available on the Open Science Framework: https://osf.io/bwk5t/. (PsycInfo Database Record (c) 2022 APA, all rights reserved)

psychology, multidisciplinary
Interrater agreement statistics under the two-rater dichotomous-response case with correlated decisions

Zizhong Tian,Vernon M. Chinchilli,Chan Shen,Shouhao Zhou

DOI: https://doi.org/10.48550/arXiv.2402.08069

2024-02-13

Abstract:Measurement of the interrater agreement (IRA) is critical in various disciplines. To correct for potential confounding chance agreement in IRA, Cohen's kappa and many other methods have been proposed. However, owing to the varied strategies and assumptions across these methods, there is a lack of practical guidelines on how these methods should be preferred even for the common two-rater dichotomous rating. To fill the gaps in the literature, we systematically review nine IRA methods and propose a generalized framework that can simulate the correlated decision processes behind the two raters to compare those reviewed methods under comprehensive practical scenarios. Based on the new framework, an estimand of "true" chance-corrected IRA is defined by accounting for the "probabilistic certainty" and serves as the comparison benchmark. We carry out extensive simulations to evaluate the performance of the reviewed IRA measures, and an agglomerative hierarchical clustering analysis is conducted to assess the inter-relationships among the included methods and the benchmark metric. Recommendations for selecting appropriate IRA statistics in different practical conditions are provided and the needs for further advancements in IRA estimation methodologies are emphasized.

Methodology
Evaluating inter-rater reliability in the context of "Sysmex UN2000 detection of protein/creatinine ratio and of renal tubular epithelial cells can be used for screening lupus nephritis": a statistical examination

Ming Li,Qian Gao,Jing Yang,Tianfei Yu

DOI: https://doi.org/10.1186/s12882-024-03540-y

2024-03-15

BMC Nephrology

Abstract:The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses.

urology & nephrology
Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements

A. Martín Andrés,M. Álvarez Hernández

DOI: https://doi.org/10.1007/s11634-024-00581-x

2024-03-08

Advances in Data Analysis and Classification

Abstract:To measure the degree of agreement between R observers who independently classify n subjects within K categories, various kappa -type coefficients are often used. When R = 2, it is common to use the Cohen' kappa , Scott's pi , Gwet's AC1/2 , and Krippendorf's alpha coefficients (weighted or not). When R > 2, some pairwise version based on the aforementioned coefficients is normally used; with the same order as above: Hubert's kappa , Fleiss's kappa , Gwet's AC1/2, and Krippendorf's alpha . However, all these statistics are based on biased estimators of the expected index of agreements, since they estimate the product of two population proportions through the product of their sample estimators. The aims of this article are three. First, to provide statistics based on unbiased estimators of the expected index of agreements and determine their variance based on the variance of the original statistic. Second, to make pairwise extensions of some measures. And third, to show that the old and new estimators of the Cohen's kappa and Hubert's kappa coefficients match the well-known estimators of concordance and intraclass correlation coefficients, if the former are defined by assuming quadratic weights. The article shows that the new estimators are always greater than or equal the classic ones, except for the case of Gwet where it is the other way around, although these differences are only relevant with small sample sizes (e.g. n ≤ 30).

statistics & probability
Reliability of Measurement and the Methods of Estimating Reliability

Yi-biao Zhou,Gen-ming Zhao

DOI: https://doi.org/10.3760/j.issn:0254-6450.2003.12.022

2003-01-01

Abstract:OBJECTIVE:To explore the methods of estimating measurement reliability and their limitations.METHODS:According to the given examples, reliabilities of measurement were calculated and limitations of the methods of estimating reliability were analyzed.RESULTS:The Kappa value of interobserver reliability was 0.793 between two pathologists. Between the two populations with different prevalence rates, the values of Kappa were 0.800 and 0.137 respectively, and with big difference. Cronbach's alpha coefficient of compositive index for A type behavior was 0.55.CONCLUSION:The Kappa index and alpha coefficient were both inherently population-specific. Before generalizing to different populations, the reliability needs to be measured.
Regard to Assessing Agreement Between Two Raters with Kappa Statistics

Tianfei Yu,Bingrui Ren,Ming Li

DOI: https://doi.org/10.1016/j.ijcard.2024.131896

IF: 4.039

2024-01-01

International Journal of Cardiology

Abstract:Paratz et al.'s [1] investigation of 380 hypertrophic cardiomyopathy (HCM) patients unveiled that cardiac arrest primarily affected younger individuals exhibiting unexplained syncope, non-sustained ventricular tachycardia, increased septal thickness, and sarcomeric gene mutations. Risk-sudden cardiac death (SCD) and American Heart Association/American College of Cardiology (AHA/ACC) scores demonstrated poor agreement (Cohen's kappa coefficient 0.38). The Risk-SCD score exhibited low sensitivity but high specificity, while the AHA/ACC risk score lacked discriminatory power.
Answering the Call for a Standard Reliability Measure for Coding Data

Andrew F. Hayes,Klaus Krippendorff

DOI: https://doi.org/10.1080/19312450709336664

IF: 8.044

2007-04-01

Communication Methods and Measures

Abstract:In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.

communication
Intercoder Reliability for Use in Qualitative Research and Evaluation

Monica L. Coleman,Moira Ragan,Tahani Dari

DOI: https://doi.org/10.1080/07481756.2024.2303715

2024-03-02

Measurement and Evaluation in Counseling and Development

Abstract:Intercoder reliability can increase trustworthiness, accuracy, rigor, collaboration, and power sharing in qualitative research. Though not every qualitative design can utilize intercoder reliability, this article highlights how positivist qualitative research, community-based participatory research, and participatory evaluation all strengthen when intercoder reliability is sought among coding teams. Basic calculations for intercoder reliability can be completed manually or with software, making it an accessible metric to inform counseling research and evaluation, benefitting counselor education and practice. Interpreting metrics like percent agreement or kappa statistic largely depends on data context and potential impact of the study, which warrants researchers and evaluators to assess and understand the consequences of their work. Using ICR with clinical data such as interviews and other qualitative sources can be utilized to teach trainees the method in counseling research and evaluation and help make visible the important contributions of qualitative research using collaborative teams in counselor education. Intercoder reliability is a method in qualitative coding analysis that helps multiple coders increase accuracy in their results based on agreement in their coding. The method is particularly important in qualitative research and evaluation like positivist qualitative research, community-based participatory research, and participatory evaluation, where rigor, power sharing, and participation, respectively, are valued among the research teams.

psychology, educational, applied
Liberal-Conservative Hierarchies of Intercoder Reliability Estimators

Yingjie Jay Zhao,Guangchao Charles Feng,Dianshi Moses Li,Song Harris Ao,Ming Milano Li,Zhan Thor Tuo,Hui Huang,Xinshu Zhao

2024-10-03

Abstract:While numerous indices of inter-coder reliability exist, Krippendorff's {\alpha} and Cohen's \{kappa} have long dominated in communication studies and other fields, respectively. The near consensus, however, may be near the end. Recent theoretical and mathematical analyses reveal that these indices assume intentional and maximal random coding, leading to paradoxes and inaccuracies. A controlled experiment with one-way golden standard and Monte Carlo simulations supports these findings, showing that \{kappa} and {\alpha} are poor predictors and approximators of true intercoder reliability. As consensus on a perfect index remains elusive, more authors recommend selecting the best available index for specific situations (BAFS). To make informed choices, researchers, reviewers, and educators need to understand the liberal-conservative hierarchy of indices, i.e., which indices produce higher or lower scores. This study extends previous efforts by expanding the math-based hierarchies to include 23 indices and constructing six additional hierarchies using Monte Carlo simulations. These simulations account for factors like the number of categories and distribution skew. The resulting eight hierarchies display a consistent pattern and reveal a previously undetected paradox in the Ir index.

Physics and Society
The Kappa Paradox Explained

Bastiaan M Derksen,Wendy Bruinsma,Johan Carel Goslings,Niels W L Schep,Bastiaan M. Derksen,Niels W.L. Schep

DOI: https://doi.org/10.1016/j.jhsa.2024.01.006

IF: 2.342

2024-02-19

The Journal Of Hand Surgery

Abstract:Observer reliability studies for fracture classification systems evaluate agreement using Cohen's κ and absolute agreement as outcome measures. Cohen's κ is a chance-corrected measure of agreement and can range between 0 (no agreement) and 1 (perfect agreement). Absolute agreement is the percentage of times observers agree on the matter they have to rate. Some studies report a high-absolute agreement but a relatively low κ value, which is counterintuitive. This phenomenon is referred to as the Kappa Paradox. The objective of this article was to explain the statistical phenomenon of the Kappa Paradox and to help readers and researchers to recognize and prevent this phenomenon.

surgery,orthopedics
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

Ka Wong,Praveen Paritosh

DOI: https://doi.org/10.48550/arXiv.2203.12913

2022-03-24

Abstract:Since the inception of crowdsourcing, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, many natural language processing (NLP) applications that rely on aggregate ratings only report the reliability of individual ratings, which is the incorrect unit of analysis. In these instances, the data reliability is under-reported, and a proposed k-rater reliability (kRR) should be used as the correct data reliability for aggregated datasets. It is a multi-rater generalization of inter-rater reliability (IRR). We conducted two replications of the WordSim-353 benchmark, and present empirical, analytical, and bootstrap-based methods for computing kRR on WordSim-353. These methods produce very similar results. We hope this discussion will nudge researchers to report kRR in addition to IRR.

Artificial Intelligence,Machine Learning
Learning more from the inter-rater reliability of interstitial fibrosis assessment beyond just a statistic

Peir-In Liang,Wei-Chou Lin,Mei-Chin Wen,Shun-Chen Huang,Pei-Wei Fang,Hao-Wen Chuang,Yi-Jia Lin,Hui-Ping Chien,Huan-Da Chen,Tai-Di Chen

DOI: https://doi.org/10.1038/s41598-023-40221-6

IF: 4.6

2023-08-17

Scientific Reports

Abstract:Interstitial fibrosis assessment by renal pathologists lacks good agreement, and we aimed to investigate its hidden properties and infer possible clinical impact. Fifty kidney biopsies were assessed by 9 renal pathologists and evaluated by intraclass correlation coefficients (ICCs) and kappa statistics. Probabilities of pathologists' assessments that would deviate far from true values were derived from quadratic regression and multilayer perceptron nonlinear regression. Likely causes of variation in interstitial fibrosis assessment were investigated. Possible misclassification rates were inferred on reported large cohorts. We found inter-rater reliabilities ranged from poor to good (ICCs 0.48 to 0.90), and pathologists' assessments had the worst agreements when the extent of interstitial fibrosis was moderate. 33.5% of pathologists' assessments were expected to deviate far from the true values. Variation in interstitial fibrosis assessment was found to be correlated with variation in interstitial inflammation assessment (r 2 = 32.1%). Taking IgA nephropathy as an example, the Oxford T scores for interstitial fibrosis were expected to be misclassified in 21.9% of patients. This study demonstrated the complexity of the inter-rater reliability of interstitial fibrosis assessment, and our proposed approaches discovered previously unknown properties in pathologists' practice and inferred a possible clinical impact on patients.

multidisciplinary sciences
Measures of interrater agreement for quantitative data

Daniela Marella,Giuseppe Bove,Marella, Daniela,Bove, Giuseppe

DOI: https://doi.org/10.1007/s10182-023-00483-x

2023-10-11

AStA Advances in Statistical Analysis

Abstract:In this paper measures of interrater absolute agreement for quantitative measurements based on the standard deviation are proposed. Such indices allow (i) to overcome the limits affecting the intraclass correlation index; (ii) to measure the interrater agreement on single targets. Estimators of the proposed measures are introduced and their sampling properties are investigated for normal and non-normal data. Simulated data are employed to demonstrate the accuracy and practical utility of the new indices for assessing agreement. Finally, an application to assess the consistency of measurements performed by radiologists evaluating tumor size of lung cancer is presented.

statistics & probability
High Agreement and High Prevalence: The Paradox of Cohen’s Kappa

Slavica Zec,Nicola Soriani,Rosanna Comoretto,Ileana Baldi

DOI: https://doi.org/10.2174/1874434601711010211

2017-10-31

The Open Nursing Journal

Abstract:Background: Cohen's Kappa is the most used agreement statistic in literature. However, under certain conditions, it is affected by a paradox which returns biased estimates of the statistic itself. Objective: The aim of the study is to provide sufficient information which allows the reader to make an informed choice of the correct agreement measure, by underlining some optimal properties of Gwet’s AC1 in comparison to Cohen’s Kappa, using a real data example. Method: During the process of literature review, we have asked a panel of three evaluators to come up with a judgment on the quality of 57 randomized controlled trials assigning a score to each trial using the Jadad scale. The quality was evaluated according to the following dimensions: adopted design, randomization unit, type of primary endpoint. With respect to each of the above described features, the agreement between the three evaluators has been calculated using Cohen’s Kappa statistic and Gwet’s AC1 statistic and, finally, the values have been compared with the observed agreement. Results: The values of the Cohen’s Kappa statistic would lead to believe that the agreement levels for the variables Unit, Design and Primary Endpoints are totally unsatisfactory. The AC1 statistic, on the contrary, shows plausible values which are in line with the respective values of the observed concordance. Conclusion: We conclude that it would always be appropriate to adopt the AC1 statistic, thus bypassing any risk of incurring the paradox and drawing wrong conclusions about the results of agreement analysis.
Intraclass correlation – A discussion and demonstration of basic features

David Liljequist,Britt Elfving,Kirsti Skavberg Roaldsen

DOI: https://doi.org/10.1371/journal.pone.0219854

IF: 3.7

2019-07-22

PLoS ONE

Abstract:A re-analysis of intraclass correlation (ICC) theory is presented together with Monte Carlo simulations of ICC probability distributions. A partly revised and simplified theory of the single-score ICC is obtained, together with an alternative and simple recipe for its use in reliability studies. Our main, practical conclusion is that in the analysis of a reliability study it is neither necessary nor convenient to start from an initial choice of a specified statistical model. Rather, one may impartially use all three single-score ICC formulas. A near equality of the three ICC values indicates the absence of bias (systematic error), in which case the classical (one-way random) ICC may be used. A consistency ICC larger than absolute agreement ICC indicates the presence of non-negligible bias; if so, classical ICC is invalid and misleading. An F-test may be used to confirm whether biases are present. From the resulting model (without or with bias) variances and confidence intervals may then be calculated. In presence of bias, both absolute agreement ICC and consistency ICC should be reported, since they give different and complementary information about the reliability of the method. A clinical example with data from the literature is given.

multidisciplinary sciences
Toward improved inference for Krippendorff's Alpha agreement coefficient

John Hughes

DOI: https://doi.org/10.1016/j.jspi.2024.106170

IF: 1.095

2024-04-08

Journal of Statistical Planning and Inference

Abstract:In this article I recommend a better point estimator for Krippendorff's Alpha agreement coefficient, and develop a jackknife variance estimator that leads to much better interval estimation than does the customary bootstrap procedure or an alternative bootstrap procedure. Having developed the new methodology, I analyze nominal data previously analyzed by Krippendorff, and two experimentally observed datasets: (1) ordinal data from an imaging study of congenital diaphragmatic hernia, and (2) United States Environmental Protection Agency air pollution data for the Philadelphia, Pennsylvania area. The latter two applications are novel. The proposed methodology is now supported in version 2.0 of my open source R package, krippendorffsalpha , which supports common and user-defined distance functions, and can accommodate any number of units, any number of coders, and missingness. Interval computation can be parallelized.

statistics & probability

Interrater reliability: the kappa statistic

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

Assessing agreement on classification tasks: the kappa statistic

Measuring agreement among several raters classifying subjects into one-or-more (hierarchical) nominal categories. A generalisation of Fleiss' kappa

Interrater reliability for multilevel data: A generalizability theory approach.

Interrater agreement statistics under the two-rater dichotomous-response case with correlated decisions

Evaluating inter-rater reliability in the context of "Sysmex UN2000 detection of protein/creatinine ratio and of renal tubular epithelial cells can be used for screening lupus nephritis": a statistical examination

Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements

Reliability of Measurement and the Methods of Estimating Reliability

Regard to Assessing Agreement Between Two Raters with Kappa Statistics

Answering the Call for a Standard Reliability Measure for Coding Data

Intercoder Reliability for Use in Qualitative Research and Evaluation

Liberal-Conservative Hierarchies of Intercoder Reliability Estimators

The Kappa Paradox Explained

k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

Learning more from the inter-rater reliability of interstitial fibrosis assessment beyond just a statistic

Measures of interrater agreement for quantitative data

High Agreement and High Prevalence: The Paradox of Cohen’s Kappa

Intraclass correlation – A discussion and demonstration of basic features

Toward improved inference for Krippendorff's Alpha agreement coefficient