Cohen's Kappa

• Your categories are ordered, so near misses should count for something Weighted Kappa →
• Three or more raters each coded every case Fleiss' Kappa →
• Three or more raters, with some codes missing or different numbers of raters per case Krippendorff's Alpha →
• Your ratings are numbers, not categories Intraclass Correlation Coefficient (ICC) →
• You're checking codes against a gold standard Sensitivity and Specificity →
• You're asking whether one rater says yes more often McNemar's Test →
• You're asking whether two different variables are related Cramér's V →
• Your items are meant to add up to one score Cronbach's Alpha →

• Don'tReport percent agreement alone as inter-rater reliability.

  InsteadReport κ alongside the percent agreement and the table, so readers can compare the agreement with what chance alone would produce (McHugh, 2012).

• Don'tTreat a significant test of κ against zero as evidence of good agreement.

  InsteadJudge the size of κ and its interval. The test only rules out agreement at chance level, a very low bar.

• Don'tCompare kappas from samples where the categories are much more or less common.

  InsteadCompare them only with similar base rates, and report agreement for each category too (Feinstein & Cicchetti, 1990).

• Don'tUse Cohen's kappa for ordered categories or for three or more raters.

  InsteadUse weighted kappa for ordered codes (Cohen, 1968) and, for three or more raters, Fleiss' kappa (Fleiss, 1971) when every case gets the same number of codes, or Krippendorff's alpha when some are missing.

• Don'tRead κ = .67 as “the raters agreed on 67% of cases.”

  InsteadRead it as the share of possible agreement beyond chance, and give the percent agreement separately.

• Don'tRead a p-value as the chance that the null hypothesis is true.

  InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Equation Form

Table 1. Agreement by Category

Full Text (APA Style)

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

What Your Data Should Look Like

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the intake-coding study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

CROSSTABS
  /TABLES=clinician_a BY clinician_b
  /STATISTICS=KAPPA
  /CELLS=COUNT.

library(psych)

intake <- read.csv("data.csv")
table(intake$clinician_a, intake$clinician_b)

ck <- cohen.kappa(intake[, c("clinician_a", "clinician_b")])
ck
sqrt(ck$var.kappa)   # Fleiss, Cohen & Everitt's SE
ck$confid[1, ]

             anxiety depression neither
  anxiety         14          2       4
  depression       3         19       3
  neither          4          4      47

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels, 
    w.exp = w.exp)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries 
                 lower estimate upper
unweighted kappa  0.54     0.67  0.79
weighted kappa    0.49     0.65  0.82

 Number of subjects = 100 
[1] 0.06567259
    lower  estimate     upper 
0.5371153 0.6658312 0.7945472 

import pandas as pd
from sklearn.metrics import cohen_kappa_score
from statsmodels.stats.inter_rater import cohens_kappa

intake = pd.read_csv("data.csv")
cohen_kappa_score(intake["clinician_a"], intake["clinician_b"])   # 0.6658

table = pd.crosstab(intake["clinician_a"], intake["clinician_b"])
table
# clinician_b  anxiety  depression  neither
# clinician_a
# anxiety           14           2        4
# depression         3          19        3
# neither            4           4       47

result = cohens_kappa(table)
print(result)

                  Simple Kappa Coefficient
              --------------------------------
              Kappa                     0.6658
              ASE                       0.0657
              95% Lower Conf Limit      0.5371
              95% Upper Conf Limit      0.7945

                 Test of H0: Simple Kappa = 0

              ASE under H0              0.0729
              Z                         9.1302
              One-sided Pr >  Z         0.0000
              Two-sided Pr > |Z|        0.0000

None needed. This test answers a single question, so there are no planned contrasts or post hoc tests to run.

• Related DesignIntraclass Correlation Coefficient (ICC): The same question for numeric ratings. Landis and Koch (1977) note that κ is directly analogous to the intraclass correlation from an ANOVA model of quantitative ratings.
• Related DesignMcNemar's Test: Both work on a table of the same cases rated twice, but they read different parts of it. McNemar's test compares the two disagreement cells to ask whether one rater (or occasion) says yes more often; kappa uses the agreement cells to ask how often the two agree. Raters can use yes equally often and still disagree on many cases.
• Related DesignCramér's V: Both summarize a square table, but V measures any association, while κ counts only matching codes. Two raters who consistently traded labels, so that A's depression was always B's anxiety, A's anxiety B's neither and A's neither B's depression, would never match: V would be large and κ negative.
• Related DesignFleiss' Kappa: The same question, agreement beyond chance as a share of what was possible, for three or more raters. With a panel of raters coding the same cases, use it in place of Cohen's kappa.
• Related DesignKrippendorff's Alpha: Krippendorff's alpha asks a similar question for any number of coders, with missing codes allowed and a choice of nominal, ordinal, interval or ratio distances. With two coders and nominal codes it comes close to Scott's pi, which pools the two coders' code rates.
• Related DesignSensitivity and Specificity: Both summarize a table of two classifications of the same people. Sensitivity and specificity treat one classification as the truth (a diagnosis) and ask how often the other (a screen) gets it right; kappa treats the two as equals.
• Alternative MeasureWeighted Kappa: Weighted kappa gives partial credit for near misses when the categories are ordered. Cohen's kappa is its special case in which every disagreement costs the full amount (Fleiss, Cohen & Everitt, 1969). For nominal codes such as depression, anxiety or neither, unweighted κ is the right choice.

• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
• Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5), 423–429. https://doi.org/10.1016/0895-4356(93)90018-V
• Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323–327. https://doi.org/10.1037/h0028106

  “Formulas for the standard errors of these two statistics have been given in the literature, but they are in error.” (Abstract, p. 323)

• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310

  “κ is directly analogous to the intraclass correlation coefficient obtained from ANOVA models for quantitative measurements” (p. 163)

  “Although these divisions are clearly arbitrary, they do provide useful “benchmarks” for the discussion of the specific example in Table 1.” (p. 165)

• Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-L

  “In one paradox, a high value of p0 can be drastically lowered by a substantial imbalance in the table's marginal totals either vertically or horizontally.” (Abstract)

• Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. https://doi.org/10.1016/0895-4356(90)90159-M

  “the omnibus value of kappa should always be accompanied by separate individual values of ppos and pneg” (Abstract)

• McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031

  “Perhaps the best advice for researchers is to calculate both percent agreement and kappa.” (Conclusions)

  “Cohen's suggested interpretation may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable.” (Cohen's kappa)

• Cantor, A. B. (1996). Sample-size calculations for Cohen's kappa. Psychological Methods, 1(2), 150–153. https://doi.org/10.1037/1082-989X.1.2.150

  “When designing a study to produce an estimate κ̂ of kappa, the sample size should be chosen so that the standard error of κ̂ will not exceed a preassigned value.” (p. 151)

  “can be written in the form Q/N” (p. 151)

• Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85(3), 257–268. https://doi.org/10.1093/ptj/85.3.257

  “Factors that can influence the magnitude of kappa (prevalence, bias, and non-independent ratings) are discussed” (Abstract)

• Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
• Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
• Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009

  “we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)