Fleiss' Kappa: Interpretation and APA Reporting

The Basics

When to Use It

Use it when three or more raters each sort the same cases into the same set of unordered categories, working separately, and you want to know how well they agree once chance agreement is taken out. Every case needs the same number of codes, but they don't have to come from the same people: each case can be coded by a different set of raters from a larger pool (Fleiss, Nee & Landis, 1979). For example:

Do five clinicians agree on whether an intake interview points to depression, anxiety or neither?
Do four coders agree on whether each classroom episode shows on-task, off-task or disruptive behavior?
Do six trained judges agree on which emotion (anger, fear, sadness, joy or disgust) each face in a new photo set shows?

Fleiss' kappa works from how many raters gave each code to each case, not from which rater gave which, so it treats the raters as interchangeable. If the same identifiable raters code every case and their individual differences matter, consider a fixed-panel alternative such as Conger's (1980) kappa, or each pair's Cohen's kappa. With exactly two raters, the same two for every case, use Cohen's kappa. With ordered categories, such as mild, moderate and severe, use weighted kappa (Cohen, 1968), which counts a near miss as partial agreement.

Vocabulary You Need

Definition Inter-Rater Reliability

How consistently different raters (observers, coders or judges) rate the same cases. It describes the whole rating procedure, the raters, their training and the coding scheme, as used on a particular kind of case, not the raters alone. Categorical codes are usually summarized with a kappa (Cohen's for two raters, Fleiss' for more), and numeric ratings with an intraclass correlation. Krippendorff's alpha covers any number of raters, missing ratings and any scale. High inter-rater reliability means the ratings depend on the cases more than on who happened to rate them; it doesn't mean the ratings are correct, since raters can agree and all be wrong. Glossary →

Definition Observed Agreement (

p_o

)

For two raters, the number of cases they put in the same category, divided by the number of cases: the diagonal of their table over the total. With more raters, it's the average agreement across every pair of raters for each case (Fleiss' kappa's P̄). Multiplied by 100 it is percent agreement. It is easy to read, but it counts the agreements that would happen by chance too, so it flatters raters when one category is very common. Glossary →

Definition Chance Agreement (

p_e

)

How often raters would be expected to agree if they assigned categories independently, at the rates the categories were actually used. For two raters (Cohen's kappa), it's a share of cases: multiply the share of cases the first rater put in each category by the second rater's share, then add up across categories. With several raters (Fleiss' kappa), it's the chance that two ratings picked at random agree: square each category's overall share and add them up. “Chance” doesn't mean the raters guess: their category rates can reflect real judgment. It's the agreement a model of unrelated codes predicts from those rates. It is high when one category dominates: raters who all call most cases “neither” will often agree even if their codes are unrelated. Glossary →

Assumptions

1. Independent Raters Each rater rates each case they're given without knowing anyone else's ratings.

What It Means: The raters reach their ratings separately: they don't discuss cases, see each other's ratings or share notes before rating. Agreement statistics such as kappa, Krippendorff's alpha and the intraclass correlation treat each rating as a separate judgment. Raters who confer agree more than the rating procedure would on its own, so the result overstates how reliable it is.
Applies To: Raters
How to Check: By design, not by a test. Did each rater work from the same material (a recording, a transcript, the same case file) without access to anyone else's ratings, and were disagreements resolved only after all had finished?
How Much It Matters: A lot, and always in the same direction. Clinicians who talk interviews over, or code in the same room, agree more than the codebook would on its own, so κ comes out too high. No statistic can undo it afterward.

If It's Violated

Design Change Have each clinician code from the same recording without seeing anyone else's codes, and hold consensus meetings only after everyone has finished.
Proceed with Caveat If some interviews were discussed before coding, compute κ from the independent codes only, or say plainly that it overstates independent agreement.

2. Same Number of Interchangeable Raters Every case gets the same number of codes, and who gave them doesn't matter.

What It Means: Each case is coded by the same number of raters, and the statistic treats those raters as drawn at random from a pool: it counts how many raters chose each category for a case, not which rater chose it. Chance agreement is worked out from how often the raters as a whole used each category, so a rater who uses a category much more or less than the others counts as disagreement rather than being set apart.
Applies To: Raters
How to Check: By design: count the codes each case received. Then tabulate each rater's use of each category; if one rater stands out, say so, and consider reporting agreement for each pair of raters as well.
How Much It Matters: It decides what κ describes. Fleiss' kappa pools the raters' code rates, so it describes agreement between two of the raters picked at random, not between any one pair. A clinician who uses one code much more than the others counts against agreement, and the statistic can't say who it is. The interval allows for a different set of cases, not a different set of raters, so with the same five clinicians throughout it speaks for these five. To speak for raters in general, have a different random set of raters code each case (Fleiss, Nee & Landis, 1979). It also needs the same number of codes for every interview.

If It's Violated

Proceed with Caveat Tabulate how often each rater used each code (the data box does this) and name any rater who stands out.
Alternative Test Cohen's Kappa. When the same fixed raters code every case and you care about them individually, report each pair's Cohen's kappa, or Conger's (1980) kappa, which keeps each rater's own code rates.
Design Change If some cases are missing a code, have them recoded so every case has the same number, or drop them and say how many.
Alternative Test Krippendorff's Alpha. Krippendorff's alpha handles missing codes, different numbers of raters per case and ordered or numeric ratings, so nothing has to be dropped.

3. Same Distinct Categories Every rater puts every case into one of the same set of distinct categories.

What It Means: Each case gets exactly one code from each rater, chosen from a list fixed before rating that every rater uses, and the categories don't overlap. Unweighted kappa counts every disagreement as equally serious and ignores any order among the categories, so it suits unordered (nominal) codes; with ordered categories, weighted kappa gives partial credit for near misses.
Applies To: Raters
How to Check: By design: read the codebook. Then tabulate each rater's use of each code, checking that every code is spelled the same way by every rater. With two raters, cross-tabulate their codes: the table should be square, with the same categories along the rows and the columns, and a category only one rater used still needs its own row and column.
How Much It Matters: It decides what κ means. Overlapping categories (an interview pointing to both depression and anxiety) force arbitrary choices that show up as disagreements, and with ordered categories a near miss counts as a full disagreement, so κ understates agreement.

If It's Violated

Design Change Add decision rules, such as coding the main presenting concern, and pilot the codebook on practice interviews before the study.
Alternative Test Weighted Kappa. For ordered categories (mild, moderate, severe), use weighted kappa, which gives partial credit for near misses (Cohen, 1968).

4. Independent Observations The people (or other units) in your study don't influence each other.

What It Means: Each interview is a separate case. The five codes on one interview are meant to be related; it's different interviews that mustn't be linked, for example by coming from the same client.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: Moderate. κ still describes the agreement in your data, but interviews that are linked, such as several with the same client, make the standard error too small, so the interval comes out too narrow and p too small.

If It's Violated

Design Change Code one interview per client, or pick one at random from each client.
Robust Estimation Resample clients rather than interviews to build the interval, so the clustering is kept intact.

Limitations

This Is Not Your Test If…

Only two raters coded the cases Cohen's Kappa →
Your categories are ordered, so near misses should count for something Weighted Kappa →
Your ratings are numbers, not categories Intraclass Correlation Coefficient (ICC) →
You're checking codes against a gold standard Sensitivity and Specificity →
You're asking whether some raters say yes more often Cochran's Q →
Some cases are missing codes or got different numbers of raters Krippendorff's Alpha →

What It Doesn't Tell You

Whether the codes are right. Five clinicians can agree and all be wrong. Agreement is reliability, not accuracy; checking codes against a structured diagnostic interview is a separate question.
Whether it's good enough. A κ of .58 might be fine for sorting a waiting list and not good enough for deciding treatment. The benchmarks can't make that call.
Which clinician is off. Fleiss' kappa pools everyone's codes, so it can't say whether one clinician drags agreement down. Each clinician's code rates (the data box) or each pair's Cohen's kappa can.
Who else. The result describes these five clinicians, this codebook and this clinic's mix of intakes. Its interval allows for other interviews, not other clinicians; a different five could agree more or less. With a different mix, say mostly anxiety referrals, chance agreement changes and so does κ.

Common Mistakes

Don'tBuild the confidence interval from the standard error software prints for the z test.

InsteadUse a standard error that doesn't assume κ = 0 (Gwet, 2021) or a bootstrap. The z-test one gives intervals far too narrow.
Don'tReport percent agreement alone as inter-rater reliability.

InsteadReport κ alongside the mean pairwise agreement and each category's κ, so readers can see how much agreement chance explains.
Don'tTreat a significant test of κ against zero as evidence of good agreement.

InsteadJudge the size of κ and its interval. The test only makes agreement at chance level implausible, a very low bar.
Don'tCall Fleiss' kappa for two raters the same as Cohen's kappa.

InsteadWith two raters it is Scott's pi, which pools the raters' code rates (Gwet, 2021). Use Cohen's kappa for a fixed pair of raters.
Don'tReport only the overall κ.

InsteadGive each category's κ too, with an interval if you can. Here anxiety (.46) is lowest, but with 60 interviews its bootstrap interval, .27 to .61 (Table 2), overlaps both of the others, so treat it as a lead to check, not a verdict.
Don'tRead κ = .58 as “the raters agreed on 58% of cases.”

InsteadRead it as the share of possible agreement beyond chance, and give the mean pairwise agreement separately.
Don'tRead a p-value as the chance that the null hypothesis is true.

InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance and Interpretation Guidelines

Benchmarks Significance (p Value)

p < .001
Very strong evidence against the null hypothesis.
.001 ≤ p < .01
Strong evidence against the null hypothesis.
.01 ≤ p < .05
Statistically significant at α = .05. Moderate evidence against the null hypothesis.
.05 ≤ p < .10
Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.
p ≥ .10
Not significant. Little evidence against the null, which is not the same as evidence of no effect.

These evidence labels are one common convention for describing p in graded language rather than as significant or not; the boundaries are rough guides, not rules, and other authors draw them differently. Muff et al. (2022) argue for this kind of language of evidence over a single cutoff. α is chosen before the analysis, and p never says how big or important an effect is.

More about statistical significance ↓

Benchmarks Fleiss' Kappa

κ < .00 Poor
The raters agree less often than chance would produce.
.00–.20 Slight
Barely more agreement than chance.
.21–.40 Fair
Some agreement beyond chance, well short of what was possible.
.41–.60 Moderate
About half the possible agreement beyond chance.
.61–.80 Substantial
Most of the possible agreement beyond chance.
.81–1.00 Almost Perfect
Nearly all the possible agreement beyond chance.

Landis and Koch (1977) proposed these labels for kappa while discussing one example and called the divisions “clearly arbitrary”; they are applied to Fleiss' kappa by convention. Moons and Vandervieren (2025) warn that comparing a κ with a benchmark scale ignores how many cases, categories and raters the study had, so read the whole interval against them. They are conventions, not a verdict on whether agreement is good enough for your purpose.

More about effect size ↓

How It's Reported

Equation Form

Example

P̄ = .74, κ = .58, SE = .053, 95% CI [.47, .69], z = 19.50, p < .001

Table 1. How the Five Codes Split on Each Interview

Example

	Interviews	Agreeing Pairs (of 10)	$P_i$	Sum of $P_i$
5–0 (All Agree)	29	10	1.00	29.0
4–1	17	6	.60	10.2
3–2	11	4	.40	4.4
3–1–1	2	3	.30	0.6
2–2–1	1	2	.20	0.2

Note. N = 60 interviews, each coded by 5 clinicians, so each has 10 pairs of clinicians. A 4–1 split means four clinicians gave one code and the fifth another. P_i is the share of the 10 pairs who agree. The sum column adds P_i over the interviews with that split; its total, 44.4, divided by 60 gives the mean pairwise agreement, .74.

Table 2. Agreement by Category

Example

	Codes Given	Share	All Five Agree	κ	95% CI
Depression	90	.30	8	.60	[.44, .72]
Anxiety	60	.20	3	.46	[.27, .61]
Neither	150	.50	18	.65	[.52, .76]
All Categories	300		29	.58	[.47, .69]

Note. N = 60 interviews × 5 clinicians = 300 codes. Share is each code's share of all 300 codes (p_j). Each category's κ compares that code against the other two combined. Its 95% CI is a percentile bootstrap: the 60 interviews were resampled with replacement 10,000 times, each keeping its five codes together, and the middle 95% of the resampled kappas kept (seed 2026 in this site's own calculation; another seed or program will differ by about .01). The overall κ's interval is Gwet's (2021), as in the text. Every interval is wide, and anxiety's overlaps both of the others, so treat its lower κ as a lead to check, not a verdict.

Full Text (APA Style)

Example

Five clinicians independently coded the main presenting concern in 60 recorded intake interviews as depression, anxiety or neither. All five gave the same code to 29 interviews (48%; Table 1), and on average 74% of the pairs of clinicians agreed on an interview (P̄ = .74). Because half of all codes were neither, 38% agreement would be expected by chance alone (P̄_e = .38). Agreement beyond chance was moderate by Landis and Koch's (1977) benchmarks, Fleiss' κ = .58, SE = .053, 95% CI [.47, .69], with the interval spanning moderate to substantial; the standard error and interval follow Gwet (2021). Agreement was clearly better than chance, z = 19.50, p < .001. It was lowest for anxiety (κ = .46) and highest for neither (κ = .65; Table 2), though with 60 interviews the category kappas are imprecise.

Give the mean pairwise agreement and each category's κ along with the overall κ, so readers can see how much of the agreement chance alone would produce and which codes might cause trouble. With few cases, give the category kappas' intervals too, or don't read much into their differences.
Report κ's standard error and 95% interval, and say which standard error you used. The one most software prints (Fleiss, Nee & Landis, 1979) is only for the test against zero; use Gwet's (2021) or a bootstrap for the interval.
Say how many raters coded each case and whether they were the same raters throughout.
The test against κ = 0 is optional. If you report it, don't present it as evidence that agreement is good. Greek κ isn't italicized; p, z and SE are.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

P̄ = .74	On an average interview, 74% of the pairs of clinicians gave the same code, counting agreements that could have happened by chance.
κ = .58	The clinicians reached a little more than half of the agreement beyond chance that was possible. That's moderate by the Landis and Koch labels above, and well below the 74% raw agreement, because 38% agreement was expected by chance. Benchmarks ↑
95% CI [.47, .69]	The true kappa is plausibly anywhere from .47, moderate by the benchmarks, to .69, substantial. With 60 interviews we can't pin down which label fits.
SE = .053	If the study were repeated with another 60 interviews like these, κ would typically vary by about .05 from study to study.
z = 19.50	Kappa is 19.50 standard errors above zero, the value it would have if the clinicians agreed only by chance. This uses the test's own standard error, .030, which assumes chance-level agreement (step 8). It is much smaller than the SE above, which is why it mustn't be used for the interval.
p < .001	If the clinicians were agreeing only by chance, a kappa at least this far from zero, in either direction, would essentially never turn up. That makes chance-level agreement implausible, which no one suspected; it says nothing about whether .58 is good enough. Benchmarks ↑

Calculations

The Running Example

Example

The intake-panel study: five clinicians, working separately, each coded the same 60 intake interviews as depression, anxiety or neither.

Null Hypothesis:: $\kappa = 0$
Alternative (Two-Sided):: $\kappa \neq 0$

κ (“kappa”) here is the population value: how far beyond chance two of these five clinicians, picked at random, would agree across all intakes like these, which the sample's .58 estimates. The interval allows for other interviews, not other clinicians. κ = 0 means agreeing only as often as clinicians picking codes at random, at the panel's overall rates, would. That's a very low bar; the interval, not the test, says how good the agreement is.

What Your Data Should Look Like

Every case coded by the same number of raters, using exactly the same codes, spelled the same way. Most software wants it wide: one row per case and one column per rater. Ours is stored long, one row per case per rater, with columns for the interview, the clinician and the code; R and Python reshape it to wide in one line. Fleiss' kappa then counts, for each case, how many raters chose each category (a cases × categories table, summarized in Table 1). Which rater gave which code doesn't enter the calculation.

Example

interview	clinician	code
1	A	neither
1	B	neither
1	C	neither
1	D	depression
1	E	depression
2	A	depression
… 294 more rows

	Clinician A	Clinician B	Clinician C	Clinician D	Clinician E	Total
Depression	18 (30%)	16 (27%)	16 (27%)	19 (32%)	21 (35%)	90
Anxiety	12 (20%)	12 (20%)	14 (23%)	12 (20%)	10 (17%)	60
Neither	30 (50%)	32 (53%)	30 (50%)	29 (48%)	29 (48%)	150
Total	60	60	60	60	60	300

Counts, with the percentage of each clinician (each column adds to 100%).

Simulated data.

Download all 300 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the intake-panel study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Count How Many Clinicians Gave Each Code to Each Interview, and How Many Pairs Agreed

Fleiss' kappa doesn't care which clinician gave which code, only how the five codes split on each interview. For each interview, count the pairs of clinicians who agree, out of the 10 pairs five clinicians make. The formula counts each pair twice (A with B and B with A), which gives the same share.

$P_i = \dfrac{\sum_j n_{ij}(n_{ij} - 1)}{m(m - 1)}$ = $\begin{array}{l|ccc} & \text{Dep.} & \text{Anx.} & \text{Neither} \\ \hline \text{Interview 1} & 2 & 0 & 3 \\ \text{Interview 2} & 5 & 0 & 0 \\ \text{Interview 3} & 0 & 0 & 5 \\ \vdots & \vdots & \vdots & \vdots \\ \text{Total} & 90 & 60 & 150 \end{array} \qquad P_1 = \dfrac{2(1) + 0 + 3(2)}{5 \times 4} = \dfrac{8}{20}$

$P_1$ = .40
$\text{All five agree}$ = 29 interviews

$n_{ij}$ is how many of our $m$ = 5 clinicians gave interview $i$ code $j$ . Interview 1 split 2 depression, 3 neither. The 2 depression codes make 1 agreeing pair and the 3 neither codes make 3, so 4 of the 10 pairs agree, $P_1$ = .40. Counting each pair both ways, as the formula does, gives 2 × 1 + 3 × 2 = 8 of 20, the same share. All five agreed on 29 of the 60 interviews (Table 1).
2. Average That Agreement across All the Interviews

The average share of agreeing pairs is the multi-rater version of percent agreement. It's easy to read, but it doesn't yet ask how many of those agreements were luck.

$\bar{P} = \dfrac{1}{N}\sum_{i} P_i$ = $\dfrac{29(1) + 17(.6) + 11(.4) + 2(.3) + 1(.2)}{60} = \dfrac{44.4}{60}$

= .74

$P_i$ is 1 for the 29 unanimous interviews, .6 for the 17 with a 4–1 split, .4 for the 11 split 3–2, and .3 or .2 for the three messier ones. They add up to 44.4, and $N$ is our 60 interviews. 44.4 ÷ 60 = .74.
3. Work Out How Often Two Clinicians Would Agree by Chance Alone

Clinicians who call half of all interviews “neither” will often agree even if they barely listen. How often depends only on how often each code was used, pooled across all five clinicians.

$\bar{P}_e = \sum_j p_j^2, \quad p_j = \dfrac{\sum_i n_{ij}}{Nm}$ = $.30^2 + .20^2 + .50^2 = .09 + .04 + .25$

= .38

$p_{\text{dep}}$ = .30
$p_{\text{anx}}$ = .20
$p_{\text{neither}}$ = .50

$p_j$ is each code's share of all 300 codes, 90 depression (.30), 60 anxiety (.20) and 150 neither (.50). Squaring and adding gives .38. Two clinicians picking codes at random at those rates would agree 38% of the time, mostly on neither.
4. Measure Agreement beyond Chance as a Share of What Was Possible

Take the agreement beyond chance that the clinicians reached, and divide it by the agreement beyond chance there was room for. That's kappa.

$\kappa = \dfrac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}$ = $\dfrac{.74 - .38}{1 - .38} = \dfrac{.36}{.62}$

= .58

$\bar{P} - \bar{P}_e$ = .74 − .38 = .36 is how far the clinicians got beyond chance. $1 - \bar{P}_e$ = .62 is how far they could have got. .36 ÷ .62 = .58, a little more than half the way from chance to perfect agreement.
5. Figure Out How Precisely Kappa Is Estimated

A different 60 interviews would give a somewhat different kappa. Gwet's (2021) standard error measures how much, by giving each interview a score whose average is kappa and asking how much those scores vary.

$SE = \sqrt{\dfrac{\sum_i (\kappa_i^{*} - \kappa)^2}{N(N - 1)}}, \quad \kappa_i^{*} = \dfrac{P_i - \bar{P}_e}{1 - \bar{P}_e} - 2(1 - \kappa)\dfrac{P_{e|i} - \bar{P}_e}{1 - \bar{P}_e}, \quad P_{e|i} = \sum_j p_j \dfrac{n_{ij}}{m}$ = $\dfrac{0.410}{\sqrt{60}}$

= .053

$\kappa_i^{*}$ is interview $i$ 's score, from 1.24 when all five chose anxiety (a rare code, so unanimity is impressive) and 0.84 when all five chose neither, down to −0.24 for the 2–2–1 split. The 60 scores average .58 and have an SD of 0.410. Dividing by √60 (about 7.75) gives $SE$ = .053.
6. Give a Range of Plausible Values for the True Kappa

Instead of a single best guess, give the range of kappas for these five clinicians, on intakes like these, that fit the data.

$\kappa \pm t^{*}_{N - 1} \times SE$ = $.581 \pm 2.001 \times .053$

= [.47, .69]

$t^{*}_{59}$ = 2.001 holds the middle 95% of a t distribution with 60 − 1 = 59 df; Gwet (2021) uses t rather than the normal distribution's 1.96, which the Cohen's kappa page uses, and with 60 interviews the two barely differ. 2.001 × .053 = .11, so the range is .58 ± .11, from .47 to .69.
7. Check Agreement on Each Code Separately

One kappa can hide a code the clinicians struggle with. Kappa for one code asks how well they agree on whether an interview gets that code or one of the others.

$\kappa_j = 1 - \dfrac{\sum_i n_{ij}(m - n_{ij})}{N m (m - 1)\, p_j q_j}$ = $1 - \dfrac{104}{60 \times 5 \times 4 \times .20 \times .80} = 1 - \dfrac{104}{192}$

$\kappa_{\text{dep}}$ = .60
$\kappa_{\text{anx}}$ = .46
$\kappa_{\text{neither}}$ = .65

For anxiety, $\sum n_{ij}(m - n_{ij})$ counts the pairs split over whether an interview is anxiety, 104. Chance would give 60 × 5 × 4 × .20 × .80 = 192. 1 − 104 ÷ 192 = .46. The same sum gives .60 for depression and .65 for neither.
8. Compare Kappa with No Agreement beyond Chance

The test asks whether kappa is further from zero than sampling error would explain. It uses a different standard error, worked out as if the clinicians were picking codes at random.

$z = \dfrac{\kappa}{SE_0}, \quad SE_0 = \dfrac{\sqrt{2}}{\sum_j p_j q_j \sqrt{N m (m - 1)}}\sqrt{\Big(\sum_j p_j q_j\Big)^2 - \sum_j p_j q_j (q_j - p_j)}$ = $\dfrac{.581}{.030}$

= 19.50

$SE_0$ = .030

$SE_0$ is the standard error if the clinicians agreed only by chance, .030 from the three codes' shares alone. .581 ÷ .030 puts kappa 19.50 of those standard errors above zero.
9. Ask How Surprising This Agreement Would Be If It Were Only Chance

Finally, ask how often clinicians agreeing only by chance would produce a kappa this far from zero. With 60 interviews and five clinicians, almost any real agreement passes this test.

$p = 2 \times P(Z \ge |z|)$ = $2 \times P(Z \ge 19.50) < .001$

$P(Z \ge 19.50)$ is the chance that a standard normal value lands 19.50 or more above zero. It's far smaller than .001, even doubled to count both directions.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

SPSS wants one column per clinician. Our file is long, so first use Data → Restructure → Restructure selected cases into variables, with interview as the Identifier Variable and clinician as the Index Variable. You get one row per interview with columns A to E.
Analyze → Scale → Reliability Analysis. Move A, B, C, D and E into the variables list.
Click Statistics. Under Inter-Rater Agreement: Fleiss' Kappa, check Display agreement on individual categories. Click Continue, then OK (or Paste, to keep the syntax).

Or paste this syntax:

* Step 1 only: restructure to one column per clinician.
* Then run steps 2 and 3 in the dialog; Paste there gives the RELIABILITY syntax.
SORT CASES BY interview clinician.
CASESTOVARS
  /ID=interview
  /INDEX=clinician.

Reading the Output

SPSS prints the overall kappa and, with the category option checked, a kappa for each category. The values below are worked out from the data with the published formulas, not copied from an SPSS run, so compare them with yours. Read Kappa first (.581), then each category's kappa: .595 for depression, .458 for anxiety and .647 for neither. SPSS sorts the codes alphabetically, so anxiety comes first.

Its z (19.50 from the formulas) and significance are the test against κ = 0, which is fine. SPSS also prints an asymptotic confidence interval. Gwet (2021) found that SPSS's standard error for Fleiss' kappa was Fleiss, Nee and Landis's (1979), meant only for the test, but his paper predates the Fleiss' kappa option in this dialog, so check yours: if SPSS's standard error is .030 (kappa ÷ z), its interval is the too-narrow one, about [.52, .64]. Then use Gwet's standard error (.053, from irrCAC in the Python section below, or step 5 above) and his interval, [.47, .69], instead.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

library(irr)

panel <- read.csv("data.csv")

# One row per interview, one column per clinician
wide <- reshape(panel, idvar = "interview", timevar = "clinician", direction = "wide")
codes <- wide[, c("code.A", "code.B", "code.C", "code.D", "code.E")]

kappam.fleiss(codes, detail = TRUE)

 Fleiss' Kappa for m Raters

 Subjects = 60
   Raters = 5
    Kappa = 0.581

        z = 19.5
  p-value = 0

            Kappa      z p.value
anxiety     0.458 11.227   0.000
depression  0.595 14.580   0.000
neither     0.647 15.840   0.000

irr gives κ, the test against zero (a p of 0 means p < .001) and, with detail = TRUE, each category's κ. It doesn't give a standard error or an interval; its z uses Fleiss, Nee and Landis's (1979) standard error under κ = 0. For the interval you need Gwet's (2021) standard error, as in step 5 above or irrCAC in the Python section. kappam.fleiss(codes, exact = TRUE) gives Conger's (1980) version, which keeps each clinician's own code rates; here it is also .581, because the five clinicians used the codes at similar rates.

Output from a real run: R 4.6.1 (irr 0.85), run 2026-10-03.

Python

import pandas as pd
from statsmodels.stats.inter_rater import aggregate_raters, fleiss_kappa

panel = pd.read_csv("data.csv")
wide = panel.pivot(index="interview", columns="clinician", values="code")

# Interviews × codes: how many clinicians gave each code
counts, categories = aggregate_raters(wide.to_numpy())
print(categories)
print(counts[:3])
print(fleiss_kappa(counts))

# Gwet's (2021) standard error and interval: pip install irrCAC
from irrCAC.raw import CAC
print(CAC(wide).fleiss()["est"])

['anxiety' 'depression' 'neither']
[[0 2 3]
 [0 5 0]
 [0 0 5]]
0.5806451612903225
{'coefficient_value': 0.58065, 'coefficient_name': "Fleiss' kappa", 'confidence_interval': (0.47461, 0.68668), 'p_value': 6.661338147750939e-16, 'z': 10.95787, 'se': 0.05299, 'pa': 0.74, 'pe': 0.38}

aggregate_raters turns one-column-per-rater data into the interviews × codes count table Fleiss' kappa works from (codes in alphabetical order), and fleiss_kappa gives κ only. irrCAC adds Gwet's standard error (.053) and 95% interval, [.47, .69], the same as this page. Its z (10.96) and p divide κ by that standard error rather than by the one under κ = 0, so they differ from R's; either way p < .001.

Output from a real run: Python 3.9.6 (statsmodels 0.14.6, irrCAC 0.4.4, numpy 1.26.4, pandas 2.3.3), run 2026-10-03.

Impact

Effect Size

Kappa is this test's measure of size. It rescales agreement so that 0 means chance level and 1 means every rater agrees on every case. That makes agreement on schemes with different numbers of codes easier to compare, though not fully comparable (see below). Fleiss' version measures agreement between pairs of raters, averaged over every pair and every case.

\kappa = \dfrac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e} = \dfrac{.74 - .38}{1 - .38} = .58

On an average interview, 74% of the pairs of clinicians agreed, but 38% agreement was expected by chance because half of all codes were “neither.” Of the 62 points of agreement beyond chance that were possible, they reached 36, so κ = .58: moderate, by convention.

The overall κ is a weighted average of the kappas for each category, weighted most toward codes used close to half the time (here neither), so it's worth looking at them one at a time.

The clinicians agreed best on neither (κ = .65) and worst on anxiety (κ = .46). All five agreed on anxiety for only 3 interviews; its 60 codes could have filled 12 interviews, so that's 3 of a possible 12, against 18 of a possible 30 for neither. With 60 interviews the category kappas are rough (anxiety's interval overlaps depression's), so anxiety is the first place to look in the codebook, not proof it's the problem.

Like Cohen's kappa, Fleiss' kappa depends on how common each category is, not just on how well the raters agree: when one code dominates, chance agreement is high and a few disagreements pull κ down, so κ can be low even when the raters almost always agree. Report the mean pairwise agreement beside it; some researchers add Gwet's AC1, which was designed to be less affected (Gwet, 2021). It also depends on the design (how many cases, categories and raters), which a benchmark label ignores (Moons & Vandervieren, 2025). Benchmarks are conventions. Whether a panel that splits on about half the interviews is consistent enough depends on what the codes will be used for.

Benchmarks Fleiss' Kappa at a Glance

Poor

κ < .00

Slight

.00–.20

Fair

.21–.40

Moderate

.41–.60

Substantial

.61–.80

Almost Perfect

.81–1.00

Full benchmarks and caveats ↑

Statistical Significance

p answers one narrow question: if the raters agreed only as often as chance would produce, how often would kappa land at least this far from zero, in either direction? Below the conventional α = .05, the result is called statistically significant. The test uses Fleiss, Nee and Landis's (1979) standard error, worked out as if the raters were picking categories at random. That's right for the test and wrong for anything else: Gwet (2021) shows intervals built from it cover the true κ far less often than they claim, which is why this page uses his standard error for the interval.

If the clinicians were agreeing only by chance, a kappa at least as far from zero as .58, in either direction, across 60 interviews and five clinicians would essentially never turn up: $z$ = 19.50, $p$ < .001. Each category's kappa passes the same test too, even anxiety's ( $z$ = 11.23, $p$ < .001), which is why Table 2 gives intervals instead.

For reliability, that's rarely the interesting question. Nobody trains five clinicians expecting chance-level agreement, and with a reasonable number of cases almost any real agreement beats zero, including agreement far too poor to use. The question that matters is how high κ is and how precisely it's estimated, which is what the interval answers.

p is not the probability that the clinicians agree only by chance, and it says nothing about whether their agreement is good enough. That's what κ and its interval are for.

Benchmarks p at a Glance

Very Strong

p < .001

Strong

.001 ≤ p < .01

Significant

.01 ≤ p < .05

Weak

.05 ≤ p < .10

Little

p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. For kappa, a test of agreement better than chance is usually a low bar, though power can still matter with few cases, a rare code, or a test against a minimum other than zero. The more useful planning question is usually precision: how many cases you need for the interval around κ to be narrow enough to tell, say, moderate from substantial agreement.

Gwet's (2021) standard error shrinks with the square root of the number of cases. How big it is for a given number depends on how often each code is used, how many raters code each case and how messy the disagreements are, so a pilot study or an earlier study with a similar panel gives a workable guess.

Our standard error, .053 with 60 interviews, works out to √(0.1685 ÷ N). If a new study used five clinicians and a similar mix of intakes, getting the interval to ±.10 would take t* × √(0.1685 ÷ N) ≤ .10. With t* just under 2 at that size, that first holds at 68 interviews (at 67 the margin is .1001, just over); ±.15 would take 32.

Plan a kappa study around the width of the interval you need, not around the test against zero, using the category mix and number of raters you expect. Gwet (2021) found his standard error gave intervals close to their stated 95% coverage with as few as 10 cases in his simulation; the planning arithmetic above is our own, from our data.

Other Info

Follow-Up and Post Hoc Tests

None needed. This test answers a single question, so there are no planned contrasts or post hoc tests to run.

Related Methods

Related DesignCohen's Kappa: The same idea, agreement beyond chance as a share of what was possible, for two raters. Despite the name, Fleiss' kappa isn't Cohen's kappa extended: with two raters it becomes Scott's pi, which pools the two raters' code rates to work out chance agreement, while Cohen's kappa keeps each rater's own rates (Gwet, 2021). The two differ little unless the raters use the codes at very different rates.
Related DesignWeighted Kappa: Weighted kappa is for two raters whose categories are ordered, so a near miss (mild vs moderate) can count as partial agreement. Fleiss' kappa treats every disagreement as equally serious, which suits unordered codes like depression, anxiety or neither.
Related DesignIntraclass Correlation Coefficient (ICC): The same question for numeric ratings from several raters. Like Fleiss' kappa, the one-way ICC(1) allows different raters for different targets; with categories instead of scores, Fleiss' kappa is the counterpart.
Alternative MeasureKrippendorff's Alpha: Krippendorff's alpha with the nominal metric gives nearly the same answer as Fleiss' kappa when every case has the same number of codes (Zapf et al., 2016). It also handles cases with missing codes or different numbers of raters, and ordered or numeric ratings, so nothing has to be dropped.

References

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
Fleiss, J. L., Nee, J. C. M., & Landis, J. R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86(5), 974–977. https://doi.org/10.1037/0033-2909.86.5.974
“Published formulas for the large sample variance of the kappa statistic that are appropriate for the case of different sets of raters for different Ss, when each set of raters is selected at random from a larger pool of available raters, are determined to be incorrect.” (Abstract)
Gwet, K. L. (2021). Large-sample variance of Fleiss generalized kappa. Educational and Psychological Measurement, 81(4), 781–790. https://doi.org/10.1177/0013164420973080
“Although Fleiss (1971) labeled his coefficient as the generalized kappa, it does not reduce to Cohen’s kappa when the number of raters is 2. Instead, it reduces to the pi coefficient proposed earlier by Scott (1955).” (Introduction)

“is not meant to be used for quantifying the precision of Fleiss’ kappa. Instead, it should be used solely for testing the hypothesis of zero agreement among raters.” (Introduction)

“However, the coverage rate of confidence intervals based of the variance proposed by Fleiss et al. (1979) is dramatically low.” (Results)
Moons, F., & Vandervieren, E. (2025). Measuring agreement among several raters classifying subjects into one or more (hierarchical) categories: A generalization of Fleiss’ kappa. Behavior Research Methods, 57(10), Article 287. https://doi.org/10.3758/s13428-025-02746-8
“The extent of agreement among J raters for the subject i can be calculated as the proportion Pi of agreeing rater pairs” (Fleiss’ kappa)

“directly comparing any κ value against a benchmark scale can be misleading, as it fails to account for differences in the number of subjects, categories, and raters across studies.” (Interpreting & benchmarking the new measure)
Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2026). irr: Various coefficients of interrater reliability and agreement (R package version 0.85) [Computer software]. https://doi.org/10.32614/CRAN.package.irr
“The coefficient described by Fleiss (1971) does not reduce to Cohen's Kappa (unweighted) for m=2 raters.” (kappam.fleiss, Details)
Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88(2), 322–328. https://doi.org/10.1037/0033-2909.88.2.322
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
“Although these divisions are clearly arbitrary, they do provide useful “benchmarks” for the discussion of the specific example in Table 1.” (p. 165)
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
“we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)