Krippendorff's Alpha: Interpretation and APA Reporting

The Basics

When to Use It

Use it when two or more coders each code the same units, working separately, and you want to know how well they agree beyond chance, especially when not every coder coded every unit, units got different numbers of codes, or the codes are ordered or numeric. You choose a distance metric to match the scale: nominal for unordered codes, ordinal for ordered categories, interval for equally spaced scores, ratio for amounts with a true zero (Krippendorff, 2011; Hayes & Krippendorff, 2007). For example:

Do trained observers agree on how engaged a child is during a classroom lesson (off task, passive, engaged or highly engaged)?
Do coders agree on which defense mechanism each therapy-session excerpt shows, when each excerpt was coded by a different two or three of six coders?
Do judges agree on how many self-critical statements each recorded interview contains (a count, with a ratio metric)?

Any number of coders can take part, but some units must be coded by two or more of them, since a unit with one code has nothing to compare. With a larger team, plan the assignments so the coders overlap and each one's share covers the kinds of units being studied; if one coder works alone on one kind of unit, that coder's habits and that kind of unit's difficulty can't be told apart.

With nominal codes and every case coded by the same number of raters, Fleiss' kappa gives nearly the same answer (Zapf et al., 2016). With two raters on an ordered scale who rated everything, weighted kappa lets you set the weights yourself. If you want the reliability of the average of several raters' scores, rather than of one rater's, use an intraclass correlation.

Vocabulary You Need

Definition Inter-Rater Reliability

How consistently different raters (observers, coders or judges) rate the same cases. It describes the whole rating procedure, the raters, their training and the coding scheme, as used on a particular kind of case, not the raters alone. Categorical codes are usually summarized with a kappa (Cohen's for two raters, Fleiss' for more), and numeric ratings with an intraclass correlation. Krippendorff's alpha covers any number of raters, missing ratings and any scale. High inter-rater reliability means the ratings depend on the cases more than on who happened to rate them; it doesn't mean the ratings are correct, since raters can agree and all be wrong. Glossary →

Definition Coincidence Matrix (

o_{ck}

)

The table Krippendorff's alpha is computed from. Every pair of values that two different coders gave the same unit is entered twice, once as c–k and once as k–c, and a unit coded by m coders has each of its pairs weighted 1 / (m − 1), so each value counts once in all. A unit with only one value can't be paired and drops out. The diagonal holds matching pairs; the margins, n_c, count how many pairable values of each kind there are, and their total, n, is the number of pairable values. Unlike a contingency table, which crosses one coder's codes with another's, it counts values, so it doesn't matter which coder gave which, or how many coded each unit. Glossary →

Definition Observed Disagreement (

D_o

)

In Krippendorff's alpha, the average disagreement between pairs of values given to the same unit by different coders: each cell of the coincidence matrix times the distance between its two values, summed and divided by the number of pairable values. With nominal codes it is simply the share of pairs that don't match. Its size depends on the distance metric, so it is read against expected disagreement, not on its own. Glossary →

Definition Distance Metric (

\delta^2_{ck}

)

The rule Krippendorff's alpha uses to say how far apart two values are, written as a squared difference. Nominal: any two different values disagree by 1. Ordinal: the distance grows with how many coded values lie between the two, counting half of each end, so it depends on how often each value was used, not on any numbers attached to it. Interval: the squared numerical difference. Ratio: the squared difference relative to the two values' sum, so differences near zero count more. Krippendorff (2011) also gives circular and bipolar metrics. Pick the metric the scale supports before looking at the results; the same codes give different alphas under different metrics. Glossary →

Definition Expected Disagreement (

D_e

)

In Krippendorff's alpha, the disagreement you would expect if the values the coders used had been handed out to units at random: the average distance between two values drawn, without replacement, from the pool of all pairable values, n_c n_k pairs for each combination, divided by n(n − 1). It depends only on how often each value was used, not on which unit got it. When most values are the same, expected disagreement is small, so a few real disagreements cost alpha a lot. Glossary →

Assumptions and Inference

1. Independent Raters Each rater rates each case they're given without knowing anyone else's ratings.

What It Means: The raters reach their ratings separately: they don't discuss cases, see each other's ratings or share notes before rating. Agreement statistics such as kappa, Krippendorff's alpha and the intraclass correlation treat each rating as a separate judgment. Raters who confer agree more than the rating procedure would on its own, so the result overstates how reliable it is.
Applies To: Raters
How to Check: By design, not by a test. Did each rater work from the same material (a recording, a transcript, the same case file) without access to anyone else's ratings, and were disagreements resolved only after all had finished?
How Much It Matters: A lot, and always in the same direction. Observers who talk clips over, or watch them together, agree more than the coding scheme would on its own, so α comes out too high. No statistic can undo it afterward.

What to Do

Design Change Have each observer code from the same recording without seeing anyone else's codes, and hold consensus meetings only after everyone has finished.
Proceed with Caveat If some clips were discussed before coding, compute α from the independent codes only, or say plainly that it overstates independent agreement.

2. Metric Matches the Scale You choose the distance metric your codes' scale supports, before seeing the results.

What It Means: Krippendorff's alpha measures disagreement with a distance metric: nominal for unordered codes, ordinal for ordered categories, interval for equally spaced scores, ratio for counts or amounts with a true zero. The metric decides how much a near miss costs, so a metric that assumes more than the scale supports, such as interval distances for categories that aren't equally spaced, gives near misses the wrong credit.
Applies To: Raters
How to Check: By reasoning about the scale, not by a test. Ask whether the codes have an order, whether the steps between them are equal, and whether zero means none. Write the choice down before computing alpha, and if you report several metrics, say which is the main one.
How Much It Matters: It decides what α means. Here the nominal metric, which treats passive against engaged as just as wrong as off task against highly engaged, gives .46 where the ordinal metric gives .76, because almost every disagreement was one step apart. The interval metric gives .76 too, but only because the four codes were used about equally often, so their ordinal distances happen to be close to equal steps.

What to Do

Correction Pick the metric from the scale before computing α: nominal for unordered codes, ordinal for ordered categories, interval for equally spaced scores, ratio for amounts with a true zero.
Proceed with Caveat Give α under the metric you chose and under nominal distances, so readers can see how much partial credit for near misses adds.

3. Interchangeable Coders (for Generalizing Across Coders) Any of your coders could have coded any unit, and who coded which doesn't matter.

What It Means: Krippendorff's alpha pools every coder's values: it counts which values were given to the same unit, not who gave them, and works out chance from how often each value was used overall. That's what lets different coders code different, overlapping sets of units. It also means a coder who uses one value much more than the others counts as disagreement, without being picked out.
Applies To: Raters
How to Check: By design: train every coder the same way, from the same codebook, and decide in advance, ideally at random, who codes which units, so that every pair of coders shares some units. Then tabulate how often each coder used each value and name any coder who stands out.
How Much It Matters: It decides what α describes, not whether you can compute it. Alpha pools everyone's codes, so it describes how well codes from this pool of observers agree, not how well any one pair does. Reading it as the agreement of coders in general needs coders who are trained alike and whose assignments overlap on the same kinds of units: if one observer coded only quiet classrooms and another only lively ones, who coded a clip and how hard it was are tangled together. An observer who uses one code much more than the others counts against agreement, and α can't say who it is. The bootstrap interval resamples clips, not observers, so with the same four observers throughout it speaks for these four.

What to Do

Proceed with Caveat Tabulate how often each observer used each code (the data box does this) and name anyone who stands out.
Design Change Decide in advance who codes which units, at random, so every pair of coders shares some units and no coder works alone on a different kind of unit.
Alternative Test Cohen's Kappa. When the same fixed coders matter individually, report each pair's agreement on the units they share, such as Cohen's kappa for nominal codes.

4. Missingness Is Not Selective (for Generalizing α) Missing codes don't systematically leave out difficult clips or particular codes, so the clips that can be compared still represent the ones you meant to study.

What It Means: Alpha can be calculated from incomplete coding data: it uses every code that can be paired. Whether that α speaks for all the clips depends on why codes are missing. Planned gaps, where units are dealt out to coders at random before coding, are missing completely at random, and alpha stays close to right even with half the codes missing that way (Zapf et al., 2016). Gaps where a coder was given a clip and didn't code it are different: if those are the ambiguous clips, the hardest cases drop out of the comparisons. Alpha can't tell which kind of gap it is looking at.
Applies To: Observations
How to Check: By design and by record-keeping: deal out units to coders at random before coding starts, and record every assigned code that wasn't given, and why. If coders skipped assigned units, or some units ended up with only one code, compare those units with the ones that can be compared on anything you know about them.
How Much It Matters: It can matter a lot. If observers skipped the hardest clips, the codes that remain agree more than a full set would, and α comes out too high. Here the clips were dealt out at random before coding, so it holds by design, and Zapf and colleagues (2016) found alpha stayed close to right with even half the codes missing completely at random.

What to Do

Design Change Decide before coding, at random, which coders get which units, so missing codes can't depend on how hard a unit is.
Proceed with Caveat Report how many codes each unit got and why codes are missing, and compute α on the fully coded units too, to show whether the answer moves.

5. Independent Clips (for the Confidence Interval) The people (or other units) in your study don't influence each other.

What It Means: Each clip is a separate case. The codes on one clip are meant to be related; it's different clips that mustn't be linked, for example by showing the same child. Here each clip shows a different child, so the clips are separate cases.
Applies To: Observations
How to Check: By design. Does each clip show a different child, from a different moment, with nothing tying clips together? If some children appear in several clips, resample children, not clips.
How Much It Matters: Moderate. α still describes the agreement in your data, but linked clips, such as several of the same child, make the bootstrap interval too narrow, because resampling clips one at a time treats them as unrelated.

What to Do

Design Change Code one clip per child, or pick one at random from each child.
Robust Estimation Resample children rather than clips to build the interval, so each child's clips stay together.

Limitations

This Is Not Your Test If…

Two coders gave nominal codes and you care about that pair Cohen's Kappa →
You want how reliable the average of several raters is Intraclass Correlation Coefficient (ICC) →
You're checking codes against a gold standard Sensitivity and Specificity →
You're asking whether some coders rate higher than others, and every coder coded every unit Friedman Test →
You want the internal consistency of a scale's items Cronbach's Alpha →
The same people were measured twice, weeks apart Test-Retest Reliability →

What It Doesn't Tell You

Whether the codes are right. Four observers can agree and all misjudge a child. Agreement is reliability, not accuracy; checking codes against another measure of engagement, such as time on task logged by the teacher, is a separate question.
Whether it's good enough. Krippendorff's .667 and .800 are conventions from content analysis. An α of .76 might be fine for describing classrooms in a research paper and not good enough for deciding which child gets extra support.
Which observer is off. Alpha pools everyone's codes, so it can't say whether one observer drags agreement down. Each observer's code rates (the data box) or each pair's agreement on the clips they share can.
Where they disagree. One number hides the pattern. Here the coincidence matrix (Table 2) shows almost every disagreement is one step apart, spread across the scale, which fits fuzzy boundaries between neighboring codes, or observers who draw those boundaries in different places, rather than one code nobody can apply. Each observer's code rates help tell these apart. To find a weak code, recode it against all the others (say, off task vs. not) and compute nominal α for that distinction. Krippendorff (2004) recommends testing the reliability of each distinction that conclusions rest on.
Who else. The result describes these four observers, this codebook and these classrooms' mix of clips. Its interval allows for other clips, not other observers, and with a different mix, say mostly highly engaged children, expected disagreement changes and so does α.

Common Mistakes

Don'tReport “Krippendorff's alpha” without saying which metric you used.

InsteadName it (nominal, ordinal, interval or ratio). Here nominal α is .46 and ordinal α .76, because the ordinal metric counts a disagreement between neighboring codes as a small one.
Don'tPick the metric that gives the highest alpha.

InsteadChoose it from the scale before computing anything, and report nominal α beside it if you want to show what partial credit adds.
Don'tTreat .800 as a pass mark for the point estimate alone.

InsteadRead the whole interval against it. Here α = .76, but the interval runs from .65 to .84, below .667 to above .800.
Don'tDrop units with missing codes so every unit has the same number.

InsteadKeep them. Alpha uses every code that can be paired; keeping only the 30 clips all four observers coded would throw away 79 of the 199 codes.
Don'tAverage alphas across several coded variables.

InsteadReport each variable's α. Krippendorff (2004) takes the smallest as the reliability of a set of variables used together.
Don'tReport a test of alpha against zero as evidence of reliability.

InsteadReport α and its interval. Agreement beyond chance is a very low bar; reliability is about how close α comes to 1.
Don'tRead α = .76 as “the coders agreed on 76% of the units.”

InsteadRead it as how far the coders' disagreement falls short of chance-level disagreement, and give the share of matching pairs separately.

Result Decoder

Interpretation Guidelines

Benchmarks Krippendorff's Alpha (Ordinal)

α < .667 Too Unreliable
Below the lowest level Krippendorff suggests accepting, even for tentative conclusions.
.667–.799 Tentative
Good enough, by Krippendorff's convention, for tentative conclusions only.
α ≥ .800 Acceptable
The level Krippendorff suggests requiring before relying on the data for scholarly conclusions; higher when decisions about people depend on it.

Krippendorff (2004) suggested requiring α ≥ .800 before relying on coded data for scholarly conclusions, and accepting α ≥ .667 only where tentative conclusions are acceptable. He also wrote that the level should be chosen “depending on the costs of drawing invalid conclusions from these data”, set far higher when decisions about people depend on the codes, and that there are “no magical numbers”. The cut-offs are conventions from content analysis, not a verdict on your data: read the whole interval against them, not just the estimate.

More about effect size ↓

How It's Reported

Equation Form

Example

ordinal α = .76, 95% CI [.65, .84]; nominal α = .46, 95% CI [.34, .56]

Table 1. Alpha for the Same Codes under Each Distance Metric

Example

	What a Disagreement Costs	$\alpha$	95% CI
Nominal	Any mismatch counts in full	.46	[.34, .56]
Ordinal	Grows with how many codes lie between the two	.76	[.65, .84]
Interval	Grows with the squared difference, 1 to 4	.76	[.65, .84]
Ratio	Like interval, relative to the two codes' sum	.73	[.61, .82]

Note. n = 198 pairable codes from 59 clips (one clip had a single code). Intervals are percentile bootstraps from 10,000 resamples of the 60 clips (this site's own random-number generator, seed 20261010; R with set.seed(2026) gives ordinal [.651, .840] and nominal [.343, .560]). Ordinal is the metric the scale supports; the others are shown to see how much the metric matters.

Table 2. Coincidence Matrix: Pairs of Codes Given to the Same Clip

Example

	Off Task	Passive	Engaged	Highly Engaged	Total ( $n_c$ )
Off Task	43.33	12.33	3.00	0.33	59.00
Passive	12.33	23.67	9.33	2.67	48.00
Engaged	3.00	9.33	22.00	12.67	47.00
Highly Engaged	0.33	2.67	12.67	28.33	44.00
Total	59.00	48.00	47.00	44.00	198.00

Note. Each pair of codes two different observers gave the same clip is counted both ways (c–k and k–c), weighted 1 ÷ (number of codes on the clip − 1), so each code counts once. The diagonal holds matching pairs, 117.33 of 198 (59%). The margins, 59, 48, 47 and 44, are the pairable codes of each kind; the lone code on clip 54 is left out.

Full Text (APA Style)

Example

Four trained observers independently coded the engagement of one target child in each of 60 recorded classroom clips, a different child in each, as off task, passive, engaged or highly engaged. Clips were assigned at random before coding, so 199 of a possible 240 codes were given; one clip with a single code could not be compared, leaving 198 codes from 59 clips. All the observers who coded a clip gave the same code to 22 of the 59 clips, and 59% of the pairs of codes on the same clip matched exactly (Table 2). With ordinal distances, Krippendorff's α = .76, 95% CI [.65, .84] (percentile bootstrap over clips, 10,000 resamples, seeded). By Krippendorff's (2004) conventions the estimate supports tentative conclusions only, and the interval runs from below his .667 floor to above .800, so these 59 clips can't tell which convention the coding meets. Treating the codes as unordered gave α = .46, 95% CI [.34, .56], because almost every disagreement was one step apart (Table 1).

Name the metric (nominal, ordinal, interval or ratio) and say why the scale supports it. If you give a second metric, label which is the main one.
Say how many units and coders there were, how many codes were missing and why, and how units were assigned to coders.
Give the bootstrap interval and how it was made: what was resampled (units), how many resamples, and that it was seeded. Another seed or program moves the limits slightly.
Read the whole interval against the .667 and .800 conventions, and say what level your use of the codes needs.
Report each coded variable's α separately, not an average. Greek α isn't italicized.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how certain.

D_o = .067	Two codes given to the same clip were, on average, .067 apart in squared distance, on a scale where off task against highly engaged counts 1 and a single one-step miss about .1. Most disagreements were one step apart, and those cost little.
α = .76 (ordinal)	The observers avoided about three-quarters of the disagreement chance would produce, counting near misses as small disagreements. By Krippendorff's conventions that is enough for tentative conclusions, short of the .800 he suggests requiring; the interval below matters more than the label. Benchmarks ↑
95% CI [.65, .84]	For these four observers coding clips like these, the true ordinal alpha is plausibly anywhere from .653, just below the .667 floor for tentative conclusions, to .838, above the .800 Krippendorff suggests requiring. With 59 comparable clips we can't tell which convention the coding meets. The interval allows for other clips, not other observers.

Calculations

The Running Example

Example

The classroom-engagement study: four observers, working separately, coded 60 classroom clips, each showing a different child, as off task, passive, engaged or highly engaged, each clip dealt out to a randomly chosen subset of them. Most clips got two to four codes; one got a single code and couldn't count toward agreement.

There's no null hypothesis to test here. Alpha is an estimate of how well the observers agree, and its uncertainty comes from a bootstrap interval. Krippendorff (2004) points out that testing whether agreement beats chance has no bearing on reliability: no one trains four observers expecting chance-level agreement. What matters is how close α comes to 1.

What Your Data Should Look Like

One variable at a time, with every coder using the same codes, spelled the same way. Most software wants it wide: one row per coder and one column per unit (R's irr and Python's krippendorff), or the transpose, one row per unit and one column per coder (Hayes's KALPHA macro), with a blank cell wherever a coder didn't code a unit. Ours is stored long, one row per code given, with columns for the clip, the observer and the engagement code; a code nobody gave is simply a missing row, and R and Python reshape it to wide in one line. Code ordered categories as numbers in scale order (1 = off task up to 4 = highly engaged) so the software sees the order. Which coder gave which code doesn't enter the calculation; which clip it belongs to does.

Example

clip	observer	engagement
1	B	1
1	C	1
1	D	1
2	A	2
2	B	1
2	C	2
… 193 more rows

	Observer A	Observer B	Observer C	Observer D	Total
1 Off Task	11 (23%)	16 (34%)	16 (29%)	16 (33%)	59
2 Passive	16 (33%)	9 (19%)	14 (25%)	9 (18%)	48
3 Engaged	10 (21%)	11 (23%)	13 (24%)	14 (29%)	48
4 Highly Engaged	11 (23%)	11 (23%)	12 (22%)	10 (20%)	44
Total	48	47	55	49	199

Counts, with the percentage of each observer (each column adds to 100%).

Simulated data.

Download all 199 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the classroom-engagement study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Lay Out Every Code and Set Aside Clips No One Can Compare

Alpha compares codes that different observers gave the same clip, so a clip coded only once has nothing to compare. Every other code counts, however many observers saw its clip.

$n = \sum_{u:\, m_u \ge 2} m_u$ = $30 \times 4 + 20 \times 3 + 9 \times 2$

$n$ = 198 codes
$\text{codes given}$ = 199 codes

$m_u$ is how many observers coded clip $u$ . Thirty clips had all four, 20 had three and 9 had two, so 30 × 4 + 20 × 3 + 9 × 2 = 198 codes can be paired. Clip 54 had a single code and drops out, which is why 199 codes were given but $n$ = 198.
2. Count How Often Each Pair of Codes Turns Up in the Same Clip

Within each clip, pair every observer's code with every other observer's. The coincidence matrix adds those pairs up across clips, weighted so that every code counts once however many observers saw its clip. The diagonal holds the matches.

$o_{ck} = \sum_{u} \dfrac{\text{number of } c\text{–}k \text{ pairs in unit } u}{m_u - 1}$ = $\begin{array}{l|cccc|c} & \text{Off} & \text{Pass.} & \text{Eng.} & \text{High} & n_c \\ \hline \text{Off task} & 43.33 & 12.33 & 3.00 & 0.33 & 59 \\ \text{Passive} & 12.33 & 23.67 & 9.33 & 2.67 & 48 \\ \text{Engaged} & 3.00 & 9.33 & 22.00 & 12.67 & 47 \\ \text{Highly eng.} & 0.33 & 2.67 & 12.67 & 28.33 & 44 \end{array}$

$\text{matching share}$ = .59
$\text{clips all agree}$ = 22 clips

Clip 2 got passive, off task, passive and passive from $m_u$ = 4 observers: 12 ordered pairs, each worth 1/3. Its 6 passive–passive pairs add 2 to the passive–passive cell, and its 3 passive–off task pairs add 1 each way, 4 in all for its 4 codes. Clip 3 (highly engaged and passive, $m_u$ = 2) adds 1 each way. Summed over the 59 clips, 117.33 of the 198 pairs match, 59%.
3. Decide How Far Apart Two Codes Are

The codes are ordered, but nobody claims the step from off task to passive equals the step from engaged to highly engaged. The ordinal metric measures distance by how many codes lie between two values, counting half of each end, like the gap between their ranks if every code were ranked.

$\delta^2_{ck} = \dfrac{\left( \sum_{g = c}^{k} n_g - \dfrac{n_c + n_k}{2} \right)^2}{\left( n - \dfrac{n_1 + n_K}{2} \right)^2}$ = $\dfrac{\left(59 + 48 - \dfrac{59 + 48}{2}\right)^2}{\left(198 - \dfrac{59 + 44}{2}\right)^2} = \left(\dfrac{53.5}{146.5}\right)^2$

= .133

$\delta^2_{\text{pass, eng}}$ = .105
$\delta^2_{\text{eng, high}}$ = .096
$\delta^2_{\text{off, eng}}$ = .475
$\delta^2_{\text{pass, high}}$ = .403

$n_g$ are the column totals above. Off task to passive is 59 + 48 less half of each, 53.5, squared 2,862.25. The furthest pair, off task to highly engaged, is all 198 codes less half the 59 off-task and half the 44 highly engaged ones, 146.5. Every distance is divided by that largest one, so the furthest pair counts 1; dividing them all by the same number leaves α unchanged. That gives (53.5 ÷ 146.5)² = .133. The other one-step distances are .105 and .096, two steps .475 and .403. These distances come from how often each code was used in this dataset, so they aren't a fixed penalty for being one step apart: off task–passive costs .133 and engaged–highly engaged .096. In a sample that used the codes at different rates, the same labels in the same order would get different distances. If you need fixed penalties that you set for each kind of disagreement, give the codes scores you choose and use the interval metric, or, with two coders, weighted kappa with your own weights.
4. Add Up the Disagreement the Observers Actually Showed

Multiply each kind of mismatched pair by how far apart its codes are, add them up, and divide by the number of pairable codes. Matches cost nothing.

$D_o = \dfrac{1}{n} \sum_{c} \sum_{k} o_{ck}\, \delta^2_{ck}$ = $\dfrac{2 \times (12.33 \times .133 + 3 \times .475 + 0.33 \times 1 + 9.33 \times .105 + 2.67 \times .403 + 12.67 \times .096)}{198}$

= .067

$o_{ck}$ are the off-diagonal cells, counted once above the diagonal and doubled for their mirror images below it, and $\delta^2_{ck}$ the distances from step 3. The 68.67 one-step mismatches cost little each; the 0.67 off task–highly engaged pairs cost the full 1. The total, 13.36, divided by $n$ = 198 gives $D_o$ = .067.
5. Work Out the Disagreement Chance Would Produce

Now imagine the same 198 codes handed out to clips at random. Expected disagreement averages the distance over every pair of codes you could draw from the pool, two at a time (198 × 197 ordered pairs), so it depends only on how often each code was used.

$D_e = \dfrac{1}{n(n - 1)} \sum_{c} \sum_{k} n_c\, n_k\, \delta^2_{ck}$ = $\dfrac{2 \times (59 \times 48 \times .133 + 59 \times 47 \times .475 + 59 \times 44 \times 1 + 48 \times 47 \times .105 + 48 \times 44 \times .403 + 47 \times 44 \times .096)}{198 \times 197}$

= .286

$n_c$ and $n_k$ are the pairable codes of each kind, 59, 48, 47 and 44, so there are 59 × 48 off task–passive pairs in the pool, and so on. Weighted by distance and doubled for both orders, they add up to about 11,159, out of $n(n - 1)$ = 198 × 197 = 39,006 pairs: $D_e$ = .286.
6. Compare the Disagreement with What Chance Would Give

Alpha asks how much of chance-level disagreement the observers avoided. If they disagreed as much as random codes would, α is 0; if they never disagreed, it's 1.

$\alpha = 1 - \dfrac{D_o}{D_e}$ = $1 - \dfrac{.0675}{.2861}$

= .76

$D_o$ = .0675 is the disagreement the observers showed and $D_e$ = .2861 what chance would give. .0675 ÷ .2861 = .236, so they showed about a quarter of chance-level disagreement, and α = 1 − .236 = .76.
7. Measure It Again as If the Codes Had No Order

The metric is part of the answer. With nominal distances, passive against engaged costs as much as off task against highly engaged, so the same codes give a much lower alpha when most disagreements are near misses.

$\alpha = 1 - \dfrac{D_o}{D_e}$ = $1 - \dfrac{.407}{.750}$

= .46

$\alpha\text{, interval}$ = .76
$\alpha\text{, ratio}$ = .73

Nominal $D_o$ = .407 is simply the share of pairs that don't match (1 − .593), and $D_e$ = .750 the share that wouldn't match by chance. 1 − .407 ÷ .750 = .46. Interval distances give .76 and ratio distances .73 (Table 1); interval matches ordinal here because the four codes were used about equally often.
8. Give a Range of Plausible Values for the True Alpha

Alpha has no simple formula for its uncertainty, so we bootstrap it: draw 60 clips at random with replacement, each keeping all its codes, recompute alpha, and repeat. The middle 95% of those alphas is the interval. Resampling whole clips keeps each clip's codes together, which gave intervals much closer to the stated 95% than resampling pairs in Zapf and colleagues' (2016) simulations.

$\left[\, \alpha^{*}_{(2.5\%)},\; \alpha^{*}_{(97.5\%)} \,\right]$ = $\alpha^{*}_{(2.5\%)} = .653, \quad \alpha^{*}_{(97.5\%)} = .838$

= [.65, .84]

$\text{share of } \alpha^{*} < .800$ = .82
$\text{share of } \alpha^{*} < .667$ = .04

$\alpha^{*}$ is the ordinal alpha from one resample of clips; across 10,000 seeded resamples, 2.5% fell below .653 and 2.5% above .838. In 82% of the resamples α was below .800, and in 4% below .667, a rough guide to how far the data are from each convention (Hayes & Krippendorff, 2007). That 82% describes the resamples; it isn't an 82% probability that the true alpha is below .800.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

SPSS has no menu for Krippendorff's alpha. Download Andrew Hayes's free KALPHA macro (from his website, afhayes.com), open it in a syntax window and run it once to define the KALPHA command.
KALPHA wants one row per unit and one column per coder, with a blank (system-missing) cell where a coder didn't code a unit. Our file is long, so first use Data → Restructure → Restructure selected cases into variables, with clip as the Identifier Variable and observer as the Index Variable. You get one row per clip with columns A to D.
Run the KALPHA command below. level = 2 asks for the ordinal metric (1 nominal, 3 interval, 4 ratio). It leaves out KALPHA's boot option, whose intervals come out too narrow (see Reading the Output); get the interval from R or Python instead.

Or paste this syntax:

SORT CASES BY clip observer.
CASESTOVARS
  /ID=clip
  /INDEX=observer.

KALPHA judges = A B C D/level = 2/detail = 1.

Reading the Output

KALPHA prints alpha for the metric you asked for, here ordinal .764, with the number of units, coders and pairs of codes. Pairs counts the pairs of codes within clips, 249 here, not the 198 pairable codes. With detail = 1 it adds the coincidence matrix (Table 2 above) and the distances, unstandardized (2862.25 for off task against passive, where step 3 shows .133 after dividing by the largest). Read Alpha first. Rerun with level = 1 for the nominal alpha (.457).

If you add boot, don't report KALPHA's confidence limits or its q values (the share of bootstrap alphas below each of several minimums). KALPHA resamples pairs of codes as if each pair were independent (Hayes & Krippendorff, 2007), although the pairs on one clip belong together, so its limits come out too narrow: mimicking its resampling on these data gives about .71 to .81, against .65 to .84 when whole clips are resampled. In Zapf and colleagues' (2016) simulations, intervals built Krippendorff's way covered the true alpha only about 60% of the time (the median across their settings), not 95%, while their bootstrap over whole units came much closer to 95%. Report α from KALPHA and take its interval from a bootstrap over units, as in the R or Python code below.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

library(irr)

codes <- read.csv("data.csv")

# One row per observer, one column per clip; a code nobody gave is NA
wide <- reshape(codes, idvar = "observer", timevar = "clip", direction = "wide")
ratings <- as.matrix(wide[, -1])

kripp.alpha(ratings, method = "ordinal")
kripp.alpha(ratings, method = "nominal")$value

# 95% interval: resample clips (columns) with replacement
set.seed(2026)
boot_alpha <- replicate(10000, {
  pick <- sample(ncol(ratings), replace = TRUE)
  kripp.alpha(ratings[, pick], method = "ordinal")$value
})
quantile(boot_alpha, c(0.025, 0.975))

 Krippendorff's alpha

 Subjects = 60
   Raters = 4
    alpha = 0.764
[1] 0.457152
     2.5%     97.5%
0.6509196 0.8401723

irr's kripp.alpha wants one row per coder and one column per unit, with NA for missing codes, and method sets the metric ("nominal", "ordinal", "interval" or "ratio"). It gives α only: ordinal .764 and nominal .457, the same as this page. It has no interval, so the code bootstraps one by resampling clips. R's random numbers differ from the generator this page uses, so its limits, [.651, .840], differ slightly from ours, [.653, .838]; both round to [.65, .84].

Output from a real run: R 4.6.1 (irr 0.85), run 2026-10-10.

Python

import numpy as np
import pandas as pd
import krippendorff

codes = pd.read_csv("data.csv")

# One row per observer, one column per clip; a code nobody gave is NaN
wide = codes.pivot(index="observer", columns="clip", values="engagement")
ratings = wide.to_numpy(dtype=float)

for level in ["ordinal", "nominal", "interval"]:
    print(level, krippendorff.alpha(reliability_data=ratings, level_of_measurement=level))

# 95% interval: resample clips (columns) with replacement
rng = np.random.default_rng(2026)
boot = [krippendorff.alpha(reliability_data=ratings[:, rng.integers(0, ratings.shape[1], ratings.shape[1])],
                           level_of_measurement="ordinal", value_domain=[1, 2, 3, 4])
        for _ in range(10000)]
print(np.percentile(boot, [2.5, 97.5]))

ordinal 0.7640822380138615
nominal 0.457151966477648
interval 0.7638833399920095
[0.64911132 0.83938106]

The krippendorff package (pip install krippendorff) takes the same coders × units layout, with NaN for missing codes, and level_of_measurement sets the metric. Its alphas match this page exactly. value_domain lists every possible code, so a resample that happens to miss one still uses the right ordinal distances. Its bootstrap limits, [.649, .839], differ slightly from ours because numpy draws different resamples; they round to [.65, .84].

Output from a real run: Python 3.9.6 (krippendorff 0.8.1, numpy 2.0.2, pandas 2.3.3), run 2026-10-10.

Impact

Effect Size

Alpha is this page's measure of size. It compares disagreement rather than agreement: the disagreement the coders showed against the disagreement chance would produce, so 0 means chance level and 1 means every pair of codes on every unit matches. Below 0 means the coders disagreed more than chance would, which points to systematic disagreement or a very small sample. If every code is the same, chance disagreement is 0 and α can't be computed. It has no units, so values can be compared across coding schemes, but it still depends on the metric, on how often each code was used and on the mix of units coded.

\alpha = 1 - \dfrac{D_o}{D_e} = 1 - \dfrac{.0675}{.2861} = .76

Two codes on the same clip were, on average, .067 apart in squared distance, on a scale where off task against highly engaged counts 1 and a single one-step miss about .1; codes handed out at random would be .286 apart. The observers avoided about three-quarters of chance-level disagreement, so α = .76: enough for tentative conclusions by Krippendorff's convention, short of .800.

The metric is part of the answer. Counting every mismatch in full, the same codes give α = .46, because 59% of pairs matched exactly but nearly all the rest were one step apart. When disagreements follow the scale's order, an ordinal or interval alpha is higher than the nominal one (Hayes & Krippendorff, 2007). When disagreements are scattered across the scale, the advantage shrinks or disappears.

Here interval and ordinal alpha agree (.76 and .76) because the four codes were used about equally often (59, 48, 47 and 44 pairable codes), so the ordinal distances between neighbors (.133, .105 and .096) are close to equal steps. If almost every clip had been coded passive, the ordinal distances would stretch and shrink with those counts and the two could differ.

Like kappa, alpha depends on how common each code is, not just on how well the coders agree: when one code dominates, expected disagreement is small and a few real disagreements pull α down. Krippendorff (2004) suggested requiring α ≥ .800, and accepting α ≥ .667 only for tentative conclusions, but also wrote that the level should depend on the costs of wrong conclusions and that there are no magical numbers. Whether codes that put two observers one step apart on about a third of pairs are consistent enough depends on what they will be used for.

Benchmarks Krippendorff's Alpha (Ordinal) at a Glance

Too Unreliable

α < .667

Tentative

.667–.799

Acceptable

α ≥ .800

Full benchmarks and caveats ↑

Uncertainty

This page gives alpha without a p-value. A test could ask whether α is above 0, but Krippendorff (2004) points out that testing whether agreement beats chance has no bearing on reliability, which is about how far agreement falls short of perfect. Nobody trains four observers expecting chance-level agreement, and with a reasonable number of units almost any real agreement beats zero, including agreement far too poor to use.

Our 95% interval, .65 to .84, is nowhere near 0, so a test against zero would come out overwhelmingly significant and tell us nothing we didn't know. The useful facts are that the interval's lower end, .653, sits just under the .667 convention and its upper end, .838, above .800.

The question that matters is how high α is and how precisely it's estimated, which is what the bootstrap interval answers. Hayes and Krippendorff (2007) also read the share of bootstrap alphas below a required minimum as a rough risk of falling short of it: here 82% of resamples fell below .800 and 4% below .667. That share depends on the resampling; it isn't a p-value, and it isn't the probability that the true alpha is below .800. A p, if you report one, is not the probability that the coders agree only by chance, and it says nothing about whether their agreement is good enough.

Precision and Sample Size

Power is the chance that a study detects an effect that's really there. For alpha, a test of agreement better than chance is usually a low bar, though power can still matter with few cases, a rare code, or a test against a minimum other than zero. The more useful planning question is usually precision: how many units you need for the interval around α to be narrow enough to tell, say, tentative from acceptable agreement.

There's no simple formula for alpha's standard error. The bootstrap's spread shrinks roughly with the square root of the number of units, and it also depends on how many coders see each unit, how often each code is used and how the disagreements fall, so a pilot study or an earlier study with a similar coding scheme gives a workable guess.

Our 10,000 bootstrap alphas, each from a resample of all 60 clips, have an SD of 0.047. Only 59 of those clips can be compared (one has a single code), so 59 is the effective sample size, and the SD works out to roughly √(0.133 ÷ N), where N is the number of comparable clips. With a similar coding plan, an interval of about ±.10 would take around 51 clips, and ±.05 about 204. Ours, with 59, runs about ±.09, which is why it can't separate .667 from .800.

Plan an alpha study around the width of the interval you need, with the coders per unit and code mix you expect, and with enough units that the whole interval clears the level your use of the codes requires. The planning arithmetic above is our own, a normal approximation from our bootstrap; with a skewed bootstrap distribution, or α near 1, simulate the planned design instead.

Other Info

Follow-Up and Post Hoc Tests

None. This page estimates rather than tests, so there are no follow-up comparisons to run.

Related Methods

Alternative MeasureFleiss' Kappa: Fleiss' kappa answers the same question for nominal codes when every case has the same number of codes. With nominal codes and nothing missing the two come out nearly the same (Zapf et al., 2016); here, on the 30 clips all four observers coded, Fleiss' κ is .45 and nominal α is .45. Alpha also handles missing codes, different numbers of coders per case and ordered or numeric codes.
Alternative MeasureWeighted Kappa: Weighted kappa gives two raters partial credit for near misses on an ordered scale, with weights you choose. Ordinal or interval alpha does the same job for any number of coders, with distances set by the metric, and it works out chance from the pooled codes rather than from each rater's own rates.
Related DesignCohen's Kappa: With two coders and nominal codes, alpha comes close to Scott's pi in large samples (Hayes & Krippendorff, 2007), not to Cohen's kappa: alpha works out chance from both coders' codes pooled, while Cohen's kappa keeps each coder's own rates. The two differ little unless the coders use the codes at very different rates.
Alternative MeasureIntraclass Correlation Coefficient (ICC): For numeric ratings, interval alpha is an agreement coefficient: with two coders and no missing ratings it comes out very close to a one-way, single-rater intraclass correlation (Hayes & Krippendorff, 2007), and like a single-rater ICC it describes one coder's ratings, not the average of several. Use the ICC when you want the reliability of averaged ratings or a fully crossed design's variance components.

References

Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1), 61–70. https://doi.org/10.1177/001316447003000105
Krippendorff, K. (2011). Computing Krippendorff's alpha-reliability. Departmental Papers (ASC), 43. Annenberg School for Communication, University of Pennsylvania. https://repository.upenn.edu/asc_papers/43
“the margins of coincidence matrices do not represent all values that occur in a reliability data matrix, only those that can be paired within units” (C. Nominal data, any number of observers, missing data)

“Ordinal differences depend on their frequencies of using values.” (D. Any metric, any number of observers, missing data)

“Any metric or level of measurement (nominal, ordinal, interval, ratio, and more)” (p. 1)
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. https://doi.org/10.1080/19312450709336664
“Alpha also allows for reliability data with missing categories or scale points, a frequent reality that none of the reviewed measures has been able to cope with.” (Criteria for a Good Measure of Reliability)

“the units of bootstrapping for reliability are the pairs of judgments associated with particular units” (SPSS and SAS Implementation)
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. https://doi.org/10.1111/j.1468-2958.2004.tb00738.x
“Testing the null-hypothesis that observed agreement deviates from chance has no bearing on reliability, which concerns deviations from perfect agreement or 1.000.” (Recommendations (iii))

“the reliability of each variable should be measured and the smallest among them should be taken as the reliability of the whole system” (Recommendations (iv))

“An acceptable level of agreement below which data are to be rejected as too unreliable must be chosen depending on the costs of drawing invalid conclusions from these data.” (Recommendations (iii))

“Except for perfect agreement, there are no magical numbers, however.” (Recommendations (iii))

“Coders must be interchangeable, may code different subsamples of data, provided there is enough duplication or overlap.” (Recommendations (i))
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016). Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate? BMC Medical Research Methodology, 16, Article 93. https://doi.org/10.1186/s12874-016-0200-9
“In the case of missing data or data or higher than nominal order, Krippendorff’s alpha is recommended.” (Abstract, Conclusion)

“Krippendorff’s alpha was very robust against missing values, even if 50 % of the values were missing.” (Results)
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552
Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40(3), 879–891. https://doi.org/10.3758/BRM.40.3.879
“percentile bootstrap CIs can be asymmetrical because they are based on an empirical estimation of the sampling distribution of the indirect effect, rather than on an assumption that the sampling distribution is normal” (p. 883)

“We recommend at least 5,000 resamples for final reporting, although 1,000 is probably sufficient for preliminary analyses.” (p. 889, note 5)
Halekoh, U., Højsgaard, S., & Yan, J. (2006). The R package geepack for generalized estimating equations. Journal of Statistical Software, 15(2), 1–11. https://doi.org/10.18637/jss.v015.i02
“In case of missing values, the GEE estimates are consistent if the values are missing completely at random (Rubin 1976).” (p. 6)