Weighted Kappa: Interpretation and APA Reporting

The Basics

When to Use It

Use it when two raters each place the same cases on the same ordered scale, working separately, and a near miss should count as partial agreement rather than a full disagreement. The same works for one rater rating the same cases twice. For example:

Do two clinicians agree on how severe each patient's depressive symptoms are (none, mild, moderate or severe)?
Do two teachers agree on which of four reading levels each child's recorded passage shows?
Do two coders agree on how hostile each couple's conflict discussion is, rated low, medium or high?

With unordered categories, such as depression, anxiety or neither, use Cohen's kappa (Cohen, 1960). With three or more raters on an ordered scale, there are two common options: an intraclass correlation on the ratings scored 1 to k, which treats the steps between categories as equal, or Krippendorff's alpha with ordinal distances, which doesn't, and also copes with missing ratings. Whether equal steps are fair for your scale is a useful first question, but the rating design and the kind of agreement you want matter too; each page covers its own. Fleiss' kappa (Fleiss, 1971) counts every disagreement as a full miss, so it suits three or more raters with unordered codes. If your ratings are scores on a long numeric scale, an intraclass correlation fits better.

Vocabulary You Need

Definition Ordinal Variable

A variable whose values can be put in order from less to more, such as a disagree–neutral–agree rating, symptom severity (mild, moderate, severe) or year in school, without the steps between them necessarily being equal. The order carries information that a nominal variable such as college major doesn't have, which is why ordinal measures like Spearman's ρ, Kendall's τ and Goodman and Kruskal's γ use it; treating the categories as equally spaced numbers adds an assumption. Glossary →

Definition Contingency Table

A table that cross-classifies people on two categorical variables, one along the rows and one along the columns, so each cell counts the people with that combination. The row and column totals (the margins) show how each variable is spread on its own; the cells show how the two go together. Tables are described by their size, rows × columns, such as a 2 × 3 table. Glossary →

Definition Disagreement Weights (

v_{ij}

)

Numbers that set how serious each possible disagreement is in weighted kappa, one for every cell of the raters' table: 0 on the diagonal, where the raters agree, up to 1 for the furthest-apart pair of ratings. Linear weights rise in equal steps with the distance between the categories, |i − j| / (k − 1); quadratic weights use the squared distance, so on a four-point scale a one-step miss costs 1/9 instead of 1/3. One minus a disagreement weight is the credit, or agreement weight, that cell earns. Any weights between 0 and 1 can be used (Fleiss, Cohen & Everitt, 1969), as long as they're set before the data are seen; linear and quadratic are the usual choices, and both treat the categories as equally spaced (van Oest & Moss, 2026). Glossary →

Definition Chance Agreement (

p_e

)

How often raters would be expected to agree if they assigned categories independently, at the rates the categories were actually used. For two raters (Cohen's kappa), it's a share of cases: multiply the share of cases the first rater put in each category by the second rater's share, then add up across categories. With several raters (Fleiss' kappa), it's the chance that two ratings picked at random agree: square each category's overall share and add them up. “Chance” doesn't mean the raters guess: their category rates can reflect real judgment. It's the agreement a model of unrelated codes predicts from those rates. It is high when one category dominates: raters who all call most cases “neither” will often agree even if their codes are unrelated. Glossary →

Assumptions

1. Independent Raters Each rater rates each case they're given without knowing anyone else's ratings.

What It Means: The raters reach their ratings separately: they don't discuss cases, see each other's ratings or share notes before rating. Agreement statistics such as kappa, Krippendorff's alpha and the intraclass correlation treat each rating as a separate judgment. Raters who confer agree more than the rating procedure would on its own, so the result overstates how reliable it is.
Applies To: Raters
How to Check: By design, not by a test. Did each rater work from the same material (a recording, a transcript, the same case file) without access to anyone else's ratings, and were disagreements resolved only after all had finished?
How Much It Matters: A lot, and always in the same direction. Clinicians who talk patients over or see each other's ratings agree more than the rating scale would on its own, so κ_w comes out too high. No statistic can undo it afterward.

If It's Violated

Design Change Have each rater work from the same recording, notes or interview without seeing the other's rating, and settle disagreements only after both have finished.
Proceed with Caveat If some cases were discussed before rating, compute κ_w from the independent ratings only, or say plainly that it overstates independent agreement.

2. Independent Observations The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: Moderate. κ_w still describes the agreement in your table, but linked cases, such as several visits by the same patient, make the standard error too small, so the interval comes out too narrow.

If It's Violated

Design Change Rate one visit per patient, or pick one at random from each patient.
Robust Estimation Resample people rather than visits to build the interval, so the clustering is kept intact.

3. Ordered Rating Scale The rating categories have a natural order, and every rater uses the same ordered set.

What It Means: The categories run from less to more of something (none < mild < moderate < severe), and every rater chooses from the same list, so the distance between two ratings, how many steps apart they are, means something. Weighted agreement statistics give partial credit by that distance; for unordered codes any distance, and so any weights, would be arbitrary.
Applies To: Raters
How to Check: By design: confirm the categories are ordered and that every rater used the same list. Then check that your software sees the categories in scale order, not alphabetical order, by coding them 1 to k first.
How Much It Matters: It decides whether weighting makes sense at all. Partial credit only means something if “one step apart” means “nearly the same.” For unordered codes (depression, anxiety, neither) any weights you pick are arbitrary, and both raters must use the same ordered scale, so the table is square.

If It's Violated

Alternative Test Cohen's Kappa. For unordered categories, use unweighted kappa, which counts every disagreement the same.

4. Meaningful Disagreement Weights The weights match how serious each kind of disagreement is.

What It Means: Weighted kappa works with any weights. The standard linear and quadratic weights set a disagreement's cost from how many steps apart the two ratings are, which treats the categories as equally spaced: a none–mild miss costs the same as a moderate–severe one. If some steps are clinically much bigger than others, those weights give the wrong credit to some near misses, and custom weights that reflect the real costs fit better.
Applies To: Raters
How to Check: By reasoning about the scale, not by a test. Ask whether some steps between categories are clinically much larger than others; if they are, write down custom weights before rating starts.
How Much It Matters: Some. Linear and quadratic weights treat the categories as equally spaced, so a none–mild miss costs the same as a moderate–severe one (van Oest & Moss, 2026). If some steps are much bigger than others, κ_w gives the wrong credit to the near misses on those steps; reporting both weightings shows how much the weights matter.

If It's Violated

Correction Weighted kappa works with any weights between 0 and 1 (Fleiss, Cohen & Everitt, 1969). Write down a cost for each kind of disagreement from clinical reasoning before rating starts, and report them.
Proceed with Caveat Report linear and quadratic κ_w (and the unweighted κ), so readers can see how much the answer depends on the weights.

Limitations

This Is Not Your Test If…

Your categories have no natural order Cohen's Kappa →
Three or more raters used an ordered scale whose steps you can treat as equal Intraclass Correlation Coefficient (ICC) →
Three or more raters used an ordered scale with unequal steps or missing ratings Krippendorff's Alpha →
Your ratings are scores on a long numeric scale Intraclass Correlation Coefficient (ICC) →
You're checking ratings against a gold standard Sensitivity and Specificity →
You're asking whether one rater rates higher overall Wilcoxon Signed-Rank Test →
You're asking whether two different ordered variables are related Goodman and Kruskal's Gamma →

What It Doesn't Tell You

Whether the ratings are right. Two clinicians can agree and both misjudge severity. Agreement is reliability, not accuracy; checking ratings against a structured diagnostic interview is a separate question.
Which weights are right. κ_w is .70 with linear weights and .79 with quadratic ones from the same 120 patients. Statistics can't say whether a mild–moderate miss is worth two-thirds or eight-ninths agreement; that's a clinical judgment, made before rating.
Who else. The result describes these two clinicians, this scale and this clinic's mix of patients. With a different mix of patients, say mostly severe cases, the same clinicians could get a noticeably different κ_w, and on a scale with more categories quadratic κ_w tends to rise (Brenner & Kliebsch, 1996).
Whether one rater runs high. A clinician who rates every patient one step more severe than the other still earns most of the credit (2/3 per patient with linear weights, 8/9 with quadratic), so κ_w can look respectable while hiding a systematic shift. Similar totals for each rating are a first check: here 30, 40, 30 and 20 against 28, 42, 31 and 19. But totals can hide a shift, so look at which way the disagreements go too: here clinician A rated higher on 18 of the 36 and clinician B on the other 18, and their average ratings on the 1-to-4 scale are 2.33 and 2.34, so neither runs high. Lopsided totals matter too: when both raters put most patients in one rating, chance agreement is high, and the same exact agreement gives a lower κ_w.
Where they disagree. One number hides the pattern. Here the table shows almost every disagreement is one step apart, spread across the scale, which points to fuzzy boundaries between neighboring categories rather than one bad category.

Common Mistakes

Don'tReport “weighted kappa” without saying which weights you used.

InsteadName them (linear or quadratic). Here the same table gives .70 with one and .79 with the other.
Don'tPick the weights after seeing which gives the highest kappa.

InsteadChoose them before rating starts, from how serious each kind of disagreement is. Quadratic weights usually give the highest value (van Oest & Moss, 2026).
Don'tCompare weighted kappas from scales with different numbers of categories.

InsteadCompare them only with the same scale and weights. Quadratic κ_w tends to rise as categories are added (Brenner & Kliebsch, 1996).
Don'tLet software put text ratings in alphabetical order.

InsteadCode the ratings 1 to k in scale order first. Sorted alphabetically, mild comes before none, and our linear κ_w comes out .58 instead of .70.
Don'tUse weighted kappa for categories with no natural order.

InsteadUse Cohen's kappa, which treats every disagreement as equally serious.
Don'tRead κw = .70 as “the raters agreed on 70% of cases.”

InsteadRead it as the share of possible agreement beyond chance with near misses partly credited, and give the exact agreement separately.

Result Decoder

Interpretation Guidelines

Benchmarks Weighted Kappa (

\kappa_w

)

κw < .00 Poor
The raters' ratings are further apart than chance would produce.
.00–.20 Slight
Barely more agreement than chance, even with near misses credited.
.21–.40 Fair
Some agreement beyond chance, with near misses credited, but well short of what was possible.
.41–.60 Moderate
About half the possible agreement beyond chance.
.61–.80 Substantial
Most of the possible agreement beyond chance, with near misses credited.
.81–1.00 Almost Perfect
Nearly all the possible agreement beyond chance, with near misses credited.

Landis and Koch (1977) proposed these labels for kappa while discussing one example and called the divisions “clearly arbitrary”; they are borrowed here for weighted kappa. The same value means different things with different weights: quadratic weights usually give the highest values (van Oest & Moss, 2026), and quadratic κ_w tends to rise with the number of categories (Brenner & Kliebsch, 1996). Compare weighted kappas only with the same weights and scale, and read the interval, not just the estimate.

More about effect size ↓

How It's Reported

Equation Form

Example

linear κ_w = .70, SE = .047, 95% CI [.60, .79]; quadratic κ_w = .79, 95% CI [.71, .88]; unweighted κ = .59, 95% CI [.48, .70]

Table 1. Agreement between the Two Clinicians, by Weighting

Example

	$p_{o(w)}$	$p_{e(w)}$	$\kappa$	$SE$	95% CI
Unweighted (Cohen's kappa)	.700	.266	.59	.057	[.48, .70]
Linear weights	.886	.625	.70	.047	[.60, .79]
Quadratic weights	.953	.771	.79	.041	[.71, .88]

Note. N = 120 patients rated none, mild, moderate or severe. p_o(w) and p_e(w) are the observed and chance agreement once near misses earn partial credit; unweighted, they count exact matches only. Linear weights give a one-step miss 2/3 credit and a two-step miss 1/3; quadratic weights give 8/9 and 5/9. Standard errors from Fleiss, Cohen and Everitt (1969); 95% CI = κ ± 1.96 × SE.

Full Text (APA Style)

Example

The two clinicians gave the same severity rating to 84 of the 120 patients (70%), and 31 of the 36 disagreements were one step apart on the four-point scale. With linear weights, which give partial credit for near misses, agreement beyond chance was substantial by Landis and Koch's (1977) benchmarks, weighted κ_w = .70, SE = .047, 95% CI [.60, .79], although the interval reaches down into the moderate range. Quadratic weights, which forgive near misses more, gave κ_w = .79, 95% CI [.71, .88]. Counting every disagreement as a full miss gave Cohen's κ = .59, 95% CI [.48, .70] (Table 1).

Say which weights you used (linear or quadratic), and that you chose them before seeing the data. The same table gives different weighted kappas with different weights.
Give the table, or at least the exact agreement and how far apart the disagreements were, so readers can see what the weights did.
Give each rater's totals for each rating (the table's margins), so readers can see whether one rater runs high.
Report the standard error and 95% interval, and read the whole interval against the benchmarks.
Greek κ isn't italicized; SE and p are. If you report the unweighted κ too, label it clearly.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how certain.

p_o(w) = .886	With linear weights, the clinicians earned 89% of the credit available. Exact agreement alone was 70%; the near misses make up the rest.
κ_w = .70 (linear)	The clinicians reached 70% of the agreement beyond chance that was possible, with one-step misses counted as two-thirds agreement. That's substantial by the Landis and Koch labels above. Benchmarks ↑
95% CI [.60, .79]	The true linear κ_w is plausibly anywhere from .60, the top of moderate, to .79, near the top of substantial. With 120 patients we can't quite rule out moderate agreement.
SE = .047	If the study were repeated with another 120 patients like these, the linear κ_w would typically vary by about .05 from study to study.

Calculations

The Running Example

Example

The severity-rating study: two clinicians, working separately, each rated the same 120 patients' symptoms as none, mild, moderate or severe.

This page doesn't test a null hypothesis. One can be written, κ_w = 0, but the goal here is to estimate how well the clinicians agree, and its uncertainty comes from a confidence interval. Software can test whether κ_w is above 0, but no one trains two clinicians expecting chance-level agreement, so that test would tell us almost nothing.

What Your Data Should Look Like

One row per case and one column per rater, with both columns using exactly the same ratings, spelled the same way. Software counts the rows into a square table of the first rater's ratings by the second's, and the weights need the categories in their real order (none, mild, moderate, severe), not alphabetical order; code them as numbers 1 to k, where k is the number of categories, or tell the software the order. A rating only one rater used still needs its own row and column.

Example

patient	clinician_a	clinician_b
1	severe	severe
2	mild	mild
3	moderate	moderate
4	none	none
5	mild	severe
6	none	none
… 114 more rows

	B: None	B: Mild	B: Moderate	B: Severe	Total
A: None	22 (73%)	6 (20%)	2 (7%)	0 (0%)	30
A: Mild	5 (13%)	29 (73%)	5 (13%)	1 (3%)	40
A: Moderate	1 (3%)	6 (20%)	19 (63%)	4 (13%)	30
A: Severe	0 (0%)	1 (5%)	5 (25%)	14 (70%)	20
Total	28	42	31	19	120

Counts, with each row's percentages.

Simulated data.

Download all 120 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the severity-rating study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Count How Often the Two Clinicians Gave Each Pair of Ratings

Each patient lands in one cell of a 4 × 4 table, depending on the rating each clinician gave. The diagonal holds the exact agreements; the cells next to it are near misses, and the corners are the worst disagreements.

$\begin{array}{l|cccc} & \text{B: None} & \text{B: Mild} & \text{B: Mod.} & \text{B: Sev.} \\ \hline \text{A: None} & n_{11} & n_{12} & n_{13} & n_{14} \\ \text{A: Mild} & n_{21} & n_{22} & n_{23} & n_{24} \\ \text{A: Mod.} & n_{31} & n_{32} & n_{33} & n_{34} \\ \text{A: Sev.} & n_{41} & n_{42} & n_{43} & n_{44} \end{array}$ = $\begin{array}{l|cccc} & \text{None} & \text{Mild} & \text{Mod.} & \text{Sev.} \\ \hline \text{None} & 22 & 6 & 2 & 0 \\ \text{Mild} & 5 & 29 & 5 & 1 \\ \text{Mod.} & 1 & 6 & 19 & 4 \\ \text{Sev.} & 0 & 1 & 5 & 14 \end{array}$

$\text{exact agreements}$ = 84 patients
$\text{one step apart}$ = 31 patients
$\text{two steps apart}$ = 5 patients

The diagonal, 22 + 29 + 19 + 14, holds 84 exact agreements. Of the 36 disagreements, 31 sit right next to the diagonal, one step apart, 5 are two steps apart, and none sets none against severe. Clinician A's rows add up to 30, 40, 30 and 20; clinician B's columns to 28, 42, 31 and 19.
2. Decide How Much a Near Miss Should Cost

This is the step that makes kappa weighted. Before looking at the data, give every kind of disagreement a cost, from 0 for an exact match to 1 for the furthest-apart ratings. A pair of ratings then earns credit of 1 minus its cost.

$v_{ij} = \dfrac{|i - j|}{k - 1} \;\;\text{(linear)}, \qquad v_{ij} = \left(\dfrac{i - j}{k - 1}\right)^2 \;\;\text{(quadratic)}$ = $\text{linear: } 0, \tfrac{1}{3}, \tfrac{2}{3}, 1 \qquad \text{quadratic: } 0, \tfrac{1}{9}, \tfrac{4}{9}, 1$

$v_{\text{1 step, linear}}$ = .333
$v_{\text{1 step, quadratic}}$ = .111

Number the categories 1 to $k$ (here $k$ = 4): $i$ is clinician A's rating and $j$ is clinician B's, so $|i - j|$ is how many steps apart they are and $k - 1 = 3$ is the most they can be. With linear weights a one-step miss costs 1/3, so it keeps 2/3 credit; with quadratic weights it costs only (1/3)² = 1/9 and keeps 8/9. A two-step miss keeps 1/3 or 5/9.
3. Add Up the Credit the Clinicians Earned

Instead of counting only exact matches, give every patient the credit their pair of ratings earns and average it. Near misses now count for something.

$p_{o(w)} = \sum_i \sum_j w_{ij}\, p_{ij}, \qquad w_{ij} = 1 - v_{ij}$ = $\dfrac{84 \times 1 + 31 \times \tfrac{2}{3} + 5 \times \tfrac{1}{3}}{120}$

= .886

$p_{o(w)}\text{, quadratic}$ = .953

With linear weights, the 84 exact agreements earn full credit, the 31 one-step misses 2/3 each (20.67) and the 5 two-step misses 1/3 each (1.67). That's 106.33 points out of 120, so $p_{o(w)}$ = .886. Quadratic weights give 114.33 points, .953.
4. Work Out How Much Credit Chance Alone Would Earn

Two clinicians who each call most patients mild or moderate would land close to each other often even if they rated at random. Chance agreement counts that credit from each clinician's own mix of ratings.

$p_{e(w)} = \sum_i \sum_j w_{ij}\, p_{i\cdot}\, p_{\cdot j}$ = $\dfrac{31.9 \times 1 + 50.6 \times \tfrac{2}{3} + 28.1 \times \tfrac{1}{3} + 9.4 \times 0}{120}$

= .625

$p_{e(w)}\text{, quadratic}$ = .771

$p_{i\cdot}\,p_{\cdot j}$ is the share of patients who would land in each cell if the two clinicians rated independently at their own rates: none–none, for instance, is 30/120 × 28/120 = .058, or 7.0 patients. Because near misses earn credit, chance agreement adds up every cell, not just the diagonal. Grouped by how far apart the ratings are, chance alone would have the clinicians match on about 31.9 patients, land one step apart on 50.6, two steps on 28.1 and three on 9.4. Crediting those as in step 3 gives $p_{e(w)}$ = .625 with linear weights and .771 with quadratic ones. The clinicians actually got 84, 31, 5 and 0: far more exact matches, and far fewer big misses, than chance.
5. Measure Agreement beyond Chance with Linear Weights

This is Cohen's kappa formula with credit in place of exact matches. Take the credit beyond chance the clinicians earned and divide it by the credit beyond chance there was room for.

$\kappa_w = \dfrac{p_{o(w)} - p_{e(w)}}{1 - p_{e(w)}}$ = $\dfrac{.886 - .625}{1 - .625} = \dfrac{.261}{.375}$

= .70

$p_{o(w)} - p_{e(w)}$ = .886 − .625 = .261 is how far the clinicians got beyond chance. $1 - p_{e(w)}$ = .375 is how far they could have got. .261 ÷ .375 = .70.
6. Do the Same with Quadratic Weights

Quadratic weights charge for a disagreement by its squared distance, so near misses cost much less than far ones, and the same table gives a different answer. Here it's higher, because nearly every disagreement was a near miss.

$\kappa_w = \dfrac{p_{o(w)} - p_{e(w)}}{1 - p_{e(w)}}$ = $\dfrac{.953 - .771}{1 - .771} = \dfrac{.182}{.229}$

= .79

$p_{o(w)}$ = .953 and $p_{e(w)}$ = .771 with quadratic weights. .182 ÷ .229 = .79, noticeably higher than the linear .70 from the same 120 patients.
7. Compare with Kappa That Counts Only Exact Matches

Unweighted kappa gives no credit for a near miss, so a mild–moderate disagreement counts the same as none–severe. It's the baseline weighted kappa improves on for ordered ratings.

$\kappa = \dfrac{p_o - p_e}{1 - p_e}$ = $\dfrac{.700 - .266}{1 - .266} = \dfrac{.434}{.734}$

= .59

$p_o$ = .700
$p_e$ = .266

$p_o$ = 84 ÷ 120 = .700 exact agreements, and $p_e$ = .266 by chance. .434 ÷ .734 = .59: moderate, where the weighted versions said substantial, because it treats the 31 one-step misses as complete failures.
8. Figure Out How Precisely Weighted Kappa Is Estimated

A different 120 patients would give a somewhat different weighted kappa. The standard error measures how much; this is Fleiss, Cohen and Everitt's (1969) formula, which works from every cell of the table and its weight.

$SE = \dfrac{1}{(1 - p_{e(w)})^2}\sqrt{\dfrac{\sum_i \sum_j p_{ij}\big[w_{ij}(1 - p_{e(w)}) - (\bar{w}_{i\cdot} + \bar{w}_{\cdot j})(1 - p_{o(w)})\big]^2 - \big(p_{o(w)}\,p_{e(w)} - 2p_{e(w)} + p_{o(w)}\big)^2}{n}}$ = $\dfrac{1}{(1 - .625)^2}\sqrt{\dfrac{.0414 - .0361}{120}}$

= .047

$SE\text{, quadratic}$ = .041
$SE\text{, unweighted}$ = .057

For each cell, the bracket compares the cell's credit with the average credit of its row and column. Weighted by the cells' shares and added up, that's .0414; the correction term is .0361. Inside the square root, .0414 − .0361 = .0053, and √(.0053 ÷ 120) = .0066, and dividing by $(1 - .625)^2$ = .1406 gives $SE$ = .047.
9. Give a Range of Plausible Values for the True Weighted Kappa

Instead of a single best guess, give the range of weighted kappas for these two clinicians, rating patients like these, that fit the data.

$\kappa_w \pm z^{*} \times SE$ = $.696 \pm 1.96 \times .047$

= [.60, .79]

$CI\text{, quadratic}$ = [.71, .88]
$CI\text{, unweighted}$ = [.48, .70]

$z^{*}$ = 1.96 holds the middle 95% of a normal distribution. 1.96 × .047 = .09, so the range is .70 ± .09, from .60 to .79. With quadratic weights it's .79 ± .08, from .71 to .88.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

Recode the ratings into numbers in scale order (none = 1, mild = 2, moderate = 3, severe = 4). SPSS orders text codes alphabetically, which would put "mild" before "none" and scramble the weights.
Analyze → Scale → Weighted Kappa.
Check Specify raters for rows and columns, and move severity_a into Row rater and severity_b into Column rater.
Click Criteria and choose Linear (the default) or Quadratic weighting. Run it once with each if you plan to report both.
Click Print, ask for the crosstabulation, and click OK.

Or paste this syntax:

RECODE clinician_a clinician_b ('none'=1) ('mild'=2) ('moderate'=3) ('severe'=4)
  INTO severity_a severity_b.
EXECUTE.

WEIGHTED KAPPA severity_a BY severity_b
  /CRITERIA WEIGHTING=LINEAR ASYMPTOTICCILEVEL=95
  /PRINT TABULATION(ALL).

Reading the Output

Read the crosstabulation first: the diagonal holds the exact agreements (22 + 29 + 19 + 14 = 84 of 120), and nearly all the other patients sit one cell off it. Then read the weighted kappa, its asymptotic standard error and the 95% asymptotic confidence interval. For this data, the linear run should show κ_w = .696, SE = .047, 95% CI [.604, .789], and the quadratic run κ_w = .794, SE = .041, 95% CI [.713, .875]. SPSS's documentation doesn't say which standard error it uses; if its values differ slightly from these, that's why.

For the unweighted κ, run Crosstabs with Kappa checked (as on the Cohen's kappa page): κ = .591.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

library(psych)

ratings <- read.csv("data.csv")

# Put the ratings in scale order, scored 1 to 4
levels <- c("none", "mild", "moderate", "severe")
scores <- data.frame(
  a = match(ratings$clinician_a, levels),
  b = match(ratings$clinician_b, levels)
)

cohen.kappa(scores, w.exp = 1)   # linear weights
cohen.kappa(scores)              # quadratic weights (the default, w.exp = 2)

ck <- cohen.kappa(scores, w.exp = 1)
sqrt(ck$var.weighted)            # Fleiss, Cohen & Everitt's SE

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries
                 lower estimate upper
unweighted kappa  0.48     0.59  0.70
weighted kappa    0.60     0.70  0.79

 Number of subjects = 120

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries
                 lower estimate upper
unweighted kappa  0.48     0.59  0.70
weighted kappa    0.71     0.79  0.88

 Number of subjects = 120

[1] 0.0473297

The first block uses linear weights, the second quadratic; the unweighted row is the same in both. psych's default is quadratic, so say w.exp = 1 for linear. Its intervals use Fleiss, Cohen and Everitt's (1969) standard error, the same as this page. If you pass your own matrix with w =, give agreement weights (1 on the diagonal); psych treats w as credit, and a matrix of disagreement weights gives the right κ_w but a slightly wrong interval.

Output from a real run: R 4.6.1 (psych 2.6.5), run 2026-10-03; Call lines trimmed.

Python

import pandas as pd
from sklearn.metrics import cohen_kappa_score
from statsmodels.stats.inter_rater import cohens_kappa

ratings = pd.read_csv("data.csv")
levels = ["none", "mild", "moderate", "severe"]
cohen_kappa_score(ratings["clinician_a"], ratings["clinician_b"], labels=levels, weights="linear")      # 0.6963
cohen_kappa_score(ratings["clinician_a"], ratings["clinician_b"], labels=levels, weights="quadratic")   # 0.7941

a = pd.Categorical(ratings["clinician_a"], categories=levels)
b = pd.Categorical(ratings["clinician_b"], categories=levels)
table = pd.crosstab(a, b, dropna=False)
print(cohens_kappa(table, wt="linear"))

                  Weighted Kappa Coefficient
              --------------------------------
              Kappa                     0.6963
              ASE                       0.0473
              95% Lower Conf Limit      0.6035
              95% Upper Conf Limit      0.7891

                 Test of H0: Weighted Kappa = 0

              ASE under H0              0.0634
              Z                         10.9897
              One-sided Pr >  Z         0.0000
              Two-sided Pr > |Z|        0.0000

scikit-learn's cohen_kappa_score gives κ_w only; pass labels in scale order, or it sorts the text codes alphabetically (mild, moderate, none, severe) and returns a wrong .585. statsmodels' cohens_kappa takes the square table (in scale order, which the categorical columns guarantee) and gives the standard error (ASE), the 95% interval and a test against zero. wt="quadratic" gives .7941, ASE .0414, [.7129, .8753].

Output from a real run: Python 3.9.6 (scikit-learn 1.6.1, statsmodels 0.14.6, pandas 2.3.3), run 2026-10-03.

Impact

Effect Size

Weighted kappa is this page's measure of size. Like Cohen's kappa, it rescales agreement so that 0 means chance level and 1 means perfect, but it counts a near miss as partly right, so it measures how close the raters' ratings are rather than only how often they match exactly.

\kappa_w = \dfrac{p_{o(w)} - p_{e(w)}}{1 - p_{e(w)}} = \dfrac{.886 - .625}{1 - .625} = .70

The clinicians matched exactly on 70% of patients, and most of the rest were one step off. Counting those near misses as two-thirds agreement, they reached 70% of the agreement beyond chance that was possible: substantial, by convention. Plain kappa, which counts them as complete misses, says .59.

The weights are part of the answer. Quadratic weights forgive near misses more and usually give the highest values (van Oest & Moss, 2026); here they give .79. Weighting can also pull the value down: if the disagreements are big ones, weighted kappa falls below the unweighted value.

Suppose the same two clinicians still agreed exactly on 84 of 120 patients, but every one of the 36 disagreements was two or three steps apart, such as none against moderate: 30 two steps apart and 6 three steps apart, with each clinician rating 30, 39, 24 and 27 patients none, mild, moderate and severe. Unweighted κ would be about the same, .60, but linear κ_w would drop to .46 and quadratic κ_w to .39. Same exact agreement, very different closeness.

That's why weighted kappa should travel with the table (or at least the exact agreement and how far apart the disagreements were) and with the name of its weights. Benchmarks are conventions. Whether two clinicians who match exactly on seven patients in ten, and are one step off on most of the rest, are consistent enough depends on what the ratings will be used for: sorting a waiting list, or deciding who gets medication.

Benchmarks Weighted Kappa (

\kappa_w

) at a Glance

Poor

κw < .00

Slight

.00–.20

Fair

.21–.40

Moderate

.41–.60

Substantial

.61–.80

Almost Perfect

.81–1.00

Full benchmarks and caveats ↑

Uncertainty

This page gives weighted kappa without a p-value. Software can test whether κ_w is above 0, using a standard error worked out as if the raters were rating independently (Fleiss, Cohen & Everitt, 1969), and some programs, such as Python's statsmodels, print it. But nobody trains two clinicians expecting chance-level agreement, and with a reasonable number of cases almost any real agreement beats zero, including agreement far too poor to use.

Our interval for the linear κ_w, .60 to .79, is nowhere near 0, so a test against zero would come out overwhelmingly significant (statsmodels gives $z$ = 10.99). That tells us nothing we didn't know. The interval's lower end, at the top of the moderate range, is the more useful fact.

The question that matters is how high κ_w is and how precisely it's estimated, which is what the interval answers. The interval is a large-sample approximation (Fleiss, Cohen & Everitt, 1969): with few cases, a sparse table or κ_w near 1 it can come out too narrow or run past 1. Other intervals, such as a bootstrap that resamples patients, may do better, though with very little data none is fully trustworthy. A p-value, if you report one, is not the probability that the raters agree only by chance, and it says nothing about whether their agreement is good enough.

Precision and Sample Size

Power is the chance that a study detects an effect that's really there. For weighted kappa, a test of agreement better than chance is usually a low bar, though power can still matter with few cases, a rare code, or a test against a minimum other than zero. The more useful planning question is usually precision: how many cases you need for the interval around κ_w to be narrow enough to tell, say, moderate from substantial agreement.

The standard error shrinks with the square root of the number of cases. It also depends on how often each rating is used and on how the disagreements fall, so a pilot study or an earlier study with a similar mix of patients gives a workable guess.

In our table, Fleiss, Cohen and Everitt's (1969) standard error for the linear κ_w works out to √(0.2688 ÷ n), .047 for our 120 patients, so the interval runs about ±.09. To get it to ±.10 would take 0.2688 × (1.96 ÷ .10)² = 103.3, rounded up to 104 patients; to get it to ±.05, 413.1, so 414. Quadratic weights, with a smaller standard error here, would need 80 and 317.

Plan a weighted kappa study around the width of the interval you need, using the rating mix you expect. The planner below works out the interval for unweighted kappa, as Cantor (1996) plans it, with both raters using the ratings at the rates you enter. It doesn't know about near misses or your weights, so it isn't a weighted kappa planner. At our values it gives ±.10 for 120 patients, where our linear κ_w came out ±.09, but that isn't guaranteed: when many disagreements are far apart, the weighted interval can be the wider one. Use it only as a rough reference. To plan a weighted kappa study, do what the example above does: take a pilot or earlier table, work out κ_w's standard error with the weights you'll use, and scale it to the sample size you need. Try it:

Other Info

Follow-Up and Post Hoc Tests

None. This page estimates rather than tests, so there are no follow-up comparisons to run.

Related Methods

Alternative MeasureCohen's Kappa: Cohen's kappa is the special case of weighted kappa in which every disagreement costs the full amount, however close the ratings are (Fleiss, Cohen & Everitt, 1969). For ordered ratings it's the stricter measure: here .59 against .70 (linear) and .79 (quadratic).
Alternative MeasureIntraclass Correlation Coefficient (ICC): With quadratic weights and the ratings scored 1, 2, 3 and so on (any equally spaced scores give the same answer), weighted kappa converges in large samples to the two-way intraclass correlation for a single rater, ICC(2,1), and to the consistency version, ICC(3,1), when the two raters' average ratings are about equal (van Oest & Moss, 2026; Fleiss & Cohen, 1973). Here the averages are almost the same, and both ICCs are .79, like quadratic κ_w. The ICC suits numeric scores; κ_w keeps the ratings as categories and lets you choose the weights.
Related DesignFleiss' Kappa: The same agreement-beyond-chance question with three or more raters, or different raters for different cases (Fleiss, 1971), for unordered codes: it counts every disagreement as a full miss. With three or more raters on an ordered scale, an intraclass correlation on the ratings scored 1 to k or Krippendorff's alpha with ordinal distances keeps the order.
Alternative MeasureKrippendorff's Alpha: Krippendorff's alpha with ordinal or interval distances gives partial credit for near misses too, for any number of raters and with missing ratings allowed. It works out chance from the raters' pooled ratings rather than from each rater's own rates.

References

Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323–327. https://doi.org/10.1037/h0028106
“weighted kappa is appropriate when the relative seriousness of the different possible disagreements can be specified” (p. 323)

“Formulas for the standard errors of these two statistics have been given in the literature, but they are in error.” (Abstract, p. 323)
van Oest, R., & Moss, J. (2026). Quadratically weighted agreement coefficients: Interpretations and connections. Psychometrika, 91(3), 1069–1091. https://doi.org/10.1017/psy.2026.10085
“Quadratic weights are most lenient to partial disagreements and usually yield the highest coefficient values” (p. 1072)

“Cohen’s quadratically weighted kappa converges to two-factor intraclass correlation coefficients with random and fixed raters (i.e., the so-called ICC(2,1) and ICC(3,1) coefficients)” (p. 1072)

“Thus, linear, quadratic, and radical weights penalize disagreements by taking the involved distance (expressed as a fraction of the maximum distance) to the power 1, 2, or 0.5.” (p. 1072)
Brenner, H., & Kliebsch, U. (1996). Dependence of weighted kappa coefficients on the number of categories. Epidemiology, 7(2), 199–202. https://doi.org/10.1097/00001648-199603000-00016
“Number of categories and type of weighting scheme therefore require careful consideration in the interpretation of weighted kappa coefficients.” (Abstract)
Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3), 613–619. https://doi.org/10.1177/001316447303300309
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
“Although these divisions are clearly arbitrary, they do provide useful “benchmarks” for the discussion of the specific example in Table 1.” (p. 165)
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Cantor, A. B. (1996). Sample-size calculations for Cohen's kappa. Psychological Methods, 1(2), 150–153. https://doi.org/10.1037/1082-989X.1.2.150
“When designing a study to produce an estimate κ̂ of kappa, the sample size should be chosen so that the standard error of κ̂ will not exceed a preassigned value.” (p. 151)