Intraclass Correlation Coefficient (ICC)

The Basics

When to Use It

Use it when two or more raters (or occasions, or instruments) give continuous scores to the same set of targets, and you want to know how much of the variation in those scores reflects real differences between the targets. It's the standard index of inter-rater reliability for ratings, and it comes in several forms, so the first job is choosing the one that matches your design (see Effect Size below). For example:

Do trained coders agree on how empathic a therapist was in a session?
Do two clinicians give the same patients the same score on the Hamilton Depression Rating Scale?
Do three judges agree on the creativity ratings they give children's drawings?

Vocabulary You Need

Definition Intraclass Correlation Coefficient (

\text{ICC}

)

How much of the variation in a set of ratings or measurements comes from real differences between the targets being measured (people, sessions, essays), rather than from differences between raters or occasions and random error. It is the variance between targets divided by that variance plus the error variance, so 1 means the scores capture nothing but real differences and 0 means they capture none. It comes in several forms that answer different questions: whether raters' systematic differences count as error (absolute agreement) or are set aside (consistency), whether raters are a sample from a larger pool or the only ones of interest, and whether the reliability is for one rater's score or for the average of several. Always say which form you report. In a sample it can dip below 0 when the true differences between targets are tiny. In multilevel models the same ratio describes how much of an outcome lies between clusters; see Intraclass Correlation (Clustering). Glossary →

Definition Absolute Agreement

A definition of reliability in which raters agree only when they give the same target the same score. A rater who scores everyone a point higher than the others still ranks the targets the same way, but under absolute agreement that systematic difference counts as error, so it lowers the ICC. It's the definition to use when scores from different raters (or occasions) will be treated as interchangeable, such as when each target in a later study is scored by just one of them. An absolute-agreement ICC is almost never noticeably larger than the matching consistency ICC (by chance it can be slightly larger, when the raters' averages differ less than chance alone would produce); a clear gap between them is a sign that raters differ systematically. Glossary →

Definition Consistency

A definition of reliability that asks whether raters' scores rise and fall together across targets, setting aside any constant difference between raters. A rater who scores everyone a point lower than the others is perfectly consistent with them if the gaps are otherwise the same. It's the right definition when raters' overall harshness is irrelevant to how the scores will be used: when only the relative standing of targets matters, or when one rater will score every target, so their harshness shifts everyone equally. If different targets will be scored by different raters and the scores pooled, harshness matters, and absolute agreement is the definition to use. Like a correlation, it can look high while raters disagree about the actual scores. Unlike a correlation, it forgives only a constant offset: a rater who spreads scores more widely than the others still counts against it. Glossary →

Definition Single and Average Measures

Every ICC comes in two types. A single-measures ICC describes how reliable one rater's (or one occasion's) score is; an average-measures ICC describes how reliable the mean of k raters' scores is. Averaging cancels out some of each rater's error, so whenever the single-measures ICC is above zero, the average-measures ICC is higher, by the Spearman-Brown formula. Report the type that matches how scores will be used: if each target will be scored by one rater in practice, the single-measures ICC is the honest one, even though the reliability study used several raters. The two types are also called “single rater” and “mean of k raters,” and written ICC(2,1) and ICC(2,k). Glossary →

Assumptions

1. Independent Raters Each rater rates each case they're given without knowing anyone else's ratings.

What It Means: The raters reach their ratings separately: they don't discuss cases, see each other's ratings or share notes before rating. Agreement statistics such as kappa, Krippendorff's alpha and the intraclass correlation treat each rating as a separate judgment. Raters who confer agree more than the rating procedure would on its own, so the result overstates how reliable it is.
Applies To: Raters
How to Check: By design, not by a test. Did each rater work from the same material (a recording, a transcript, the same case file) without access to anyone else's ratings, and were disagreements resolved only after all had finished?
How Much It Matters: A lot. If coders discussed sessions or saw each other's scores, their agreement reflects the conversation, not the coding manual, and the ICC overstates how well a coder working alone would do.

If It's Violated

Design Change Have each coder rate every session alone, without seeing the others' scores, and hold calibration discussions only on practice sessions outside the reliability sample.
Proceed with Caveat If coders did confer, say so, and describe the ICC as agreement after discussion rather than independent reliability.

2. Independent Observations The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: Matters for the interval. Several sessions from the same therapist, or the same client, are more alike than sessions from different people, so the sample holds less information than 30 independent sessions and the confidence interval comes out too narrow.

If It's Violated

Design Change Sample one session per therapist (and client) for the reliability study.
Alternative Test Multilevel Regression (HLM). A multilevel model with sessions nested in therapists estimates the variance components with the clustering accounted for.

3. Complete, Balanced Ratings For the classical ANOVA ICC, the same raters rate every target, with no ratings missing.

What It Means: The classical two-way ANOVA ICCs, the formulas and software output on these pages, need a complete table: every target (a person, a session, a recording) rated by every rater, or measured on every occasion. A target missing any rating drops out of that calculation entirely. This is a requirement of the calculation, not of the ICC itself: a mixed-effects model with targets and raters as random effects estimates the same variance components from incomplete or partly crossed ratings, and when each target has its own randomly assigned raters, the one-way ICC fits.
Applies To: Observations
How to Check: Lay the ratings out with one row per target and one column per rater or occasion, and count the rows with a gap. Ask why they are missing: if the hardest targets are the ones skipped, dropping them makes reliability look better than it is.
How Much It Matters: Any session missing a coder's rating drops out entirely. That wastes data, and if the hardest sessions to rate are the ones skipped, the ICC looks better than it is.

If It's Violated

Alternative Test Multilevel Regression (HLM). Estimates the session, coder and error variances from every rating that exists, then forms the same ratio.
Proceed with Caveat Say how many sessions were dropped and why.

4. Roughly Normal True Scores and Errors Targets' true scores and the measurement errors are roughly bell-shaped, for the interval.

What It Means: The intraclass correlation itself is a ratio of variances and needs no distribution. Its F-based confidence interval and F test assume that the targets' true scores and the errors around them are roughly normally distributed.
Applies To: Residuals
How to Check: Look at a histogram of the targets' average scores and a histogram or Q–Q plot of the residuals (each rating minus its target's average and its rater's average, plus the overall average) for strong skew or a few extreme values. Don't rely on a normality test.
How Much It Matters: The ICC itself is just a ratio of variances and doesn't need normality, but its confidence interval and F test assume normal targets and errors. Mild skew in a 1-to-10 rating barely matters; a few wildly discrepant ratings can pull the ICC down on their own. Our residuals look close to normal.

If It's Violated

Robust Estimation Bootstrap the sessions (resample whole sessions with all their ratings) for a confidence interval that doesn't lean on normality.
Proceed with Caveat Look at the sessions where coders disagree most; a data-entry error or a misread manual item is worth fixing before anything else.

Limitations

This Is Not Your Test If…

Your raters sort targets into categories, not scores Cohen's Kappa →
You're checking a scale's items, not its raters Cronbach's Alpha →
You measured the same people twice with one instrument Test-Retest Reliability →
You want to know whether raters' averages differ One-Way Repeated-Measures ANOVA →
Each target was rated by different raters, with gaps Multilevel Regression (HLM) →

What It Doesn't Tell You

Whether the ratings are right. High agreement means the coders see the same thing, not that they're measuring empathy. Three coders trained on the same flawed manual can agree closely and all be wrong; that's a question of validity.
Which coder is off. The ICC pools everyone's disagreement into one number. Coder C's harshness shows up only because we also looked at each coder's mean and at the consistency ICC.
Who else. The ICC depends on how much the sessions differ. The same coders rating a narrower range of sessions (say, only experienced therapists) would get a lower ICC with no change in their skill.
That reliability is good enough. An ICC of .73 whose interval runs from .48 to .87 is consistent with poor reliability. It's a reason to collect more data or retrain, not evidence that one coder can be trusted alone.

Common Mistakes

Don'tReport a consistency ICC when scores from different raters will be used interchangeably.

InsteadUse absolute agreement, which counts a harsh or lenient rater against reliability. Here it's .73, not .80.
Don'tReport the average-measures ICC when each target will be scored by one rater.

InsteadReport the single-measures ICC. The average-measures one (.89 here) describes the mean of three coders, which only applies if you'll average three coders.
Don'tPick the ICC form that gives the biggest number.

InsteadChoose the model, type and definition from the design before running it, and say which you used.
Don'tTreat a significant F test against 0 as evidence of good reliability.

InsteadRead the ICC and its confidence interval. Almost any real rating scheme beats zero reliability.
Don'tRun a reliability study on a homogeneous sample, such as only highly empathic sessions.

InsteadSample targets across the full range you'll meet in practice. The ICC shrinks when targets barely differ, however good the raters are.
Don'tReport “the ICC” without saying which one.

InsteadName the model, the type (single or average) and the definition (agreement or consistency), such as a two-way random-effects, absolute-agreement, single-rater ICC(2,1).
Don'tLabel reliability from the point estimate alone.

InsteadJudge the whole 95% confidence interval against the benchmarks, as Koo and Li (2016) recommend.

Result Decoder

Interpretation Guidelines

Benchmarks ICC(2,1)

ICC < .50 Poor
Less than half the variation in scores reflects real differences between targets. Scores from different raters or occasions can differ a lot for the same target.
.50 ≤ ICC < .75 Moderate
Most of the variation is real, but a sizable share is raters, occasions or noise. Sometimes treated as enough for comparing groups, rarely for decisions about individuals.
.75 ≤ ICC ≤ .90 Good
About three quarters or more of the variation reflects real differences between targets.
ICC > .90 Excellent
Almost all the variation reflects real differences. Often treated as the minimum for decisions about individuals.

Koo and Li's (2016) labels, which they suggest for studies with at least 30 targets and 3 raters, judged against the 95% confidence interval rather than the estimate alone. They are conventions; how much reliability is enough depends on what the scores will be used for. Some papers use other labels with different cut-offs. Cicchetti (1994) gives a widely used scheme for kappa, weighted kappa and the ICC: below .40 poor, .40 to .59 fair, .60 to .74 good and .75 or above excellent. On that scale this page's interval, .48 to .87, would run from fair to excellent rather than poor to good, so check which scheme a paper used before comparing its labels with these.

More about effect size ↓

How It's Reported

Equation Form

Example

ICC(2,1) = .73, 95% CI [.48, .87]; ICC(C,1) = .80, 95% CI [.67, .89]

Table 1. All Six ICC Forms for the Empathy Ratings

Example

	ICC	95% CI	$F$	$df_1$	$df_2$	$p$
ICC(1,1): One-Way Random, Single Rater	.72	[.56, .84]	8.74	29	60	< .001
ICC(2,1) or ICC(A,1): Two-Way, Absolute Agreement, Single Rater	.73	[.48, .87]	13.12	29	58	< .001
ICC(3,1) or ICC(C,1): Two-Way, Consistency, Single Rater	.80	[.67, .89]	13.12	29	58	< .001
ICC(1,3): One-Way Random, Average of 3	.89	[.79, .94]	8.74	29	60	< .001
ICC(2,3) or ICC(A,3): Two-Way, Absolute Agreement, Average of 3	.89	[.73, .95]	13.12	29	58	< .001
ICC(3,3) or ICC(C,3): Two-Way, Consistency, Average of 3	.92	[.86, .96]	13.12	29	58	< .001

Note. Forms and confidence intervals follow Shrout and Fleiss (1979) and McGraw and Wong (1996). Each two-way row gives the same number whether the coders are treated as random or fixed (mixed); the model changes who the result generalizes to, not the calculation. The A and C names are McGraw and Wong's, for absolute agreement and consistency. F tests each ICC against 0. The average-measures rows describe the mean of all 3 coders' ratings. We report ICC(2,1).

Table 2. Two-Way ANOVA of the Empathy Ratings (Sessions × Coders)

Example

	$SS$	$df$	$MS$	$F$	$p$
Sessions	349.33	29	12.05	13.12	< .001
Coders	29.40	2	14.70	16.01	< .001
Error	53.27	58	0.92

Note. The mean squares every ICC is built from. The coders' F shows their averages differ, which is why absolute agreement comes out lower than consistency.

Full Text (APA Style)

Example

Three coders independently rated therapist empathy in 30 recorded sessions on a 1-to-10 scale. Coder C rated lower on average (M = 5.20, SD = 2.22) than coders A (M = 6.50, SD = 1.83) and B (M = 6.30, SD = 2.37). Inter-rater reliability was estimated with a two-way random-effects, absolute-agreement, single-rater intraclass correlation (pingouin 0.5.5; McGraw & Wong, 1996). Reliability was poor to good, ICC(2,1) = .73, 95% CI [.48, .87]; the estimate itself falls in the moderate range. Consistency was higher, ICC(C,1) = .80, 95% CI [.67, .89]: coders largely agreed on which sessions were more empathic, and much of their disagreement was coder C's harsher scoring. The average of all three coders' ratings was more reliable, ICC(2,3) = .89, 95% CI [.73, .95].

Name the form in words and symbols: model (one-way random, two-way random or two-way mixed), type (single or average of k raters) and definition (absolute agreement or consistency), plus the software (Koo & Li, 2016).
Always give the 95% confidence interval, and base the label (poor, moderate, good, excellent) on the interval, not the point estimate.
When raters differ systematically, report both the absolute-agreement and the consistency ICC, and each rater's mean, so readers can see the bias (Liljequist et al., 2019).
The F test against 0 is usually left out; if you give it, don't present it as evidence of good reliability.
A sample ICC can come out below 0 when the targets barely differ. Report it as it is rather than rounding it to 0; it means the data show no reliable differences between targets.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how certain.

ICC(2,1) = .73

About 73% of the variation in a single coder's rating reflects real differences between sessions; the rest is coder harshness and noise. The estimate falls in the moderate range of the benchmarks above, but the label belongs to the interval below. Benchmarks ↑

95% CI [.48, .87]

The true reliability of one coder's rating could plausibly be anywhere from .48 (poor) to .87 (good). By Koo and Li's rule of judging the interval, we can't yet call it more than poor to good; more sessions or more coders would narrow it. Which helps more depends on how much of the variation comes from sessions, coders and error; in our simulation (Precision and Sample Size), doubling the sessions and doubling the coders did about equally well, and with only three coders, adding sessions alone soon stops helping.

Calculations

The Running Example

Example

The empathy-coding study: 3 coders, each rating therapist empathy (1 to 10) in the same 30 recorded sessions, to see whether one coder's rating could stand in for any other's.

Null Hypothesis:: $\rho_{\text{ICC}} = 0$
Alternative (One-Sided):: $\rho_{\text{ICC}} > 0$

$\rho_{\text{ICC}}$ (“rho”) is the population ICC, the reliability of one coder's rating across all the sessions and trained coders this study stands for, which our .73 estimates. Software tests it against 0 (Table 1), but no one doubts that trained coders beat pure noise. The real question is how large it is, and the confidence interval answers that.

What Your Data Should Look Like

Every target rated by every rater. Most software wants it wide: one row per target, one column per rater. Ours is stored long, one row per target per rater, with columns for the target, the rater and the score; Python's pingouin uses it as is, while SPSS and R's psych need it restructured to wide. In the classical ANOVA calculation shown here, any target missing a rating drops out; a mixed-effects model (see Assumptions) can use incomplete ratings instead.

Example

session	coder	empathy
1	A	8
1	B	9
1	C	6
2	A	5
2	B	8
2	C	5
… 84 more rows

	n	M	SD
Coder A	30	6.50	1.83
Coder B	30	6.30	2.37
Coder C	30	5.20	2.22

30 sessions, each rated by all 3 coders (90 ratings).

Simulated data.

Download all 90 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the empathy-coding study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Find Each Coder's Average and the Overall Average

Before asking whether coders agree, look at where each one sets the bar. If one coder's average sits well below the others, that coder is harsher across the board.

$\bar{x}_{\text{grand}} = \dfrac{\sum x_i}{N}$ = $\dfrac{540}{90}$

= 6.00 points

$\bar{x}_{\text{A}}$ = 6.50
$\bar{x}_{\text{B}}$ = 6.30
$\bar{x}_{\text{C}}$ = 5.20

$\sum x_i$ is all 90 ratings added up, 540, and $N$ is the 90 ratings (30 sessions × 3 coders). 540 ÷ 90 = 6.00 points. Coders A and B average 6.50 and 6.30, but coder C only 5.20, more than a point lower.
2. Measure How Much the Sessions Really Differ

Reliability is about telling targets apart. If the sessions truly differ in empathy, their average ratings should spread out a lot. This is the signal every ICC is built on.

$SS_{\text{subjects}} = k \sum (\bar{x}_i - \bar{x}_{\text{grand}})^2$ = $3\left[(7.67 - 6)^2 + (6.00 - 6)^2 + \dots + (7.33 - 6)^2\right]$

= 349.33

$MS_{\text{subjects}}$ = 12.05

$\bar{x}_i$ is one session's average over the three coders, from 1.67 for the coldest session to 9.33 for the warmest, and $k$ is our 3 coders. The 30 squared distances from 6 add up to 116.44, and 3 × 116.44 = 349.33. Divided by $n - 1 = 29$ , that's a mean square of 12.05. A mean square is a sum of squares divided by its degrees of freedom, an average squared distance.
3. Measure How Much the Coders Differ in Their Overall Level

This is the harshness we saw in step 1, turned into a sum of squares. It says nothing about which sessions each coder thought were better, only how high or low they scored overall.

$SS_{\text{raters}} = n \sum (\bar{x}_j - \bar{x}_{\text{grand}})^2$ = $30\left[(6.50 - 6)^2 + (6.30 - 6)^2 + (5.20 - 6)^2\right]$

= 29.40

$MS_{\text{raters}}$ = 14.70

$n$ is our 30 sessions and $\bar{x}_j$ each coder's average. Coder C is 0.80 below 6, so the squares are 0.25, 0.09 and 0.64, and 30 × 0.98 = 29.40. Divided by $k - 1 = 2$ , the mean square is 14.70.
4. Find the Disagreement That's Left Over

Once the sessions' real differences and the coders' overall harshness are taken out, what remains is coders disagreeing about particular sessions, the plain noise in a single rating.

$SS_{\text{error}} = SS_{\text{total}} - SS_{\text{subjects}} - SS_{\text{raters}}$ = $432 - 349.33 - 29.40$

= 53.27

$SS_{\text{total}}$ = 432
$MS_{\text{error}}$ = 0.918

$SS_{\text{total}}$ is every rating's squared distance from 6, added up, 432. Taking out the sessions' 349.33 and the coders' 29.40 leaves 53.27. Divided by $(n - 1)(k - 1) = 29 \times 2 = 58$ , $MS_{\text{error}}$ = 0.918.
5. Turn the Mean Squares into Three Variances

The ICC is a share of variation, so first pull the three sources apart. Each mean square mixes one source with the leftover noise, so subtract the noise and rescale.

$\sigma^2_{\text{sessions}} = \dfrac{MS_{\text{subjects}} - MS_{\text{error}}}{k}, \quad \sigma^2_{\text{coders}} = \dfrac{MS_{\text{raters}} - MS_{\text{error}}}{n}, \quad \sigma^2_{\text{error}} = MS_{\text{error}}$ = $\dfrac{12.05 - 0.918}{3}, \quad \dfrac{14.70 - 0.918}{30}, \quad 0.918$

$\sigma^2_{\text{sessions}}$ = 3.71
$\sigma^2_{\text{coders}}$ = 0.46
$\sigma^2_{\text{error}}$ = 0.92

The sessions' mean square counts each session's real level 3 times (once per coder) on top of the noise, so (12.05 − 0.918) ÷ 3 = 3.71. The coders' counts each coder's harshness 30 times, so (14.70 − 0.918) ÷ 30 = 0.46. The noise is 0.92. In all, one coder's rating varies by 3.71 + 0.46 + 0.92 = 5.09, and 3.71 of that is real differences between sessions.
6. Ask What Share of One Coder's Rating Is Real, Counting Harshness as Error

This is the ICC for our question: could any one trained coder's rating stand in for any other's? The top is the sessions' spread with the noise taken out (3 × 3.71 from the last step). The bottom adds everything that would make two coders' scores for the same session differ: the leftover noise and each coder's overall harshness.

$\text{ICC}(2,1) = \dfrac{MS_{\text{subjects}} - MS_{\text{error}}}{MS_{\text{subjects}} + (k - 1)\,MS_{\text{error}} + \dfrac{k}{n}\left(MS_{\text{raters}} - MS_{\text{error}}\right)}$ = $\dfrac{12.05 - 0.918}{12.05 + 2(0.918) + \frac{3}{30}(14.70 - 0.918)} = \dfrac{11.13}{13.88 + 1.38}$

= .73

$MS_{\text{subjects}}$ is the sessions' 12.05, $MS_{\text{error}}$ the leftover 0.918 and $MS_{\text{raters}}$ the coders' 14.70, with $k = 3$ coders and $n = 30$ sessions. The top is 11.13; the bottom is 13.88 plus 1.38 for the coders' harshness, 15.26. 11.13 ÷ 15.26 = .73. It's the same share as 3.71 ÷ 5.09 from the last step; the mean-square formula does it in one line.
7. Set the Coders' Harshness Aside to See How Consistently They Rank the Sessions

If what matters is only which sessions are more empathic, not the exact score, a coder who is a point harsher on every session is no problem. Dropping the raters' term from the bottom gives the consistency ICC.

$\text{ICC}(3,1) = \dfrac{MS_{\text{subjects}} - MS_{\text{error}}}{MS_{\text{subjects}} + (k - 1)\,MS_{\text{error}}}$ = $\dfrac{12.05 - 0.918}{12.05 + 2(0.918)} = \dfrac{11.13}{13.88}$

$\text{ICC}(C,1)$ = .80

The same top, 11.13, over the same bottom without the coders' 1.38, 13.88. 11.13 ÷ 13.88 = .80, the same as 3.71 ÷ (3.71 + 0.92) = 3.71 ÷ 4.63. The gap from .73 is coder C's harshness. Software labels this ICC(3,1) or “fixed raters”; with coders who stand for a larger pool, like ours, it's McGraw and Wong's ICC(C,1), the same number. It sets aside only a constant offset. A coder who uses more or less of the scale than the others, or is harsh on only some kinds of session, still counts against both ICCs. Coder B's ratings spread wider than coder A's (SD 2.37 against 1.83); plot each session's ratings by coder to see differences like that.
8. See How Much More Reliable the Average of All Three Coders Would Be

Averaging several coders cancels out part of each one's noise, so the mean of three ratings is more reliable than any single rating. That only helps if you'll actually average three coders in the main study.

$\text{ICC}(2,k) = \dfrac{MS_{\text{subjects}} - MS_{\text{error}}}{MS_{\text{subjects}} + \dfrac{MS_{\text{raters}} - MS_{\text{error}}}{n}}, \quad \text{ICC}(3,k) = \dfrac{MS_{\text{subjects}} - MS_{\text{error}}}{MS_{\text{subjects}}}$ = $\dfrac{11.13}{12.05 + \frac{14.70 - 0.918}{30}} = \dfrac{11.13}{12.51}, \quad \dfrac{11.13}{12.05}$

$\text{ICC}(2,3)$ = .89
$\text{ICC}(3,3)$ = .92

For absolute agreement, the bottom is the sessions' 12.05 plus the coders' harshness spread over 30 sessions, 0.46, so 11.13 ÷ 12.51 = .89. For consistency, it's just 11.13 ÷ 12.05 = .92. For these balanced ANOVA estimates, each is exactly the single-rater ICC stepped up by the Spearman-Brown formula, k × ICC ÷ (1 + (k − 1) × ICC), so 3 × .73 ÷ (1 + 2 × .73) = .89. Turned around, the same formula says how many coders you'd need to average. To reach .90 from .73 takes .90 × (1 − .73) ÷ (.73 × .10) = 3.3, so 4.
9. Give a Range of Plausible Values for the True ICC

Thirty sessions and three coders pin the ICC down only roughly, and with just three coders we know little about how much coders differ in general. The interval shows how far the true reliability could plausibly be from .73.

$L = \dfrac{n\,(MS_{\text{subjects}} - F^{*} MS_{\text{error}})}{F^{*}\left[k\,MS_{\text{raters}} + (kn - k - n)\,MS_{\text{error}}\right] + n\,MS_{\text{subjects}}}, \quad U = \dfrac{n\,(F^{**} MS_{\text{subjects}} - MS_{\text{error}})}{k\,MS_{\text{raters}} + (kn - k - n)\,MS_{\text{error}} + n\,F^{**} MS_{\text{subjects}}}$ = $L = \dfrac{30\,(12.05 - 2.56 \times 0.918)}{2.56\,[3(14.70) + 57(0.918)] + 30(12.05)}, \quad U = \dfrac{30\,(2.29 \times 12.05 - 0.918)}{3(14.70) + 57(0.918) + 30(2.29)(12.05)}$

= [.48, .87]

$v$ = 16.2

$F^{*}$ = 2.56 cuts off the top 2.5% of an F distribution with 29 and $v$ = 16.2 degrees of freedom, and $F^{**}$ = 2.29 the same with the degrees of freedom swapped (16.2 and 29). $v$ blends the coders' and error mean squares; with only 3 coders it's small, which widens the interval. $kn - k - n$ = 90 − 3 − 30 = 57. The lower limit is 290.8 ÷ 608.5 = .48 and the upper 801.0 ÷ 924.9 = .87.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

SPSS wants one column per coder. Our file is long, so first use Data → Restructure → Restructure selected cases into variables, with session as the Identifier Variable and coder as the Index Variable. You get one row per session with columns A, B and C.
Analyze → Scale → Reliability Analysis. Move A, B and C into Items.
Statistics: check Intraclass correlation coefficient. Set Model to Two-Way Random, Type to Absolute Agreement, Confidence interval to 95% and Test value to 0.
Click Continue, then OK.

Or paste this syntax:

SORT CASES BY session coder.
CASESTOVARS
  /ID=session
  /INDEX=coder.

RELIABILITY
  /VARIABLES=A B C
  /SCALE('ALL VARIABLES') ALL
  /MODEL=ALPHA
  /ICC=MODEL(RANDOM) TYPE(ABSOLUTE) CIN=95 TESTVAL=0.

Reading the Output

Read the Single Measures row: the Intraclass Correlation, then its 95% Confidence Interval. The Average Measures row is the reliability of the mean of all three coders, which only applies if you'll average three coders. The F test against 0 is rarely worth reporting.

Intraclass Correlation Coefficient

	Intraclass Correlation	95% CI Lower Bound	Upper Bound	F Test with True Value 0: Value	df1	df2	Sig
Single Measures	.729	.478	.866	13.116	29	58	.000
Average Measures	.890	.733	.951	13.116	29	58	.000

SPSS's footnote reads “Two-way random effects model where both people effects and measures effects are random.” Its “people” are our sessions and its “measures” our coders. For the consistency ICC, rerun with TYPE(CONSISTENCY): .802 [.672, .892] single and .924 [.860, .961] average. The Reliability Statistics table above it shows Cronbach's alpha = .924, the same number as the average-measures consistency ICC.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

coding <- read.csv("data.csv")

# One row per session, one column per coder
wide <- reshape(coding, idvar = "session", timevar = "coder", direction = "wide")

library(psych)
ICC(wide[, c("empathy.A", "empathy.B", "empathy.C")])

Call: ICC(x = wide[, c("empathy.A", "empathy.B", "empathy.C")])

Intraclass correlation coefficients
                         type  ICC    F df1 df2       p lower bound upper bound
Single_raters_absolute   ICC1 0.72  8.7  29  60 1.2e-12        0.56        0.84
Single_random_raters     ICC2 0.73 13.1  29  58 2.4e-16        0.48        0.87
Single_fixed_raters      ICC3 0.80 13.1  29  58 2.4e-16        0.67        0.89
Average_raters_absolute ICC1k 0.89  8.7  29  60 1.2e-12        0.79        0.94
Average_random_raters   ICC2k 0.89 13.1  29  58 2.4e-16        0.73        0.95
Average_fixed_raters    ICC3k 0.92 13.1  29  58 2.4e-16        0.86        0.96

 Number of subjects = 30     Number of Judges =  3
See the help file for a discussion of the other 4 McGraw and Wong estimates,

psych prints all six forms at once; pick the row that matches your design before you look at the numbers. ICC2 is Shrout and Fleiss's ICC(2,1), two-way random with absolute agreement; ICC3 is ICC(3,1), consistency, which is also McGraw and Wong's ICC(C,1) for random coders like ours: random and mixed models give the same number. The irr package's icc(..., model = "twoway", type = "agreement", unit = "single") gives the same ICC(2,1) one form at a time, using McGraw and Wong's names.

Output from a real run: R 4.6.1 (psych 2.6.5), run 2026-10-03.

Python

import pandas as pd
import pingouin as pg

coding = pd.read_csv("data.csv")
icc = pg.intraclass_corr(data=coding, targets="session", raters="coder", ratings="empathy")
print(icc.round(3).to_string())

    Type              Description    ICC       F  df1  df2  pval         CI95%
0   ICC1   Single raters absolute  0.721   8.743   29   60   0.0  [0.56, 0.84]
1   ICC2     Single random raters  0.729  13.116   29   58   0.0  [0.48, 0.87]
2   ICC3      Single fixed raters  0.802  13.116   29   58   0.0  [0.67, 0.89]
3  ICC1k  Average raters absolute  0.886   8.743   29   60   0.0  [0.79, 0.94]
4  ICC2k    Average random raters  0.890  13.116   29   58   0.0  [0.73, 0.95]
5  ICC3k     Average fixed raters  0.924  13.116   29   58   0.0  [0.86, 0.96]

Pingouin takes the long file as it is. Read the ICC2 row: .729, 95% CI [.48, .87]. The pval column shows 0.0 only because it's rounded; the exact values are far below .001. Pingouin rounds the interval to two decimals.

Output from a real run: Python 3.9.6 (pingouin 0.5.5, pandas 2.3.3), run 2026-10-03.

Impact

Effect Size

The ICC is the answer itself, so it plays the part an effect size plays elsewhere: the share of the variation in scores that reflects real differences between targets, usually between 0 and 1. It has no units, so it can be compared across scales, but it still depends on the design and on how much the targets differ: the same raters get a lower ICC on a narrower sample. And “the ICC” is ten different coefficients (Koo & Li, 2016). Three questions, answered from the design before looking at any numbers, pick the one you need:

Did every target get the same raters? If each target was rated by its own randomly assigned raters, with no rater tied to particular targets, the one-way model, ICC(1), may fit. If raters overlap only partly, or some ratings are missing, a mixed-effects model with targets and raters as crossed random effects estimates the same variance components from every rating that exists (see the multilevel model in Assumptions). If every target got the same raters, go on.
Are the raters a sample from a larger pool you want to generalize to? If yes, the two-way random-effects model, ICC(2). If these raters are the only ones that matter, the two-way mixed model, ICC(3) (Shrout & Fleiss, 1979).
Does a rater's overall harshness matter? If scores from different raters will be treated as interchangeable, use absolute agreement. If only the ranking of targets matters, consistency. Then choose single or average measures by whether you'll use one rater's score or the mean of several.

The numbers in ICC(2,1) are Shrout and Fleiss's model (1 one-way, 2 two-way random, 3 two-way mixed) and how many ratings are averaged (1, or k for the mean of k raters). Shrout and Fleiss's ICC(2,1) is the two-way random-effects model with absolute agreement, and their ICC(3,1) is the two-way mixed model with consistency. McGraw and Wong (1996) name the definition separately from the model: ICC(A,1) for absolute agreement and ICC(C,1) for consistency, each of which can come from a random or a mixed model. With complete, balanced data, the random and mixed models give the same estimate for the same definition, so the ten forms come down to six calculations. The labels still aren't interchangeable, because the model changes who you can generalize the result to: other raters like these, or only these raters (Koo & Li, 2016). Software built on Shrout and Fleiss's names labels the consistency rows “ICC3” or “fixed raters” whichever model you meant, so report the model, the definition and single or average, not just the label.

Our coders are three of many who could be trained on the manual, the main study will have each session coded by just one of them, and a 6 from coder C has to mean the same as a 6 from coder A. That's two-way random, absolute agreement, single rater: ICC(2,1).

\text{ICC}(2,1) = \dfrac{MS_{\text{subjects}} - MS_{\text{error}}}{MS_{\text{subjects}} + (k - 1)MS_{\text{error}} + \frac{k}{n}(MS_{\text{raters}} - MS_{\text{error}})} = \dfrac{11.13}{15.26} = .73

About 73% of the variation in one coder's empathy rating is real differences between sessions. The consistency ICC, ICC(C,1), is .80, because coder C's ratings run about a point low: the coders mostly agree on which sessions were warmer, but not on the number.

When raters differ systematically, report both, since they answer different questions (Liljequist et al., 2019). Koo and Li's (2016) labels are conventions, and they ask you to apply them to the whole confidence interval: ours, [.48, .87], spans poor to good. Whether that's enough depends on the use. For comparing average empathy between two training programs, it may do; for deciding whether an individual therapist passes a competency check, it doesn't.

Benchmarks ICC(2,1) at a Glance

Poor

ICC < .50

Moderate

.50 ≤ ICC < .75

Good

.75 ≤ ICC ≤ .90

Excellent

ICC > .90

Full benchmarks and caveats ↑

Uncertainty

Software also tests each ICC against 0 with an F test: the sessions' mean square divided by the error mean square. It answers a narrow question: if the population ICC were 0 (no real differences between targets at all), how often would the targets' averages spread out at least this much? For a reliability study, that's rarely the question. Trained raters almost always beat pure noise, so the test is nearly always significant, and it says nothing about whether reliability is good enough. That's why this page reports no p in the main result.

For our coders, F(29, 58) = 13.12, p < .001. That only rules out the idea that the ratings are pure noise, which no one believed. The confidence interval, .48 to .87, is the useful part: it says reliability is somewhere between poor and good.

One F test is worth a look: the coders' F in the two-way ANOVA (Table 2), F(2, 58) = 16.01, p < .001, which says the coders' averages differ. That bias is the gap between absolute agreement and consistency. Even there, look at the size of the gap (coder C is 1.1 to 1.3 points lower) rather than just at p; with many targets, trivial biases come out significant (Liljequist et al., 2019).

p is not the probability that the null hypothesis is true, and a tiny p is not a measure of how reliable the ratings are.

Precision and Sample Size

Power is the chance that a study detects an effect that's really there. For reliability, detecting that the ICC isn't zero is almost never in doubt, so plan instead for precision: enough targets and raters that the confidence interval is narrow enough to say which benchmark range the ICC falls in. Bonett (2002) gives a method for finding the number of targets needed for an exact interval of a chosen width. Exact intervals exist for the consistency ICC, whose interval comes straight from the F test, and that's what the planner below calculates. For the absolute-agreement ICC, the number of raters matters more: with only a few raters, the study learns little about how much raters differ in general, and the interval stays wide however many targets you add.

We simulated studies like ours with a true ICC(2,1) of .73. With 3 coders and 30 sessions, a typical 95% interval for the absolute-agreement ICC is .33 wide. Doubling to 60 sessions narrows it only to .27, and 100 sessions to .25; keeping 30 sessions but using 6 coders also gets it to .27, and 6 coders with 100 sessions to .19. The consistency ICC's interval, which the planner below calculates, is narrower: at a consistency ICC of .73, the planner's starting value, it is .28 wide with 3 coders and 30 sessions, .20 with 60 and .15 with 100. (In the simulated studies the true consistency ICC is higher than .73, because coder C's harshness no longer counts as error, so its interval there is narrower still.)

Koo and Li (2016) suggest at least 30 heterogeneous targets and at least 3 raters. Sample targets across the full range you'll meet in practice, since a narrow range lowers the ICC however good the raters are.

Plan the number of targets and raters before collecting data, for the interval width you need. The planner calculates precision for a single-rater consistency ICC, not the absolute-agreement ICC this page features, so its answer is a reference, not a guaranteed minimum. For an absolute-agreement study, plan from expected target, rater and error variances, preferably by simulation as above, and add raters rather than only targets if raters are likely to differ on average. Try it:

Other Info

Follow-Up and Post Hoc Tests

None. This page estimates rather than tests, so there are no follow-up comparisons to run.

Related Methods

Related DesignOne-Way Repeated-Measures ANOVA: Both run the same two-way ANOVA, with the targets as the “subjects” and the raters as the “conditions.” The repeated-measures ANOVA asks whether the coders' averages differ (here, F(2, 58) = 16.01, p < .001, because coder C is harsher). The ICC uses the same mean squares to ask how much of the variation is real differences between sessions.
General CaseMultilevel Regression (HLM): A random-intercept model with sessions (and, for absolute agreement, coders) as random effects estimates the same variance components; the ICC is the session variance's share of the total. With complete data the two agree, and the multilevel model can also handle missing ratings and clustering.
Related DesignCronbach's Alpha: Treat the three coders as three items and Cronbach's α equals the average-measures consistency ICC, ICC(3,k): .92 for our coders either way.
Related DesignCohen's Kappa: The same question for categorical codes from two raters: how much they agree beyond what chance would produce. Kappa plays the part for categories that the ICC plays for numeric ratings.
Related DesignFleiss' Kappa: The same question for categorical codes from three or more raters: how much they agree beyond what chance would produce. Like the one-way ICC(1), it allows different raters for different targets.
Alternative MeasureKrippendorff's Alpha: For numeric ratings with missing values or different raters per target, Krippendorff's alpha with the interval metric is an agreement coefficient that uses every rating; with two raters and no missing ratings it comes out very close to a one-way, single-rater intraclass correlation.
Related DesignMultilevel SEM: For people nested in groups, the clustering ICC (ICC(1), the share of a variable's variance between groups) is multilevel SEM's first step; that page then splits each variable into within-group and between-group parts and fits paths at each level.
Special CaseTest-Retest Reliability: Test-retest reliability is this ICC with two occasions in place of raters, usually in its two-way, absolute-agreement, single-measures form, so a practice effect between sessions counts against reliability.
Alternative MeasureWeighted Kappa: For two raters' ordered categories scored as equally spaced numbers (1, 2, 3 and so on), weighted kappa with quadratic weights is closely related to the single-rater ICC: in large samples it converges to the two-way, absolute-agreement ICC(2,1), and to the consistency ICC(3,1) as well when the raters' average ratings are about equal (van Oest & Moss, 2026). In a given sample the two can be identical or just close, so they aren't interchangeable. Weighted kappa keeps the ratings as categories and lets you choose the weights; the ICC suits numeric scores.

References

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
“Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results.” (Abstract)

“guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges” (Abstract)
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284
“when the reliability coefficient is below .40, the level of clinical significance is poor; when it is between .40 and .59, the level of clinical significance is fair; when it is between .60 and .74, the level of clinical significance is good; and when it is between .75 and 1.00, the level of clinical significance is excellent.” (p. 286)
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
“We suggest that the best practice of reporting ICC should include the following items: software information, “Model,” “Type,” and “Definition” selections.” (How to Report ICC)

“We should use the 2-way mixed-effects model if the selected raters are the only raters of interest.” ((A) “Model” Selection)

“Under such conditions, we suggest that ICC values less than 0.5 are indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values greater than 0.90 indicate excellent reliability.” (ICC Interpretation)

“the 95% confident interval of the ICC estimate (not the ICC estimate itself) should be used as the basis to evaluate the level of reliability” (Discussion for Readers)

“Absolute agreement concerns if different raters assign the same score to the same subject.” ((C) “Definition” Selection)

“Conversely, consistency definition concerns if raters’ scores to the same group of subjects are correlated in an additive manner.” ((C) “Definition” Selection)

“Generally speaking, ICC of the “mean of k raters” type is larger than the corresponding “single rater” type.” (ICC Characteristics)
Liljequist, D., Elfving, B., & Skavberg Roaldsen, K. (2019). Intraclass correlation – A discussion and demonstration of basic features. PLOS ONE, 14(7), e0219854. https://doi.org/10.1371/journal.pone.0219854
“In presence of bias, both absolute agreement ICC and consistency ICC should be reported, since they give different and complementary information about the reliability of the method.” (Abstract)

“However, whether an ICC value is good enough should depend on the intended use of the method.” (1. Introduction)
Bonett, D. G. (2002). Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine, 21(9), 1331–1335. https://doi.org/10.1002/sim.1108
“A method is developed to calculate the approximate number of subjects required to obtain an exact confidence interval of desired width for certain types of intraclass correlations in one-way and two-way ANOVA models.” (Abstract)
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.