Stats for
Menu

Agreement and Reliability

Test-Retest Reliability

Test-retest reliability checks whether a measure gives people the same scores twice.

Test-retest reliability takes the same measure given twice to the same people and estimates how closely their two scores agree, which shows how much of the variation between people's scores reflects stable differences rather than occasion effects and measurement error.

Example

Does our working-memory task give students the same score when they take it again? Sixty undergraduates took a computerized working-memory span task twice, two weeks apart; each score is the number of letters recalled in the correct position, out of 75.

Same People, Two Occasions + Continuous Scores → Test-Retest Reliability
Estimates
Population ICC (absolute agreement)
Observations
Related
Parametric
Yes
Headline Statistic
\text{ICC}(2,1)
Key Estimate
ICC(2,1)

The Basics

When to Use It

Use it when the same people take the same continuous measure twice, under the same conditions, and you want to know how far a single score can be trusted: whether people get the same score again when nothing real has changed. For example:

  • Does our working-memory task give students the same score when they take it again?
  • Do adolescents' depression symptom scores stay the same when they fill in the questionnaire again a week later?
  • Is a person's average reaction time on a go/no-go task the same across two lab visits a month apart?

Vocabulary You Need

Definition Reliability

The extent to which a measure's scores can be replicated: measured again, by another rater or with other items, would people get the same scores? Formally, it's the share of the differences between people's scores that reflects real differences in what's measured rather than measurement error, so it runs from 0 (all noise) to 1 (no noise). It belongs to scores from a particular group of people under particular conditions, not to the test itself: the same test can be more reliable in a group whose scores vary widely. There are several kinds, each asking about a different source of noise: test-retest (occasions), inter-rater (raters) and internal consistency (items). A measure has to be reliable to be valid, but reliable scores can still measure the wrong thing. Glossary →

Definition Practice Effect

A shift in scores on a repeated measurement that comes from familiarity with the task rather than from any change in what's measured: people remember items or learn strategies. Broader retest effects, such as being calmer in a familiar setting, push the same way. On ability and memory tasks it usually raises retest scores. If everyone gains about the same amount, a correlation between sessions can't see it, but it still means a person's two scores disagree. Longer gaps between sessions, alternate forms or a practice session before the first real one reduce it. Glossary →

Definition Intraclass Correlation Coefficient (\text{ICC})

How much of the variation in a set of ratings or measurements comes from real differences between the targets being measured (people, sessions, essays), rather than from differences between raters or occasions and random error. It is the variance between targets divided by that variance plus the error variance, so 1 means the scores capture nothing but real differences and 0 means they capture none. It comes in several forms that answer different questions: whether raters' systematic differences count as error (absolute agreement) or are set aside (consistency), whether raters are a sample from a larger pool or the only ones of interest, and whether the reliability is for one rater's score or for the average of several. Always say which form you report. In a sample it can dip below 0 when the true differences between targets are tiny. In multilevel models the same ratio describes how much of an outcome lies between clusters; see Intraclass Correlation (Clustering). Glossary →

Definition Standard Error of Measurement (SEM)

The standard deviation of the measurement errors: roughly how far one observed score typically lands from the person's true score. In classical test theory it's one number for everyone, reliability turned back into the measure's units, SEM = SD × √(1 − reliability), where SD is the spread of all the scores, so it shrinks as reliability rises. An observed score ± 1.96 × SEM is the conventional rough 95% band for the person's true score; it assumes error is the same size at every score level, and it isn't a prediction of their next score. Item response theory gives a conditional version instead, 1 ÷ √(test information), which changes with the trait level: small where the items measure precisely, large at the extremes. Despite the similar abbreviation, it isn't the standard error of the mean: that describes how uncertain an average is; the SEM describes how noisy one person's score is. Glossary →

Assumptions and Inference

1. Stable Trait (for Reading Retest Agreement) What you're measuring shouldn't really change between the two sessions.
What It Means
Test-retest reliability treats any difference between a person's two scores as measurement error. That's only fair if the construct itself stayed put: a stable ability or trait, measured over a gap short enough that nothing real changed, but long enough that people don't simply remember their answers. If people truly changed, and by different amounts, the coefficient mixes real change with noise and makes the measure look less reliable than it is.
Applies To
Design
How to Check
By reasoning about the construct and the gap, not by a test. Is the construct expected to be stable over this interval (a trait rather than a mood)? Did anything happen between sessions, such as treatment, training or a stressful event? Could people remember specific items or answers? Report the interval, and look at the paired plot for a few people whose change is far from everyone else's.
How Much It Matters
A lot. Real change between sessions is counted as error, so reliability comes out too low. Working-memory capacity is fairly stable over two weeks, but any student who was ill or sleep-deprived at one session adds real change to the noise.

What to Do

  • Design Change Retest over a gap short enough that the construct can't change much, but long enough that people don't remember specific answers.
  • Design Change If people really do change, record what happened between sessions (illness, treatment, training) and analyze those people separately.
  • Proceed with Caveat State the retest interval, and describe the result as the reliability over that interval.
2. No Extreme Outliers No single case is far enough from the rest to drive the result.
What It Means
A few points far from the rest can pull a statistic around on their own, so it describes those few points more than everyone else. In a test, that can create an effect that isn't there or hide one that is.
Applies To
Observations
How to Check
Plot the data and look for points far from the rest. With two or more variables, flag cases with large squared Mahalanobis distances across them: points can be ordinary on each variable but unusual in combination. Check each against the raw records, and compare the result with and without it.
How Much It Matters
A lot. One student with a wildly different score at one session (a distraction, a misunderstood instruction) adds a big chunk of error variance and can pull the ICC down by itself; one extreme but consistent student can prop it up. Here no score is more than 2.8 standard deviations from its session's mean, and the largest change, a gain of 19 letters, is 2.4 standard deviations from the average change.

What to Do

  • Correction Check extreme scores and extreme changes against the session notes, and fix or remove data-entry and administration errors.
  • Proceed with Caveat If the student is real, report the ICC with and without them and say which you trust.
3. Independent Observations (for the Confidence Interval) The people (or other units) in your study don't influence each other.
What It Means
Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To
Observations
How to Check
By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters
Moderate. Students who are tested together or compare notes between sessions are not separate pieces of evidence, which makes the interval too narrow. Their own two scores are meant to be related.

What to Do

  • Design Change Test people separately, and ask them not to discuss the task between sessions.
  • Alternative Test Multilevel Regression (HLM). If people come in clusters such as classrooms, estimate the variance components with a multilevel model.
4. Roughly Normal True Scores and Errors (for the Confidence Interval) Targets' true scores and the measurement errors are roughly bell-shaped, for the interval.
What It Means
The intraclass correlation itself is a ratio of variances and needs no distribution. Its F-based confidence interval and F test assume that the targets' true scores and the errors around them are roughly normally distributed.
Applies To
Residuals
How to Check
Look at a histogram of the targets' average scores and a histogram or Q–Q plot of the residuals (each rating minus its target's average and its rater's average, plus the overall average) for strong skew or a few extreme values. Don't rely on a normality test.
How Much It Matters
Mostly for the confidence interval, which assumes roughly normal differences between people and roughly normal errors. The ICC itself is a ratio of variances and is still a fair description. Our scores are mildly left-skewed (skewness −0.33 and −0.48), which barely matters with 60 students.

What to Do

  • Robust Estimation Bootstrap the ICC by resampling students, and report the percentile interval.
  • Transformation Log-transform right-skewed measures such as reaction times before computing the ICC. Their errors also tend to grow with the score, which a log scale evens out. The result describes reliability on the log scale.
5. Same Size of Error Across Scores (for Judging Individual Change) Scores are about equally noisy for low and high scorers.
What It Means
The measurement error is about the same size whatever a person's score, so one standard error of measurement fits everyone. When error grows with the score, as it often does for reaction times, the SEM, the smallest detectable change and the limits of agreement are too wide for low scorers and too narrow for high ones.
Applies To
Residuals
How to Check
Plot each person's change between sessions against their average score (a Bland–Altman plot) and look for a funnel that widens as scores rise.
How Much It Matters
For the standard error of measurement, the smallest detectable change and the limits of agreement, which use one error size for everyone. If error grows with the score, they're too wide for low scorers and too narrow for high ones. The ICC itself is less affected.

What to Do

  • Transformation If the spread of the differences grows in proportion to the score, work on the log scale and report the limits as ratios.
  • Proceed with Caveat Give the standard error of measurement separately for low, middle and high scorers.

Limitations

This Is Not Your Test If…

What It Doesn't Tell You

  • Validity. Consistent scores aren't necessarily scores of working memory; a task could reliably measure typing speed instead.
  • Reliability everywhere. The ICC describes these 60 students over a two-week gap. In a group whose scores vary less, such as only high-performing students, the same measurement error would give a lower ICC.
  • Whether it's good enough. Benchmarks are conventions. A .74 may do for comparing group averages in research but not for decisions about one person, where the 12-to-14-letter smallest detectable change is what matters.
  • Why scores rose. The 4-letter gain looks like practice, but the design can't rule out other reasons, such as students sleeping better or being tested at a better time of day in the second session.

Common Mistakes

  • Don'tReport Pearson's r as the test-retest reliability.

    InsteadReport a single-measures, absolute-agreement ICC when scores from the two sessions should be interchangeable, say whether the sessions are treated as random or fixed, and report the mean change with it. Or remove the shift by design and report the consistency ICC alongside the shift. r ignores a practice effect, so it can look high when people's two scores disagree.

  • Don'tTreat any retest gain in one person as real improvement.

    InsteadSubtract the average practice gain first, then compare what's left with the smallest detectable change. Smaller changes can't be told apart from measurement error.

  • Don'tCompare ICCs from samples whose scores vary very differently.

    InsteadCompare SEMs, or note the spread. A more varied sample gives a higher ICC with the same measurement error.

  • Don'tReport “the ICC” without saying which one.

    InsteadName the model, the type (single or average) and the definition (agreement or consistency), such as a two-way random-effects, absolute-agreement, single-rater ICC(2,1).

  • Don'tLabel reliability from the point estimate alone.

    InsteadJudge the whole 95% confidence interval against the benchmarks, as Koo and Li (2016) recommend.

  • Don'tConfuse the standard error of measurement with the standard error of the mean.

    InsteadSay which you mean. The SEM is about one person's score and doesn't shrink with a bigger sample; the standard error of a mean does.

Result Decoder

Interpretation Guidelines

Benchmarks ICC(2,1)
  • ICC < .50 Poor

    Less than half the variation in scores reflects real differences between targets. Scores from different raters or occasions can differ a lot for the same target.

  • .50 ≤ ICC < .75 Moderate

    Most of the variation is real, but a sizable share is raters, occasions or noise. Sometimes treated as enough for comparing groups, rarely for decisions about individuals.

  • .75 ≤ ICC ≤ .90 Good

    About three quarters or more of the variation reflects real differences between targets.

  • ICC > .90 Excellent

    Almost all the variation reflects real differences. Often treated as the minimum for decisions about individuals.

Koo and Li (2016) proposed these ranges, and recommend judging reliability from where the 95% confidence interval falls, not the point estimate alone. They offer them for studies with at least 30 heterogeneous people and at least 3 raters; with only two sessions, lean even more on the interval. They are conventions, not a verdict on whether a measure is good enough for your purpose.

More about effect size ↓

How It's Reported

Equation Form

Example

ICC(2,1) = .74, 95% CI [.46, .87]; r = .80; Mchange = 4.00, 95% CI [2.37, 5.63]; SEM = 5.15 (4.47 from residual error); SDC = 14.27 (12.40 around the 4-letter gain)

Table 1. Reliability Coefficients for the Working-Memory Span Task

Example
Estimate95% CI
ICC(2,1), absolute agreement .74[.46, .87]
ICC(3,1), consistency .80[.69, .88]
Pearson's r .80[.69, .88]

Note. n = 60 students, two sessions two weeks apart. ICC(2,1) = two-way, absolute agreement, single measures; ICC(3,1) = two-way, consistency, single measures. ICC intervals follow McGraw and Wong (1996); the interval for r uses Fisher's z. We report ICC(2,1).

Table 2. Agreement between Sessions, in Letters Recalled

Example
Letters95% CI
Mean change (retest gain) 4.00[2.37, 5.63]
SEM, pooled-SD approach 5.15
Smallest detectable change, pooled-SD approach 14.27
SEM, residual-error approach 4.47
Individual-change threshold around +4 (±) 12.40

Note. Change = session 2 − session 1. Pooled-SD approach: SEM = SD of all 120 scores (10.16) × √(1 − ICC(2,1)); SDC = 1.96 × √2 × SEM, measured from zero. Residual-error approach: SEM = √MSE, which leaves the average gain out; the threshold is 1.96 × √2 × SEM either side of the 4.00-letter gain, so an observed change between −8.40 and +16.40 letters is within measurement error.

Full Text (APA Style)

Example

Sixty students completed a working-memory span task twice, two weeks apart. Scores rose from the first session (M = 50.00, SD = 10.00) to the second (M = 54.00, SD = 10.00), a mean gain of 4.00 letters, 95% CI [2.37, 5.63], consistent with a practice effect. Test-retest reliability, estimated with a two-way, absolute-agreement, single-measures intraclass correlation, was moderate, ICC(2,1) = .74, 95% CI [.46, .87], although the two sessions ranked students similarly (r = .80). The conventional pooled-SD calculation gave a standard error of measurement of 5.15 letters and a smallest detectable change of 14.27 letters. To judge whether one student changed beyond the expected 4-letter retest gain, we used the residual error instead (SEM = 4.47 letters), so a change had to fall more than 12.40 letters from +4 to stand out from measurement error.

  • Name the ICC fully: model (two-way), definition (absolute agreement) and unit (single measures), or the Shrout and Fleiss label, ICC(2,1).
  • Report the mean change between sessions. A practice effect is part of the answer, not a nuisance to leave out.
  • Give the SEM and smallest detectable change in the measure's own units, so readers can judge individual scores.
  • State the retest interval and the sample. Reliability belongs to scores from these people over this gap, not to the task in general.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how certain.

Mchange = 4.00 Students recalled 4 more letters on average the second time. That's most likely practice, though the design can't rule out other reasons, such as a better time of day for the second session. Either way, test-retest reliability counts it as disagreement between the sessions.
r = .80 Students kept much the same order across the two sessions. This is shown only for contrast; it isn't the reliability, because it ignores the 4-letter gain.
ICC(2,1) = .74 About three quarters of the variation in a single score is real, stable difference between students; the rest is noise and the average gain. That's moderate by the benchmarks above, just short of good. The consistency version, .80, and r, .80, look better only because they ignore the average gain. Benchmarks ↑
95% CI [.46, .87] The true reliability is plausibly anywhere from poor (.46) to good (.87). Judged by its interval, as Koo and Li (2016) recommend, the task's reliability is poor to good: we can't yet call it good.
SEM = 5.15 A single score is typically about 5 letters from the student's true score. For a student who scores 50, a rough 50 ± 1.96 × 5.15 gives about 40 to 60, which shows how large that uncertainty is; it isn't a formal confidence interval for that student's true working memory.
SDC = 14.27 Because students gained 4 letters on average, judge one student's change against +4: it stands out from measurement error only if it's more than about 12 letters away from that expected gain (the conventional 14-letter figure measures from zero and counts the shared gain as error). A change beyond that is unusual for this task, not proof of real improvement or decline, and the threshold is itself an estimate. The group's 4-letter gain is well pinned down, 2.37 to 5.63 letters, yet far too small to see in any one student.

Calculations

The Running Example

Example

The working-memory study: 60 students, each taking the same working-memory span task twice, two weeks apart, scored as letters recalled (0 to 75).

The main goal is to estimate agreement, not to test whether it's greater than zero; a significant ICC can still be too low for the measure's intended use. The question is how closely the sessions agree, so the page estimates the population ICC(2,1), with a confidence interval, and turns it into points on the task. Change scores are session 2 minus session 1, so a positive change is a gain at retest.

What Your Data Should Look Like

One row per person and one numeric column for each session, in the same units. Both scores in a row must belong to the same person; anyone missing a session drops out. Software that expects long data wants one row per person per session instead, with a column saying which session.

Example
studentsession_1session_2
14152
25651
35163
43448
55150
65756
… 54 more rows
nMSD
Session 16050.0010.00
Session 26054.0010.00
Difference604.006.32

Difference = Session 2 − Session 1.

Pearson correlation between the two: r = .80

Simulated data.

Download all 60 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the working-memory study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

  1. 1. Find How Much Scores Rose the Second Time

    Before asking whether students kept their scores, look at whether scores moved on average. A shift shared across students is exactly what a correlation can't see.

    \bar{D} = \dfrac{\sum D_i}{n} =\dfrac{240}{60}

    = 4.00 letters

    s_D = 6.32 letters

    D_i is each student's session 2 score minus their session 1 score; student 1 went from 41 to 52, a gain of 11. \sum D_i adds all 60 changes to 240, and 240 ÷ 60 = 4.00 letters, the same as 54.00 − 50.00.

  2. 2. Give a Range of Plausible Values for the Retest Gain

    A 4-letter gain in 60 students could be partly chance. The interval for the average change shows how large the group's gain plausibly is.

    \bar{D} \pm t^{*}_{df} \times SE =4.00 \pm 2.001 \times \dfrac{6.32}{\sqrt{60}}

    \text{lower} = 2.37 letters

    \text{upper} = 5.63 letters

    s_D is the 6.32-letter spread of the changes, so SE = 6.32 ÷ √60 = 0.816. t^{*}_{59} = 2.001, and 2.001 × 0.816 = 1.63, so the range is 4.00 ± 1.63, from 2.37 to 5.63 letters.

  3. 3. Check Whether the Two Sessions Put Students in the Same Order

    The usual first look is the correlation between sessions. It says whether students who did well the first time also did well the second, but it's blind to an average gain.

    r = \dfrac{\text{cov}_{xy}}{s_x \, s_y} =\dfrac{79.97}{10.00 \times 10.00}

    = .80

    \text{cov}_{xy} is how the two sessions move together, 79.97 letter², and s_x and s_y are both 10.00 letters. 79.97 ÷ 100.0 = .80. Add 4 letters to everyone's second score, or 40, and r stays .80.

  4. 4. Split the Variation into Students, Sessions and Noise

    Scores differ for three reasons. Some students have better working memory than others, scores were higher on average the second time, and each score has some noise. Each source's sum of squares adds up squared distances from the grand mean; dividing it by its degrees of freedom gives a mean square, an average square that puts the three on a common scale.

    MS_S = \dfrac{SS_{\text{students}}}{n - 1}, \quad MS_O = \dfrac{SS_{\text{sessions}}}{k - 1}, \quad MS_E = \dfrac{SS_{\text{error}}}{(n - 1)(k - 1)} =\dfrac{10616}{59}, \;\; \dfrac{480}{1}, \;\; \dfrac{1180}{59}

    = 20.00

    MS_S = 179.93

    MS_O = 480.00

    n is our 60 students and k our 2 sessions; the O in MS_O stands for occasions. For sessions, each session mean is 2 letters from the grand mean of 52, and each counts once for every student, so the sum of squares is 60 × (2² + 2²) = 480, on 1 degree of freedom, and MS_O = 480. For noise, with two sessions the leftover inconsistency is half the variance of the change scores, MS_E = s_D^2 ÷ 2 = 40.00 ÷ 2 = 20.00, the same as 1180 ÷ 59. For students, their own averages spread out to MS_S = (s_1^2 + s_2^2 + 2\,\text{cov}) ÷ 2 = (99.93 + 100.00 + 2 × 79.97) ÷ 2 = 179.93, the same as 10616 ÷ 59.

  5. 5. Find the Share of the Variation That's Real Differences between Students

    Reliability is the share of the spread in single scores that reflects stable differences between people. Absolute agreement puts the session gap in the denominator, so the practice effect counts against the measure.

    \text{ICC}(2,1) = \dfrac{MS_S - MS_E}{MS_S + (k - 1)\,MS_E + \dfrac{k}{n}(MS_O - MS_E)} =\dfrac{179.93 - 20.00}{179.93 + 20.00 + \tfrac{2}{60}(480 - 20.00)}

    = .74

    \text{ICC}(3,1) = .80

    The top, 179.93 − 20.00 = 159.93, is the real spread between students. The bottom adds the noise and \tfrac{2}{60} × 460 = 15.33 for the session gap, 215.27 in all, and 159.93 ÷ 215.27 = .74. Leave the 15.33 out and it's .80, the consistency ICC(3,1).

  6. 6. Give a Range of Plausible Values for the True Reliability

    Sixty students and only two sessions leave a lot of uncertainty. This standard interval treats the two sessions as a sample of possible occasions, so most of its width comes from having just two session averages to judge occasion-to-occasion variation by. (An interval targeting agreement between these two fixed visits may differ; see Precision and Sample Size.) The interval shows how low and how high the true value could plausibly be.

    \text{ICC}_{\text{lower}} = \dfrac{n\,(MS_S - F^{*}\,MS_E)}{F^{*}\left[k\,MS_O + (kn - k - n)\,MS_E\right] + n\,MS_S} =\dfrac{60\,(179.93 - 2.693 \times 20.00)}{2.693\,[2 \times 480 + 58 \times 20.00] + 60 \times 179.93}

    = [.46, .87]

    F^{*} = 2.693 is the 97.5th percentile of an F distribution with 59 and 13.26 degrees of freedom, and kn - k - n = 120 − 2 − 60 = 58. The 13.26 isn't a whole number because it's an approximation that mixes the session gap with the noise; the bigger the gap relative to the noise, the smaller it gets, and that small number is why the lower limit sits so low. The top is 7564.4 and the bottom 16505.0, so the lower limit is .46. The upper limit uses a second F value in the same way and comes to .87; software finds both.

  7. 7. Turn Reliability into Letters on the Task

    A coefficient is hard to picture. The standard error of measurement says how far one student's score typically sits from their true score, in the task's own units.

    SEM = SD\sqrt{1 - \text{ICC}} =10.16 \times \sqrt{1 - .743}

    = 5.15 letters

    SD = 10.16 letters

    \sqrt{MS_E} = 4.47 letters

    SD is the spread of all 120 scores around their grand mean of 52, 10.16 letters, and 1 - \text{ICC} = .257, whose square root is 0.507. 10.16 × 0.507 = 5.15 letters. Weir and Vincent's (2021) preferred alternative, \sqrt{MS_E} = √20.00 = 4.47 letters, leaves the practice effect out.

  8. 8. Find the Smallest Change That Stands Out from the Noise

    When one student's score changes, both scores carry error. The smallest detectable change is the gap that noise alone would rarely produce, so smaller changes can't be taken as real for that person.

    SDC = 1.96 \times \sqrt{2} \times SEM =1.96 \times 1.414 \times 5.15

    = 14.27 letters

    1.96\sqrt{2}\sqrt{MS_E} = 12.40 letters

    \sqrt{2} × SEM = 1.414 × 5.15 = 7.28 letters is the standard error of one student's change, and 1.96 of those is 14.27 letters; 5 of our 60 students gained more than that. But students gain about 4 letters on average, so judge one student's change against +4, not 0, and use the \sqrt{MS_E} SEM, which leaves the shift out so it isn't counted twice. The same sum then gives 12.40 letters. Only 2 students changed by more than 12.40 letters from +4, one gaining 19 and one dropping 9.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)
  1. Analyze → Scale → Reliability Analysis.
  2. Move session_1 and session_2 into Items.
  3. Click Statistics. Under ANOVA Table choose F test, and check Intraclass correlation coefficient.
  4. Set Model to Two-Way Random and Type to Absolute Agreement, leave the confidence interval at 95% and the test value at 0, and click Continue, then OK.

Or paste this syntax:

RELIABILITY
  /VARIABLES=session_1 session_2
  /SCALE('ALL VARIABLES') ALL
  /MODEL=ALPHA
  /STATISTICS=ANOVA
  /ICC=MODEL(RANDOM) TYPE(ABSOLUTE) CIN=95 TESTVAL=0.

Reading the Output

Read the Single Measures row of the Intraclass Correlation Coefficient table: the ICC, then its 95% Confidence Interval. That's ICC(2,1). Skip the F test: it only asks whether the ICC is above zero.

Intraclass Correlation Coefficient

Intraclass Correlation 95% CI Lower Bound Upper Bound F Test with True Value 0: Value df1 df2 Sig
Single Measures .743 .458 .867 8.997 59 59 <.001
Average Measures .853 .629 .929 8.997 59 59 <.001

Two-way random effects model, absolute agreement definition. Average Measures is the reliability of the average of both sessions, not of one score, so don't report it unless you'll use averages. With Two-Way Mixed chosen instead, SPSS gives the same absolute-agreement estimate and the same confidence limits; choosing Mixed doesn't produce the fixed-visit bootstrap interval described under Precision and Sample Size, which answers a different question.

ANOVA

Sum of Squares df Mean Square F Sig
Between People 10616.000 59 179.932
Within People Between Items 480.000 1 480.000 24.000 <.001
Residual 1180.000 59 20.000
Total 1660.000 60 27.667
Total 12276.000 119 103.160

Grand Mean = 52.00. These are the Napkin Math mean squares: MS_S = 179.93, MS_O = 480 and MS_E = 20.00. SPSS doesn't print the SEM, so compute it from this table: the SD of all 120 scores is √103.160 = 10.16, so SEM = 10.16 × √(1 − .743) = 5.15 and SDC = 1.96 × √2 × 5.15 = 14.27. The F for Between Items tests the practice effect (it equals the paired t squared). The Reliability Statistics table's Cronbach's alpha, .889, treats the two sessions as items; ignore it here.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R 
library(irr)
retest <- read.csv("data.csv")
sessions <- retest[, c("session_1", "session_2")]

icc(sessions, model = "twoway", type = "agreement", unit = "single")
cor(retest$session_1, retest$session_2)
t.test(retest$session_2, retest$session_1, paired = TRUE)$conf.int

# SEM and smallest detectable change
icc2 <- icc(sessions, model = "twoway", type = "agreement")$value
sem <- sd(c(retest$session_1, retest$session_2)) * sqrt(1 - icc2)
c(sem = sem, sdc = 1.96 * sqrt(2) * sem)

 Single Score Intraclass Correlation

   Model: twoway
   Type : agreement

   Subjects = 60
     Raters = 2
   ICC(A,1) = 0.743

 F-Test, H0: r0 = 0 ; H1: r0 > 0
   F(59,59) = 9 , p = 5.27e-15

 95%-Confidence Interval for ICC Population Values:
  0.458 < ICC < 0.867

[1] 0.7999322

[1] 2.366194 5.633806
attr(,"conf.level")
[1] 0.95

      sem       sdc
 5.149453 14.273556

ICC(A,1) is McGraw and Wong's name for ICC(2,1). type = "consistency" gives ICC(3,1), .80. psych::ICC() prints all six Shrout and Fleiss forms at once; read the ICC2 row.

Output from a real run: R 4.6.1 (irr 0.85), run 2026-10-03.

Python 
import numpy as np
import pandas as pd
import pingouin as pg
from scipy import stats

retest = pd.read_csv("data.csv")

# pingouin wants long data: one row per student per session
long = retest.melt(id_vars="student", var_name="session", value_name="score")
icc = pg.intraclass_corr(data=long, targets="student", raters="session", ratings="score")
print(icc.set_index("Type").loc[["ICC2", "ICC3"], ["ICC", "CI95%"]])

r = stats.pearsonr(retest["session_1"], retest["session_2"]).statistic
ci = stats.ttest_rel(retest["session_2"], retest["session_1"]).confidence_interval()
print(f"r = {r:.4f}; mean change 95% CI [{ci.low:.4f}, {ci.high:.4f}]")

# SEM and smallest detectable change
icc2 = icc.set_index("Type").loc["ICC2", "ICC"]
sem = long["score"].std() * np.sqrt(1 - icc2)
print(f"SEM = {sem:.4f}, SDC = {1.96 * np.sqrt(2) * sem:.4f}")

           ICC         CI95%
Type                        
ICC2  0.742953  [0.46, 0.87]
ICC3  0.799932  [0.69, 0.88]
r = 0.7999; mean change 95% CI [2.3662, 5.6338]
SEM = 5.1495, SDC = 14.2736

Pingouin needs long data, one row per student per session, hence the melt. Read the ICC2 row for ICC(2,1); pingouin rounds its interval to two decimals.

Output from a real run: Python 3.9.6 (pingouin 0.5.5, scipy 1.12.0, pandas 2.3.3), run 2026-10-03.

Impact

Effect Size

Test-retest reliability has no effect size in the usual sense; the ICC itself is the size of the result. It usually runs from 0 to 1 and says what share of the variation in single scores is real, stable difference between people. It has no units, so it can be compared across measures scored on different scales, but it still depends on how varied the sample is. The absolute-agreement version is the usual choice for test-retest studies, because scores from the two sessions are meant to be interchangeable. Koo and Li (2016) recommend absolute agreement in a two-way mixed-effects model. For a study concerned specifically with its two scheduled visits, that's often appropriate; a random-effects model instead targets agreement across a broader population of possible occasions. Its single-measures value comes from the same formula as Shrout and Fleiss's (1979) two-way random ICC(2,1), also written ICC(A,1) (McGraw & Wong, 1996), so this page reports it under that label. The point estimate is the same either way, but the claims differ: a random model treats the two sessions as a sample of possible occasions, while a mixed model is about agreement between these two visits. Say which you mean. It matters most for the confidence interval (see Precision and Sample Size). Weir (2005) takes another route: remove the shift by design, such as with a practice session before the first real one, and then use the consistency ICC. Either way, report the shift.

\text{ICC}(2,1) = \dfrac{179.93 - 20.00}{179.93 + 20.00 + \tfrac{2}{60}(480 - 20.00)} = .74 \qquad r = .80

The sessions kept students in much the same order (r = .80), and scores rose by 4 letters on average, though 16 of the 60 students scored lower at retest and 5 the same. Counting that gain as disagreement brings the reliability down to .74: moderate by convention, just short of good.

The gap between r and the ICC is the average gain. A correlation only asks whether the two sets of scores rise and fall together, so adding the same amount to everyone's second score would leave it at .80 however large the gain (Aldridge et al., 2017). A practice effect doesn't stop the ICC being calculated; it changes what it means, so report the gain with it. The consistency ICC, ICC(3,1), ignores the gain the same way, and when the two sessions are about equally spread out it's all but identical to r (.80 here too).

In the task's own units, the standard error of measurement is 5.15 letters and the smallest detectable change is 14.27 letters. We computed the SEM as SD × √(1 − ICC(2,1)), one of the two formulas Weir and Vincent (2021) give; because it uses the absolute-agreement ICC, the average gain counts against it.

An SEM of 5.15 letters is large next to one student's score: a rough 50 ± 1.96 × 5.15 gives about 40 to 60 for a student who scores 50. That shows the scale of the uncertainty; it isn't a formal confidence interval for that student's true working memory, since it leans on classical assumptions and ignores regression toward the mean. To say one student's score really changed, it would need to move more than 12 letters away from the average gain of 4 (see below).

Weir and Vincent (2021) recommend the other formula, √MSE, because it doesn't depend on which ICC you choose. It leaves the average gain out, and here gives √20.00 = 4.47 letters, with a smallest detectable change of 12.40 letters. Either SEM gives a threshold beyond which an observed change is unlikely under the measurement-error model (Weir, 2005). Exceeding it is evidence of change, not proof of its cause or that it matters clinically. Say which formula you used.

When scores rise on average between sessions, don't measure one person's change from zero. Judge it against the average gain, with the √MSE margin so the shift isn't counted twice: here a change counts only if it's more than 12.40 letters away from +4, which 2 of our 60 students show. A change outside that range is unusual relative to this study's retest variation, not automatic proof of real improvement or decline, and the threshold is itself estimated with uncertainty. In clinical psychology this is the reliable change index (Jacobson & Truax, 1991), centred on the average retest gain when there is one (Chelune et al., 1993). Bland and Altman's (1986) limits of agreement describe the same spread for the group: about 95% of students' changes should fall between 4.00 − 1.96 × 6.32 and 4.00 + 1.96 × 6.32, or −8.40 to 16.40 letters. These limits describe how individual changes spread, not a confidence interval for the average change. All of these assume the error is about the same size for low and high scorers (Atkinson & Nevill, 1998); here the spread of the changes doesn't grow with students' average scores.

Whether a 12-to-14-letter margin is precise enough depends on the decision the scores feed, not on the benchmarks.

Benchmarks ICC(2,1) at a Glance
Poor
ICC < .50
Moderate
.50 ≤ ICC < .75
Good
.75 ≤ ICC ≤ .90
Excellent
ICC > .90

Full benchmarks and caveats ↑

Uncertainty

There's no p-value worth reporting for test-retest reliability. Software will test whether the ICC is above zero, but that answers a question nobody asked: the same test given twice to the same people is almost always related, and a significant result says nothing about whether the measure is reliable enough. The confidence interval does the work instead. Read where it falls against the benchmarks, or check that its lower limit clears the smallest reliability you'd accept, set before collecting the data.

SPSS and R report F(59, 59) = 9.00, p < .001 for our ICC, which only says the reliability isn't zero. The interval, .46 to .87, is the informative part: it runs from poor to good, so 60 students can't show that the task is good.

The retest gain's interval, 2.37 to 5.63 letters, lies wholly above zero: if the true average change were zero, a mean change at least this far from zero, in either direction, would turn up less than 5% of the time. That's a finding about the task, worth reporting, not a problem with the analysis. A p-value is not a measure of how reliable a measure is, and neither the ICC's test nor the practice effect's says whether single scores are precise enough to use.

Precision and Sample Size

Power is the chance that a study detects an effect that's really there. For a reliability study the question is different: not whether reliability is above zero, which it almost always is, but whether the estimate is precise enough to say how reliable the measure is. So plan the sample for a narrow enough confidence interval rather than for power.

Suppose the true values were the ones behind our simulation: a reliability of .80 without the practice effect, and a 4-letter gain at retest. With 30 students the expected interval for ICC(2,1) runs from about .40 to .88; with 60, .45 to .87; even with 200, .48 to .85. Without a practice effect, 60 students would give about .69 to .88, and 200 would give .74 to .84.

The usual interval treats the two sessions as a sample from all the occasions you might have used, so with a practice effect it stays wide however many people you add: two occasions say little about how much occasions vary. SPSS and R give this interval for the random and mixed models alike. If you care only about agreement between these two fixed visits, the gain itself is pinned down better as people are added, and a participant-level bootstrap can estimate the uncertainty for that question. Resampling whole students (each keeping both scores), recomputing ICC(A,1) each time, 4,000 times, gives a 95% percentile interval of about .60 to .85 here (the lower limit moves by about .01 between runs), rather than .46 to .87. That answers a different question from the usual interval, and how much narrower it comes out depends on the data. More people help when sessions agree on average; when they don't, a practice session before the first real one, which uses up the gain, helps more. Koo and Li (2016) suggest at least 30 people with varied scores as a rule of thumb; the numbers above show that 30 can still leave a very wide interval.

The planner below works out the exact interval for the consistency ICC, the kind of exact interval Bonett's (2002) method plans for. That's the case with no practice effect, so its numbers match the no-practice-effect figures above: 55 students for an interval .20 wide at a reliability of .80. If you expect scores to shift between sessions, expect to need more students than it says, though it isn't a guaranteed minimum. Try it:

Other Info

Follow-Up and Post Hoc Tests

None. This page estimates rather than tests, so there are no follow-up comparisons to run.

References

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
    “guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges” (Abstract)
  • McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30
  • Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
    “absolute agreement definition should always be chosen for both test-retest and intrarater reliability studies” (How to Select the Correct ICC Form for Test-Retest and Intrarater Reliability Studies)
    “Reliability is defined as the extent to which measurements can be replicated.” (Introduction)
    “Under such conditions, we suggest that ICC values less than 0.5 are indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and values greater than 0.90 indicate excellent reliability.” (ICC Interpretation)
    “the 95% confident interval of the ICC estimate (not the ICC estimate itself) should be used as the basis to evaluate the level of reliability” (Discussion for Readers)
  • Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19(1), 231–240. https://doi.org/10.1519/15184.1
    “It is shown how the SEM and its variants can be used to construct confidence intervals for individual scores and to determine the minimal difference needed to be exhibited for one to be confident that a true change in performance of an individual has occurred.” (Abstract)
  • Weir, J. P., & Vincent, W. J. (2021). Statistics in kinesiology (5th ed.). Human Kinetics. https://web.archive.org/web/20251007145530/https://us.humankinetics.com/blogs/excerpt/standard-error-of-measurement
    “Instead, we recommend using an alternative approach to estimating the standard error of measurement” (Standard error of measurement (Human Kinetics excerpt))
    “This standard error of measurement value does not vary depending on the intraclass correlation coefficient model used because the mean square error is constant for a given set of data.” (Standard error of measurement (Human Kinetics excerpt))
    “SD is the standard deviation of all the scores about the grand mean.” (Standard error of measurement (Human Kinetics excerpt))
    “the ability to estimate the change in performance or minimal difference needed to be considered real (sometimes called the minimal detectable change or the minimal detectable difference)” (Standard error of measurement (Human Kinetics excerpt))
  • Aldridge, V. K., Dovey, T. M., & Wade, A. (2017). Assessing test-retest reliability of psychological measures: Persistent methodological problems. European Psychologist, 22(4), 207–218. https://doi.org/10.1027/1016-9040/a000298
  • Bonett, D. G. (2002). Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine, 21(9), 1331–1335. https://doi.org/10.1002/sim.1108
    “A method is developed to calculate the approximate number of subjects required to obtain an exact confidence interval of desired width for certain types of intraclass correlations in one-way and two-way ANOVA models.” (Abstract)
  • Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. https://doi.org/10.1037/0022-006X.59.1.12
  • Chelune, G. J., Naugle, R. I., Lüders, H., Sedlak, J., & Awad, I. A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7(1), 41–52. https://doi.org/10.1037/0894-4105.7.1.41
  • Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
  • Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26(4), 217–238. https://doi.org/10.2165/00007256-199826040-00002
  • Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
  • Liljequist, D., Elfving, B., & Skavberg Roaldsen, K. (2019). Intraclass correlation – A discussion and demonstration of basic features. PLOS ONE, 14(7), e0219854. https://doi.org/10.1371/journal.pone.0219854
    “However, whether an ICC value is good enough should depend on the intended use of the method.” (1. Introduction)

Spot an error? Something still confusing? Tell us at support@statsforpsychology.com.