Stats for
Menu

ANOVA Family · General Linear Model

Factorial Repeated-Measures ANOVA

A factorial repeated-measures ANOVA tests two within-person factors and their interaction.

A factorial repeated-measures ANOVA takes one continuous outcome, measured in the same people under every combination of two within-person factors, and tests each factor's main effect and their interaction.

Example

Does time pressure change how much conflicting color words slow people down? Twenty-eight adults named the ink color of color words that matched or clashed with the ink, with and without a response deadline. Matching and clashing words were mixed within each block, and the deadline and no-deadline blocks came in counterbalanced order. Each person's average reaction time in all four conditions is one score.

Two Crossed Factors + Both Within Subjects + Continuous Outcome → Factorial Repeated-Measures ANOVA
Tests
Main effects and interaction of two within-subjects factors on a population mean
Observations
Related
Parametric
Yes
Headline Statistic
F
Effect Size
Generalized Eta Squared
Groups
4+

The Basics

When to Use It

Use it when the same people are measured under every combination of two factors, on one continuous outcome, and you want to know whether each factor matters and whether one factor's effect depends on the other. For example:

  • Does time pressure change how much conflicting color words slow people down?
  • Does the benefit of retrieval practice over rereading on recall scores grow from a same-day test to a one-week test?
  • Do heart-rate responses to angry versus neutral faces differ between masked and unmasked presentations?

Vocabulary You Need

Definition Main Effect

How much the outcome differs across the levels of one factor, averaging over the levels of the other factors. It is read from the marginal means: for each level, the average of its cell means, each cell counted equally. When the factors interact, a main effect is an average of effects that differ, so it can describe no actual condition well. Glossary →

Definition Interaction Effect

The part of a factorial design's result that neither factor explains on its own: the effect of one factor changes depending on the level of the other. In a 2 × 2 design it is a difference of differences, the effect of one factor at one level of the other minus its effect at the other level. On a plot of the cell means, clearly non-parallel lines suggest one; the interaction test says whether the departure is bigger than noise. An interaction is symmetric: if A's effect depends on B, B's effect depends on A. Glossary →

Definition Within-Subjects Factor

A factor whose levels are all measured in the same people, such as before and after a program, or three task conditions each person completes. Differences between its levels are compared within each person, so stable differences between people cancel out, which usually makes these comparisons more precise than comparisons between groups. The price is that a person's scores are related, and the analysis has to allow for that. Also called a repeated-measures factor. Glossary →

Definition Generalized Eta Squared (\eta^2_G)

An effect's sum of squares divided by itself plus every source of variation that doesn't come only from manipulated factors: stable differences between people, measured factors such as gender (and their interactions), covariates, and error. The other manipulated effects are left out. For manipulated factors, that makes it comparable between between-subjects and within-subjects designs, where partial eta squared isn't. In a design with only manipulated between-subjects factors and no covariate it equals partial eta squared. A covariate, a measured factor, or differences between people in a repeated-measures design stay in its denominator, so there it's smaller, often much smaller. Glossary →

Definition F Statistic (F)

The variance due to an effect divided by the error variance it is tested against: in a between-subjects ANOVA, the between-groups variance over the within-groups variance; in a repeated-measures ANOVA, the conditions' variance over the inconsistency in how people change across them. It is near 1 when there's no real effect and grows as the effect stands out from the error. It is compared with an F distribution, which has two degrees of freedom, to get a p-value. Glossary →

Assumptions and Inference

1. Independent Participants (for the p-Values and Intervals) Each person's four scores belong together, but different people are independent of one another. The scores within a person are expected to be related; treating them as independent would use the wrong error terms.
What It Means
Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To
Observations
How to Check
By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters
A lot. Each person's four scores are meant to be related, but different people must not influence each other. If people were tested in pairs and could hear each other respond, their reaction times are more alike than the test expects, and p comes out too small.

What to Do

  • Alternative Test Multilevel Regression (HLM). A multilevel model with random effects for people and for the clusters they share.
  • Design Change Test each person alone, so no one's responses can affect anyone else's.
2. Normality of the Within-Person Contrasts (for the p-Values and Intervals) The classical tests assume normal errors for the within-person contrasts that define each effect, not that reaction times as a whole are bell-shaped. Moderate departures are often tolerated, especially with more people, but no sample size makes them automatically irrelevant, and strong skew or extreme values can still matter.
What It Means
The test's model assumes the scores it relies on are roughly normal: the scores within each group (or the scores themselves, for a one-sample test), the difference scores in a paired design, or the residuals (the errors left around a model's predictions) in a regression. For p-values and confidence intervals, what really matters is that the estimates (such as a mean, a difference between means or a model's coefficients) are roughly normal across repeated samples. A large enough sample usually provides that; how large depends on how skewed the scores are and how extreme their outliers.
Applies To
Residuals
How to Check
Compute each person's three contrasts: their congruency difference (incongruent − congruent, averaged over deadline), their time-pressure difference (no deadline − deadline, averaged over congruency) and their difference of differences. Look at each one's histogram or Q–Q plot for strong skew or extreme values. Histograms of the raw reaction times help less: they mostly show how differently fast people are. Don't rely on a normality test: with small samples it misses real problems, and with large ones it flags trivial ones.
How Much It Matters
Less as n grows. With two levels per factor, each effect is a test on one difference score per person, so what matters is whether those differences are roughly normal. With 28 people, moderate skew barely matters; a few extreme outliers can. Here the difference of differences is symmetric (skewness 0.01, no value more than 2.5 SDs from its mean); the time-pressure difference is too (skewness −0.29). The congruency difference leans right (skewness 0.94, one person 3.1 SDs out), but its F of 132.69 leaves no doubt. Reaction times are right-skewed, but averaging many trials per condition makes each person's cell mean much less so.

What to Do

  • Transformation Log-transform reaction times before averaging. Effects are then ratios rather than milliseconds, and an interaction can grow, shrink or disappear, so choose the scale before seeing the results.
  • Robust Estimation Summarize each person's trials with a trimmed mean or median instead of the mean, so a few slow trials can't dominate.
  • Alternative Test Multilevel Regression (HLM). A multilevel model fit to the individual trials, which can use a skewed distribution for reaction times.
3. Sphericity For an effect with three or more levels, each person's differences between every pair of levels vary equally. Every effect in this 2 × 2 design has one degree of freedom, so sphericity is automatic; it becomes a concern for effects with more.
What It Means
In a factorial design, sphericity belongs to each effect, not to the conditions as a whole. For an effect with three or more levels, take each person's difference score between every pair of its levels; sphericity means those differences have the same variance for every pair. When it fails, that effect's F comes out too liberal. An effect with two levels has only one set of differences, so it can't fail.
Applies To
Covariance structure
How to Check
For each effect with more than 1 df (a factor with three or more levels, or an interaction involving one), look at its Greenhouse-Geisser epsilon (ε), which software prints per effect; 1 means sphericity holds. Mauchly's test is common, but it misses real violations in small samples and flags trivial ones in large samples, so don't let it decide whether to correct.
How Much It Matters
None here: with two levels per factor, each effect has just one difference score per person, so there's only one variance to compare and sphericity holds automatically (ε = 1). With three or more levels of a factor it can fail, for that factor's main effect and for the interaction separately, and uncorrected F-tests then give p-values that are too small.

What to Do

  • Correction With three or more levels, multiply both df of the affected effect by the Greenhouse-Geisser ε (from 1 down to its floor of 1 over the effect's df) and read p from the smaller df.
  • Alternative Test MANOVA. For an effect with two or more df (a factor with three or more levels, or an interaction involving one), test it as a set of difference scores with MANOVA, which doesn't assume sphericity but needs more people. In a 2 × 2 design there's nothing for it to fix.
4. Complete Data (for Standard Repeated-Measures ANOVA) The standard analysis uses only people with a score in every cell. Missing scores don't break the test's logic, but anyone incomplete is dropped, which costs precision and can bias the result.
What It Means
The test compares conditions within each person, so it needs each person's full set of scores. Anyone missing a condition is dropped from the analysis entirely.
Applies To
Observations
How to Check
Count the people with a missing condition and ask why they're missing. If dropouts are more likely in one condition, dropping them can bias the result.
How Much It Matters
Depends on why data are missing. The ANOVA drops anyone missing a cell. If people who struggled with the deadline stopped responding in that block, dropping them would hide the very effect you're looking for. A mixed model keeps their available scores, but it only fixes that bias if the missingness is explained by what the model includes; it can't correct dropout driven by something unmeasured.

What to Do

  • Alternative Test Multilevel Regression (HLM). A multilevel model uses everyone's available scores instead of dropping incomplete people.
  • Proceed with Caveat Say how many people were dropped and which cells they were missing, so readers can judge the risk.

Limitations

This Is Not Your Test If…

What It Doesn't Tell You

  • Why. Counterbalanced conditions let us say the deadline changed the interference effect in milliseconds, but not why: less time for the word to be read, a different strategy, or more errors on the hard trials that were left out.
  • Whether it depends on the scale. The deadline made every condition faster. Interference was 13% of the congruent time without a deadline and 11% with one, and on log reaction times, which compare ratios instead of milliseconds, the interaction is no longer significant, F(1, 27) = 2.39, p = .13. When one condition's line simply sits lower than the other's, as here, whether there's an interaction can depend on how the outcome is measured; a crossover, where the effect reverses, can't be removed that way (Wagenmakers et al., 2012). Choose the scale before seeing the results, from the question and from how the outcome behaves, not by which one gives the p-value you hoped for.
  • Order. Each person did every condition, so later blocks came after more practice or more fatigue. Counterbalancing the deadline blocks (half the people got the deadline first) keeps order from favoring either one, but it doesn't remove order effects: they add noise to the error terms, and if doing one block first changes how people do the other, no ordering fixes that. Check whether the effects differ by order.
  • Whether it matters. A 20-ms change in interference is statistically significant, but it's under half a percent of the variation in reaction times apart from the two main effects. In the task's own terms, the deadline cut interference by a quarter; whether that matters is a question about attention.
  • Who else. The result describes adults like these 28, doing this task with this deadline. A tighter or looser deadline could change the pattern.
  • What happens on each trial. Averaging each person's trials per condition hides how the deadline changed the slowest responses and the error rate, which a trial-level model could show.

Common Mistakes

  • Don'tAnalyze repeated measurements as if different people produced them.

    InsteadUse the repeated-measures model, so each effect is judged against how consistently people show it.

  • Don'tHold partial η² from a repeated-measures design against Cohen's benchmarks.

    InsteadReport generalized η² for comparisons with other designs, and plan power with the per-person effect size (Cohen's d_z).

  • Don'tRun a separate Friedman test for each factor when the outcome is ranks or a coarse rating.

    InsteadUse a rank-based factorial method that tests both main effects and the interaction, such as the aligned rank transform (in R, ARTool::art with a random effect for each person), or an ordinal mixed model for ratings. Separate Friedman tests can't test the interaction.

  • Don'tIgnore sphericity when a factor has three or more levels.

    InsteadReport Greenhouse-Geisser-corrected df and p, and say that you did.

  • Don'tInterpret a main effect without first looking at the interaction.

    InsteadCheck the interaction first. If it's there, describe the main effect as an average over the other factor, and let simple effects say where the effect holds.

  • Don'tConclude that two effects differ because one is significant and the other isn't.

    InsteadTest the interaction, which compares the two effects directly. The difference between significant and not significant isn't itself significant.

  • Don'tAdd up the partial η² values as if they split the variation between them.

    InsteadRemember each has its own denominator, so they can sum past 1. Report η² if you want shares of the total.

  • Don'tReport partial η² under the label η².

    InsteadSay which one you report. SPSS prints partial η², which is never smaller than η² and is usually larger, especially with other effects or repeated measures.

  • Don'tRead a p-value as the chance that the null hypothesis is true.

    InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance and Effect Size Guidelines

Benchmarks Significance (p Value)
  • p < .001

    Very strong evidence against the null hypothesis.

  • .001 ≤ p < .01

    Strong evidence against the null hypothesis.

  • .01 ≤ p < .05

    Statistically significant at α = .05. Moderate evidence against the null hypothesis.

  • .05 ≤ p < .10

    Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.

  • p ≥ .10

    Not significant. Little evidence against the null, which is not the same as evidence of no effect.

These evidence labels are one common convention for describing p in graded language rather than as significant or not; the boundaries are rough guides, not rules, and other authors draw them differently. Muff et al. (2022) argue for this kind of language of evidence over a single cutoff. α is chosen before the analysis, and p never says how big or important an effect is.

More about statistical significance ↓

Benchmarks Effect Size (Generalized Eta Squared)
  • η²G < .01 Below Small

    Smaller than the conventional small benchmark. Whether an effect this size matters depends on the setting.

  • .01 ≤ η²G < .06 Small

    A small effect next to the background variation: differences between people, measured factors and error. Hard to see without measuring it.

  • .06 ≤ η²G < .14 Medium

    A noticeable effect next to the background variation.

  • η²G ≥ .14 Large

    A large effect next to the background variation.

Cohen's (1988) η² benchmarks, which generalized η² can be held against because it keeps the differences between people in the denominator, as a between-subjects η² would. Lakens (2013) calls this a last resort: comparing with similar effects in the literature is better. Don't hold partial η² against them in a repeated-measures design.

More about effect size ↓

How It's Reported

Equation Form

Example

Mdiff = 20 ms, 95% CI [3.61, 36.39], η²G = .004, η²p = .19, F(1, 27) = 6.27, p = .019

Table 1. Two-Way Repeated-Measures ANOVA for Reaction Time (ms)

Example
SSdfMSFp\eta^2_G\eta^2_p
People 5594582720720.67
Congruency 1372001137200.00132.69< .001.178.83
Error (congruency × person) 27918271034.00
Time pressure 1008001100800.0085.08< .001.138.76
Error (time pressure × person) 31990271184.81
Congruency × time pressure 280012800.006.27.019.004.19
Error (interaction × person) 1206427446.81

Note. Each effect is tested against the error row beneath it, the effect × person interaction: how inconsistently people show that effect. The People row is the stable differences in overall speed; it's set aside from every test but counted in η²G. With two levels per factor, sphericity holds, so no correction applies. η²G = generalized eta squared; η²p = partial eta squared. The interaction is read first: it's the study's question.

Full Text (APA Style)

Example

Incongruent color words slowed people down more without a deadline (interference = 80 ms; congruent M = 600, SD = 70.04; incongruent M = 680, SD = 80.59) than with one (60 ms; congruent M = 550, SD = 74.82; incongruent M = 610, SD = 79.93), a difference of 20 ms, 95% CI [3.61, 36.39]. The interaction was a very small share of the variation in reaction times apart from the two main effects, below the conventional small benchmark, η²G = .004 (η²p = .19). A two-way repeated-measures ANOVA showed it was statistically significant, F(1, 27) = 6.27, p = .019. Both main effects were far larger: incongruent words slowed responses by 70 ms, F(1, 27) = 132.69, p < .001, η²G = .178, large by convention, and the deadline sped them up by 60 ms, F(1, 27) = 85.08, p < .001, η²G = .138, medium and just under the large cutoff. Simple effects showed interference in both time-pressure conditions, t(27) = 9.55 without a deadline and 10.06 with one, both p < .001.

  • Report all three effects, each with its own two degrees of freedom, and start with the interaction if it's the question.
  • Give the cell means, or better, plot them, so readers can see the pattern the interaction summarizes.
  • Say which effect size you report. Give generalized η² for comparisons with other studies; partial η² is what SPSS prints.
  • With three or more levels of a factor, say whether and how you corrected for sphericity (for example, Greenhouse-Geisser), effect by effect.
  • If one factor shifts every condition up or down, as the deadline does here, say that the interaction is on the scale you analyzed; on log reaction times it may not hold.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

Mdiff = 20 ms, 95% CI [3.61, 36.39] Conflicting words cost people 20 ms less when a deadline was running: 80 ms of interference without it, 60 ms with it. The true change could plausibly be anywhere from about 4 to 36 ms.
η²p = .19 The interaction is 19% of itself plus the inconsistency it was tested against. It's what SPSS prints; don't compare it with other designs or with the benchmarks. For planning the next study, use the per-person effect size in Power and Sample Size (Cohen's d_z), or, in G*Power, enter partial η² with the "as in SPSS" effect-size option.
η²G = .004 The interaction is less than half a percent of the variation in reaction times apart from the two main effects, most of which is people differing in overall speed. That's below the small benchmark. In the task's own terms, the deadline cut interference by a quarter, from 80 to 60 ms; whether that matters is a question about attention, not statistics. Benchmarks ↑
F(1, 27) = 6.27 The variation the interaction produces is 6.27 times what people's inconsistency in it would produce by chance alone. In milliseconds, the 20-ms change is 2.50 standard errors from zero; F is that number squared, because it compares squared variation. The 1 and 27 are the degrees of freedom: 1 for the effect and n − 1 for its error.
p = .019 If the interference effect were the same with and without a deadline, an F of 6.27 or more would turn up about 1.9% of the time, under the test's assumptions. That's below .05, so the interaction is statistically significant. Benchmarks ↑

Calculations

The Running Example

Example

The Stroop deadline study: 28 adults, each with an average reaction time in all four combinations of congruency (congruent, incongruent) and time pressure (no deadline, deadline).

Null HypothesisAlternative
Congruency\mu_{\text{I}} = \mu_{\text{C}}\mu_{\text{I}} \neq \mu_{\text{C}}
Time Pressure\mu_{\text{D}} = \mu_{\text{N}}\mu_{\text{D}} \neq \mu_{\text{N}}
Interaction(\mu_{\text{IN}} - \mu_{\text{CN}}) = (\mu_{\text{ID}} - \mu_{\text{CD}})(\mu_{\text{IN}} - \mu_{\text{CN}}) \neq (\mu_{\text{ID}} - \mu_{\text{CD}})

Each \mu (“mu”) is a population mean reaction time: C or I for congruent or incongruent words, N or D for no deadline or deadline. Two-letter \mus, such as \mu_{\text{IN}}, are single conditions. One-letter \mus, such as \mu_{\text{I}}, average over the other factor: the marginal means. This is the interaction's null: the interference effect, incongruent minus congruent, is the same with and without a deadline. The two main effects get their own nulls, that the congruency averages are equal and that the time-pressure averages are equal, and each is tested separately.

What Your Data Should Look Like

One score per person per combination of the two factors. Ours is stored long, one row per person per condition, with columns for the person, each factor and the score (plus a column naming the combined condition). Most point-and-click software, SPSS included, wants it wide instead: one row per person and one column per combination, here four. Anyone missing a combination drops out.

Example
participantcongruencytime_pressureconditionrt_ms
1congruentnonecongruent_none686
1incongruentnoneincongruent_none736
1congruentdeadlinecongruent_deadline600
1incongruentdeadlineincongruent_deadline674
2congruentnonecongruent_none636
2incongruentnoneincongruent_none751
… 106 more rows
No DeadlineDeadlineAll
Congruent600.00 (70.04)550.00 (74.82)575.00
Incongruent680.00 (80.59)610.00 (79.93)645.00
All640.00580.00610.00

Cell M (SD), congruency by time pressure. All gives the marginal means; the grand mean is 610.00.

Both factors are within subjects: each of the 28 participants has a score in all 4 cells, so n = 28 in every cell.

Simulated data.

Download all 112 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the Stroop deadline study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

  1. 1. Find the Average Reaction Time in Each Condition

    Everything in a two-factor design comes from a 2 × 2 table of averages. Its edges, the averages for each level of one factor, show the main effects; its inside shows whether they combine simply.

    \bar{x}_{\text{grand}} = \dfrac{\sum x_i}{N} =\dfrac{68{,}320}{112}

    = 610 ms

    \bar{x}_{\text{CN}} = 600 ms

    \bar{x}_{\text{IN}} = 680 ms

    \bar{x}_{\text{CD}} = 550 ms

    \bar{x}_{\text{ID}} = 610 ms

    \sum x_i is all 112 scores (28 people × 4 conditions) added up, and N is 112. 68,320 ÷ 112 = 610 ms. The cells are 600 and 680 ms with no deadline, 550 and 610 ms with one, so congruent words average 575 ms and incongruent 645; no deadline averages 640 and deadline 580.

  2. 2. Measure How Much Each Effect Moves the Averages

    Each of the three effects boils down to one difference between two averages, each of two cells. For a main effect, those are one factor's two edge averages. For the interaction, they're the table's two diagonals, which works out to half the change in the interference effect. Squaring that difference, once per person, gives each effect's sum of squares.

    SS_{\text{effect}} = n\,\bar{d}^{\,2} =28 \times 70^2,\quad 28 \times 60^2,\quad 28 \times 10^2

    SS_{\text{congruency}} = 137200

    SS_{\text{time}} = 100800

    SS_{\text{interaction}} = 2800

    \bar{d} is 645 − 575 = 70 ms for congruency, 640 − 580 = 60 ms for time pressure, and (680 + 550) ÷ 2 − (600 + 610) ÷ 2 = 615 − 605 = 10 ms for the interaction, half of the 80 − 60 = 20 ms change in interference; n is our 28 people. That's 137,200, 100,800 and 2,800. With three or more levels there's no single difference: a main effect's sum of squares adds up each level average's squared distance from the grand mean, times the number of scores behind that average.

  3. 3. Measure How Much People Disagree about Each Effect

    Some people show a big interference effect and some a small one. That person-to-person inconsistency in an effect is its error term, the noise the effect has to stand out from. People who are simply faster or slower overall don't count here; they're set aside.

    SS_{\text{effect} \times S} = \sum (d_i - \bar{d})^2

    SS_{\text{congruency} \times S} = 27918

    SS_{\text{time} \times S} = 31990

    SS_{\text{interaction} \times S} = 12064

    SS_S = 559458

    d_i is one person's own effect, and \bar{d} the average one. For a main effect, it's the person's difference between the two levels; for the interaction, their own diagonal difference, half their difference of differences. Squaring and adding how far our 28 people's congruency effects sit from 70 ms gives 27,918; for time pressure, 31,990; for the interaction, 12,064. The stable differences in overall speed that were set aside add up to 559,458, far more than any effect.

  4. 4. Turn Each Sum into an Average

    Divide each sum by the pieces of information behind it. With two levels, each effect is a single difference, so it has 1 degree of freedom and its mean square is just its sum of squares. Each error term has one piece per person, minus one for the average effect.

    MS_{\text{effect} \times S} = \dfrac{SS_{\text{effect} \times S}}{(n - 1)} =\dfrac{12{,}064}{28 - 1}

    = 446.81

    df_{\text{effect} \times S} = 27

    MS_{\text{congruency} \times S} = 1034.00

    MS_{\text{time} \times S} = 1184.81

    n - 1 is 27 for every effect here, the error df. For the interaction, 12,064 ÷ 27 = 446.81; for congruency, 27,918 ÷ 27 = 1034.00; for time pressure, 31,990 ÷ 27 = 1184.81. With two levels per factor, sphericity holds automatically: there's only one difference per effect, so no correction is needed. In general, with a levels of one factor and b of the other, the effects have a - 1, b - 1 and (a - 1)(b - 1) df, and each error term has its effect's df times n - 1. A 3 × 2 design with 28 people tests the three-level factor and the interaction on F(2, 54); each of those can violate sphericity, so check and correct each one separately.

  5. 5. Compare Each Effect with How Inconsistently People Show It

    F divides each effect by its own error. If an effect weren't real, the two would estimate the same noise and F would hover around 1.

    F = \dfrac{MS_{\text{effect}}}{MS_{\text{effect} \times S}} =\dfrac{2800}{446.81}

    = 6.27

    F_{\text{congruency}} = 132.69

    F_{\text{time}} = 85.08

    MS_{\text{effect}} for the interaction is 2,800 and MS_{\text{effect} \times S} is 446.81. 2,800 ÷ 446.81 = 6.27. The main effects are far bigger: 137,200 ÷ 1034.00 = 132.69 for congruency and 100,800 ÷ 1184.81 = 85.08 for time pressure.

  6. 6. Ask How Surprising the Interaction Would Be If the Deadline Didn't Change Interference

    Ask how often chance alone would produce an F this large if the interference effect were really the same with and without a deadline.

    p = P(F_{df_1,\,df_2} \ge F) =P(F_{1,\,27} \ge 6.27)

    = .019

    F_{1,\,27} is an F distribution with 1 and 27 df. The chance it lands at or above 6.27 is .019, about 1 in 54. Both main effects have p < .001.

  7. 7. Measure the Effect against the Noise It Was Tested Against

    Partial η² gives each effect's share of itself plus its own error. It's what SPSS prints, but it ignores the big differences between people, so it runs large in repeated-measures designs.

    \eta^2_p = \dfrac{SS_{\text{effect}}}{SS_{\text{effect}} + SS_{\text{effect} \times S}} =\dfrac{2800}{2800 + 12{,}064}

    = .19

    \eta^2_{p,\,\text{congruency}} = .83

    \eta^2_{p,\,\text{time}} = .76

    SS_{\text{effect}} is the interaction's 2,800 and SS_{\text{effect} \times S} its 12,064. 2,800 ÷ 14,864 = .19. Congruency's is .83, and time pressure's .76.

  8. 8. Measure the Effect against Everything Except the Other Effects

    Generalized η² puts the differences between people back in the denominator, as a study with different people in each condition would have them, but leaves out the other two manipulated effects. That makes it more comparable across designs, and the one to hold against the benchmarks.

    \eta^2_G = \dfrac{SS_{\text{effect}}}{SS_{\text{effect}} + SS_S + \sum SS_{\text{error}}} =\dfrac{2800}{2800 + 559{,}458 + 27{,}918 + 31{,}990 + 12{,}064}

    = .004

    \eta^2_{G,\,\text{congruency}} = .178

    \eta^2_{G,\,\text{time}} = .138

    SS_S is the 559,458 of stable speed differences between people, and the sum of the three error terms is 71,972. 2,800 ÷ 634,230 = .004. The two main effects' 238,000 isn't in the denominator, which is why it's smaller than the table's total of 872,230. Congruency's is .178 and time pressure's .138.

  9. 9. Break the Interaction into Simple Effects

    A significant interaction says the interference effect differs between the time-pressure conditions. Simple effects show what it is in each one, and the difference between them, with its range of plausible values, is the interaction in milliseconds.

    (\bar{x}_{\text{IN}} - \bar{x}_{\text{CN}}) - (\bar{x}_{\text{ID}} - \bar{x}_{\text{CD}}) =(680 - 600) - (610 - 550)

    = 20 ms

    \text{Interference, no deadline} = 80 ms

    \text{Interference, deadline} = 60 ms

    t_{\text{no deadline}} = 9.55

    t_{\text{deadline}} = 10.06

    \text{95\% CI lower} = 3.61 ms

    \text{95\% CI upper} = 36.39 ms

    With no deadline, incongruent words cost 680 − 600 = 80 ms; with a deadline, 610 − 550 = 60 ms. 80 − 60 = 20 ms. Each person's difference of differences has an SD of 42.28 ms, so its standard error is 42.28 ÷ √28 = 7.99 ms. With 27 df, 95% of a t distribution lies within ±2.052, so the interval is 20 ± 2.052 × 7.99, from 3.61 to 36.39 ms. The simple effects are paired t-tests built the same way: t(27) = 80 ÷ 8.379 = 9.55 and 60 ÷ 5.963 = 10.06, both p < .001. Interference under the deadline is smaller but more consistent across people, so its t is larger.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)
  1. SPSS wants one column per condition. Our file is long, so first use Data → Restructure → Restructure selected cases into variables, with participant as the Identifier Variable and condition as the Index Variable, and leave congruency and time_pressure out. You get one row per person with columns congruent_none, incongruent_none, congruent_deadline and incongruent_deadline.
  2. Analyze → General Linear Model → Repeated Measures.
  3. Define the factors in the order the columns change, slowest first: time_pressure with 2 levels, then congruency with 2 levels. Click Define.
  4. Move the four columns into the slots in the order shown (none-congruent, none-incongruent, deadline-congruent, deadline-incongruent).
  5. Options: check Descriptive statistics and Estimates of effect size. EM Means: add time_pressure*congruency and check Compare simple main effects with a Bonferroni adjustment. Plots: put time_pressure on the horizontal axis and congruency as separate lines. Click OK.

Or paste this syntax:

GLM congruent_none incongruent_none congruent_deadline incongruent_deadline
  /WSFACTOR=time_pressure 2 Polynomial congruency 2 Polynomial
  /METHOD=SSTYPE(3)
  /PLOT=PROFILE(time_pressure*congruency)
  /EMMEANS=TABLES(time_pressure*congruency) COMPARE(congruency) ADJ(BONFERRONI)
  /PRINT=DESCRIPTIVE ETASQ
  /CRITERIA=ALPHA(.05)
  /WSDESIGN=time_pressure congruency time_pressure*congruency.

Reading the Output

Read the tables in this order: the cell Means in Descriptive Statistics (and the profile plot), then Partial Eta Squared, then F and Sig. for each effect in Tests of Within-Subjects Effects, starting with the interaction.

Tests of Within-Subjects Effects (Sphericity Assumed rows)

Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared
time_pressure 100800.000 1 100800.000 85.077 <.001 .759
Error(time_pressure) 31990.000 27 1184.815
congruency 137200.000 1 137200.000 132.689 <.001 .831
Error(congruency) 27918.000 27 1034.000
time_pressure * congruency 2800.000 1 2800.000 6.267 .019 .188
Error(time_pressure*congruency) 12064.000 27 446.815

SPSS also prints Greenhouse-Geisser, Huynh-Feldt and Lower-bound rows for each effect. With two levels per factor they're identical to these, and Mauchly's test has no df to test with, so it prints W = 1.000 with 0 df and no p-value. SPSS doesn't print generalized η². Work it out from the sums of squares: the effect's SS divided by itself plus the three error SS and the Error SS in Tests of Between-Subjects Effects (559,458 here).

Pairwise Comparisons (congruency within each time_pressure level)

time_pressure (I) congruency (J) congruency Mean Difference (I−J) Std. Error Sig.
1 (none) 2 1 80.000* 8.379 <.001
2 (deadline) 2 1 60.000* 5.963 <.001

These are the simple effects: incongruent (2) minus congruent (1) at each time-pressure level. SPSS lists each pair in both directions; one is shown.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R 
stroop <- read.csv("data.csv")
stroop$participant <- factor(stroop$participant)
stroop$congruency <- factor(stroop$congruency, levels = c("congruent", "incongruent"))
stroop$time_pressure <- factor(stroop$time_pressure, levels = c("none", "deadline"))

fit <- aov(rt_ms ~ congruency * time_pressure +
             Error(participant / (congruency * time_pressure)), data = stroop)
summary(fit)

# Generalized eta squared (ges) and sphericity corrections, with the afex package
library(afex)
aov_ez("participant", "rt_ms", stroop, within = c("congruency", "time_pressure"))

Error: participant
          Df Sum Sq Mean Sq F value Pr(>F)
Residuals 27 559458   20721

Error: participant:congruency
           Df Sum Sq Mean Sq F value   Pr(>F)
congruency  1 137200  137200   132.7 6.29e-12 ***
Residuals  27  27918    1034

Error: participant:time_pressure
              Df Sum Sq Mean Sq F value   Pr(>F)
time_pressure  1 100800  100800   85.08 7.81e-10 ***
Residuals     27  31990    1185

Error: participant:congruency:time_pressure
                         Df Sum Sq Mean Sq F value Pr(>F)
congruency:time_pressure  1   2800  2800.0   6.267 0.0187 *
Residuals                27  12064   446.8

Anova Table (Type 3 tests)

Response: rt_ms
                    Effect    df     MSE          F  ges p.value
1               congruency 1, 27 1034.00 132.69 *** .178   <.001
2            time_pressure 1, 27 1184.81  85.08 *** .138   <.001
3 congruency:time_pressure 1, 27  446.81     6.27 * .004    .019

Each Error: stratum is one error term: the first holds the stable differences between people, and each of the others tests the effect above its Residuals row. afex's ges is generalized η².

Output from a real run: R 4.6.1 (afex 1.5.1), run 2026-09-29.

Python 
import pandas as pd
import pingouin as pg

stroop = pd.read_csv("data.csv")
aov = pg.rm_anova(data=stroop, dv="rt_ms", within=["congruency", "time_pressure"],
                  subject="participant", effsize="np2")
print(aov[["Source", "ddof1", "ddof2", "F", "p-unc", "np2"]].round(4))

# Generalized eta squared instead of partial
aov_g = pg.rm_anova(data=stroop, dv="rt_ms", within=["congruency", "time_pressure"],
                    subject="participant", effsize="ng2")
print(aov_g[["Source", "ng2"]].round(4))

                       Source  ddof1  ddof2         F   p-unc     np2
0                  congruency      1     27  132.6886  0.0000  0.8309
1               time_pressure      1     27   85.0766  0.0000  0.7591
2  congruency * time_pressure      1     27    6.2666  0.0187  0.1884
                       Source     ng2
0                  congruency  0.1785
1               time_pressure  0.1377
2  congruency * time_pressure  0.0044

np2 is partial η² and ng2 generalized η². The main effects' p-values are below .0001, so they round to 0. statsmodels.stats.anova.AnovaRM gives the same Fs and p-values without effect sizes. For the simple effects, pivot to wide and run scipy.stats.ttest_rel on incongruent versus congruent within each time-pressure level: t = 9.55 and 10.06.

Output from a real run: Python 3.9.6 (pingouin 0.5.5, pandas 2.3.3), run 2026-10-10.

Impact

Effect Size

A repeated-measures design has two reasonable versions of η², and they can tell very different stories. Partial eta squared compares an effect with its own error term only. Generalized eta squared also counts the stable differences between people, as a study with different people in each condition would have to (Olejnik & Algina, 2003; Bakeman, 2005). That makes generalized η² the one to compare across designs and to hold against benchmarks.

\eta^2_G = \dfrac{SS_{\text{effect}}}{SS_{\text{effect}} + SS_S + \sum SS_{\text{error}}} = \dfrac{2800}{634{,}230} = .004 \qquad \eta^2_p = \dfrac{2800}{2800 + 12{,}064} = .19

People differed in overall speed by far more than any condition moved them: those differences alone are 559,458 of the 634,230 in the denominator. Against all of that, the interaction is a sliver, η²G = .004. Against only itself plus people's inconsistency in it, it's .19. Both numbers are correct; they answer different questions.

The main effects look very different on the same scale: η²G = .178 for congruency, large by convention, and .138 for time pressure, medium and just under the large cutoff of .14. Stroop interference and deadline speed-ups are big, reliable effects; the interaction is a small change in one of them.

For readers, the clearest size is in milliseconds: incongruent words cost 80 ms without a deadline and 60 ms with one, 20 ms less, 95% CI [3.61, 36.39]. That interval is the study's interval estimate for the interaction; an η² from a sample this small has a very wide interval of its own (partial η² = .19, 95% CI [.00, .43], from the noncentral F). Benchmarks are conventions. Whether shaving 20 ms off interference says something important about how time pressure changes reading is a question about attention, not statistics.

Benchmarks Generalized Eta Squared at a Glance
Below Small
η²G < .01
Small
.01 ≤ η²G < .06
Medium
.06 ≤ η²G < .14
Large
η²G ≥ .14

Full benchmarks and caveats ↑

Statistical Significance

The ANOVA gives three p-values, one for each main effect and one for the interaction. Each answers one narrow question: if that effect were zero in the population, how often would its F come out at least this large? The three effects are separate questions about the same four means, so a significant main effect says nothing about the interaction, or the other way round. Below the conventional α = .05, the result is called statistically significant.

If the deadline didn't change the interference effect at all, an interaction F of 6.27 or more would turn up about 1.9% of the time, roughly 1 in 54, under the test's assumptions. F measures the 20-ms change in interference relative to how much that change varied from person to person. That's below .05, so the interaction is significant. Both main effects have p < .001.

p is not the probability that the null hypothesis is true, and it says nothing about how large an effect is. Here the interaction is significant yet tiny next to the differences between people. Three tests also means three chances of a false positive. If the tests were independent and all three nulls were true, the chance of at least one would be about 14% (Cramer et al., 2016). They come from the same people, so the exact rate depends on how the three contrasts are related; in a simulation using this study's contrast variances and correlations (up to .36), it came out at about 14% too. Decide in advance which effect is your question, as the interaction is here.

Benchmarks p at a Glance
Very Strong
p < .001
Strong
.001 ≤ p < .01
Significant
.01 ≤ p < .05
Weak
.05 ≤ p < .10
Little
p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. A real effect doesn't guarantee a significant result: with too few people, it can easily come out non-significant. Power depends on how big the true effect is, how many people you have, α, and in repeated-measures designs how consistently people show the effect, which depends on how strongly their scores in different conditions are correlated.

With two levels per factor, power has an exact shortcut. Each effect boils down to one score per person, and its F is the square of a paired t on that score, so its power is a paired t-test's. The calculator below does that calculation. Its effect size is Cohen's dz for the effect you're planning: the average of that per-person score divided by its standard deviation across people. For the interaction, the score is each person's difference of differences (their interference effect without the deadline minus with it), so dz is the average difference of differences divided by its SD. (The Napkin Math halves each person's score, which halves the SD too and leaves dz the same.) For a main effect, it's each person's difference between the two levels, averaged over the other factor. "Number of pairs" is the number of people. Interactions usually need more people than main effects, because they're smaller and a difference of differences is noisier.

Our difference of differences averaged 20 ms with a standard deviation of 42.28 ms across people, dz = 20 ÷ 42.28 = 0.47. If that were the true size, 28 people would detect it about 67% of the time, and 40 people about 83%.

Don't plan the next study from our observed effect alone: it's a noisy estimate from 28 people, and a significant result tends to overestimate. The shortcut doesn't hold with three or more levels of a factor. For those, use software built for repeated measures, such as G*Power, or a simulation; both need a guess at the correlations between conditions.

Researchers usually aim for at least 80% power. Plan it before collecting data, using the smallest effect that would matter. Try it:

Other Info

Follow-Up and Post Hoc Tests

Planned Contrasts

  • Planned Contrasts: Use when theory predicted a specific comparison before data collection. In a 2 × 2 design the interaction already is the one contrast worth planning, the difference of differences. With three or more levels, test the predicted comparisons directly, correcting with Holm if there are several.

Post Hoc Tests

  • Simple Effects Analysis: Use after a significant interaction, to see each factor's effect at each level of the other. In the Stroop deadline study the interference effect was 80 ms with no deadline, t(27) = 9.55, and 60 ms with a deadline, t(27) = 10.06, both p < .001. Each simple effect uses only its own two cells, so it gets its own error term.
  • Holm Correction: Use across a set of simple effects or pairwise comparisons. Here, two simple effects are both p < .001, so Holm and Bonferroni change nothing.
  • Bonferroni Correction: Works for any set of follow-up tests, but it's conservative; Holm gives the same protection with more power.

References

  • Fisher, R. A. (1925). Statistical methods for research workers. Oliver and Boyd.
  • Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434–447. https://doi.org/10.1037/1082-989X.8.4.434
    “Because research-design features can have a large effect on the estimated proportion of explained variance, the use of partial eta or omega squared can be misleading.” (Abstract)
    “generalized eta and omega squared statistics, which provide estimates of effect size that are comparable across a variety of research designs” (Abstract)
  • Bakeman, R. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37(3), 379–384. https://doi.org/10.3758/BF03192707
    “it provides comparability across between-subjects and within-subjects designs” (Abstract)
  • Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863
    “Although η2G can be compared against the benchmarks provided by Cohen (1988), this should only be done as a last resort” (Eta-squared in between and within-subjects comparisons)
    “using these benchmarks when interpreting the η2p effect size in designs that include covariates or repeated measures is not consistent with the considerations upon which the benchmarks were based” (Eta-squared in between and within-subjects comparisons)
  • Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24(2), 95–112. https://doi.org/10.1007/BF02289823
  • Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., Waldorp, L. J., & Wagenmakers, E.-J. (2016). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. Psychonomic Bulletin & Review, 23(2), 640–647. https://doi.org/10.3758/s13423-015-0913-5
  • Wagenmakers, E.-J., Krypotos, A.-M., Criss, A. H., & Iverson, G. (2012). On the interpretation of removable interactions: A survey of the field 33 years after Loftus. Memory & Cognition, 40(2), 145–160. https://doi.org/10.3758/s13421-011-0158-0
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
  • Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018). Designing experiments and analyzing data: A model comparison perspective (3rd ed.). Routledge. https://doi.org/10.4324/9781315642956
  • Rosnow, R. L., & Rosenthal, R. (1989). Definition and interpretation of interaction effects. Psychological Bulletin, 105(1), 143–146. https://doi.org/10.1037/0033-2909.105.1.143
  • Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328–331. https://doi.org/10.1198/000313006X152649
    “In making a comparison between two treatments, one should look at the statistical significance of the difference rather than the difference between their significance levels.” (p. 329)
  • Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14(9), 1105–1107. https://doi.org/10.1038/nn.2886
    “In theory, a comparison of two experimental effects requires a statistical test on their difference.” (Abstract, p. 1105)
  • Cohen, J. (1973). Eta-squared and partial eta-squared in fixed factor ANOVA designs. Educational and Psychological Measurement, 33(1), 107–112. https://doi.org/10.1177/001316447303300111
  • Pierce, C. A., Block, R. A., & Aguinis, H. (2004). Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement, 64(6), 916–924. https://doi.org/10.1177/0013164404264848
  • Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
    “we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)
  • Blanca, M. J., Arnau, J., García-Castro, F. J., Alarcón, R., & Bono, R. (2023). Repeated measures ANOVA and adjusted F-tests when sphericity is violated: Which procedure is best? Frontiers in Psychology, 14, 1192453. https://doi.org/10.3389/fpsyg.2023.1192453
    “the F-statistic becomes more liberal as sphericity violation increases, whereas both F-HF and F-GG control Type I error” (Abstract)

Spot an error? Something still confusing? Tell us at support@statsforpsychology.com.