One-Way Between-Subjects ANOVA

The Basics

When to Use It

Use it when you have three or more separate groups of different people, one continuous outcome, and you want to know whether the groups' averages differ. For example:

Does how students study (rereading, concept mapping or retrieval practice) change their test scores a week later?
Do depression scores after 12 weeks differ between CBT, medication and a waitlist?
Does reaction time differ after caffeine, a nap or neither, each tested on a different group?

Why not run a t-test on every pair? Three uncorrected tests at α = .05 give three chances at a false positive, so the chance of at least one (the familywise error rate) climbs above 5%. If the tests were independent it would be about 14% (1 − .95³); pairwise tests share groups, so they're correlated. In a simulation with normal scores, equal variances, no real differences and 25 students per group, it came out at about 12%. Tukey's HSD or a Holm correction keeps it at 5%. ANOVA, short for analysis of variance, asks the question once, by comparing how much the group averages vary with how much individuals vary within their groups.

Vocabulary You Need

Definition Between-Groups Variance (

MS_{\text{between}}

)

How much the group means vary around the grand mean, as a variance (the between-groups sum of squares divided by k − 1). If the groups don't really differ, it estimates the same thing as the within-groups variance; real differences make it bigger. Glossary →

Definition Within-Groups Variance (

MS_{\text{within}}

)

How much individuals vary around their own group's mean, pooled across all the groups (the within-groups sum of squares divided by N − k). It is the background noise that differences between groups are judged against. Glossary →

Definition F Statistic (

F

)

The variance due to an effect divided by the error variance it is tested against: in a between-subjects ANOVA, the between-groups variance over the within-groups variance; in a repeated-measures ANOVA, the conditions' variance over the inconsistency in how people change across them. It is near 1 when there's no real effect and grows as the effect stands out from the error. It is compared with an F distribution, which has two degrees of freedom, to get a p-value. Glossary →

Definition Omnibus Test

A single test of whether any of several groups differ, without saying which. A significant omnibus result is followed by comparisons of specific groups; a non-significant one doesn't prove the groups are equal. Glossary →

Assumptions and Inference

1. Independent Students (for the p-Value and Intervals) The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: A lot. If students studied together or were tested in clusters, scores are more alike than the F-test expects, and p comes out too small. Random assignment doesn't prevent this: students assigned one by one can still share a classroom, a team or a session.

What to Do

Alternative Test Multilevel Regression (HLM). A multilevel model, which accounts for the clustering.
Design Change Assign and measure each person separately, or analyze cluster averages.

2. Normality Within Groups (for the p-Value and Intervals) The scores within each group (the residuals) are roughly normal; the 75 scores pooled together needn't be. F is exact under normal residuals and equal variances. Moderate departures are usually tolerated, especially with bigger, equal-sized groups, but strong skew, heavy tails or extreme outliers can still matter.

What It Means: The F-test treats each student's distance from their own group's mean as normal error. Its false-positive rate holds up well when that error isn't normal, but outliers pull group means and inflate the within-groups variance, and heavy tails cost power. If the scores are skewed but the population means are still the question, consider Welch's ANOVA or another method for comparing means, with interval methods suited to the distributions and group sizes; bootstrap intervals can help, but they don't cure severe skew or outliers in small groups. Trimmed means and ranks answer a different question.
Applies To: Residuals
How to Check: Here a residual is each student's score minus their group's average. Plot those in a histogram or Q–Q plot, not the raw scores pooled across groups, which mix three different centers. Look for strong skew or extreme outliers. Don't rely on a normality test: with small samples it misses real problems, and with large ones it flags trivial ones.
How Much It Matters: Little, for the false-positive rate: in simulations the F-test held α across a wide range of non-normal distributions (Blanca et al., 2017). Extreme outliers matter more, because they pull group means and inflate the within-groups variance. Power is another matter: with heavy-tailed data, a trimmed-means ANOVA or the Kruskal-Wallis test can detect differences the F-test misses.

What to Do

Alternative Test Kruskal-Wallis Test. It asks whether some groups tend to score higher than others, using ranks rather than means, so the question changes: use it when that's your question, not just because the scores are skewed.
Robust Estimation Compare trimmed means with a robust ANOVA (R: WRS2::t1way), which resists outliers. It compares trimmed means, not ordinary means.
Transformation Log-transform right-skewed outcomes such as reaction times. Results are then on the log scale.

3. Equal Variances (for the Classic F-Test) The classic F-test assumes every group's population is equally spread out. Welch's ANOVA doesn't, and is generally preferable when spreads or group sizes differ a lot.

What It Means: The outcome varies about as much within each group's population. Tests that pool the groups' variances into one estimate, like Student's t-test and the classic ANOVA, rely on it.
Applies To: Variances
How to Check: Compare the groups' standard deviations and look at side-by-side box plots. Don't use Levene's test as a gatekeeper: it misses real differences in small samples and flags trivial ones in large samples.
How Much It Matters: Less with equal group sizes, like our 25 per group, where the largest variance is only 1.21 times the smallest. When groups differ in both size and spread, the F-test's error rate drifts: too many false positives if the smaller groups vary more, too few (and less power) if the larger groups do. That's why Welch's ANOVA (Welch, 1951) is often recommended by default (Delacre et al., 2019).

What to Do

Correction Welch's ANOVA doesn't assume equal variances and loses little when they are equal. For our data it gives F(2, 47.93) = 3.58, p = .035.
Correction Use Games-Howell instead of Tukey's HSD for the pairwise comparisons.

Limitations

This Is Not Your Test If…

You have only two groups Independent-Samples t-Test →
You measured the same people in every condition One-Way Repeated-Measures ANOVA →
You have two or more grouping variables Factorial Between-Subjects ANOVA →
You want to adjust for a covariate, like a pretest ANCOVA →
Your outcome is ordered ratings, or your question is which groups tend to score higher rather than whether the means differ Kruskal-Wallis Test →
Your outcome isn't continuous (yes/no, categories) Categorical Data and Proportions →

What It Doesn't Tell You

Which groups. The omnibus test only says the three averages aren't all equal. Which ones differ comes from the follow-up comparisons.
Why. Random assignment lets us say the technique made a difference, but not why retrieval practice helped: the retrieving itself, more effort, or more time on task.
Whether it matters. Significance doesn't make 8 points on a test important for grades or long-term learning.
That there's no difference, for concept mapping. It didn't differ reliably from either technique, but each of those differences has an interval about 14 points wide, from −3.11 to 11.11 points. That's inconclusive, not evidence the techniques are equal.

Common Mistakes

Don'tRun an ordinary t-test for every pair of groups.

InsteadUse Tukey's HSD or a Holm correction, so the chance of any false positive stays at or below α.
Don'tRead a significant F as telling you which groups differ.

InsteadFollow it with post hoc tests or planned contrasts, and report those too.
Don'tCall a comparison “planned” after seeing which means look different.

InsteadPlan contrasts before collecting data. Comparisons chosen afterward are post hoc and need a correction.
Don'tConclude two groups are equal because their post hoc comparison wasn't significant.

InsteadReport the interval. A wide interval that includes zero means the data can't tell.
Don'tLet Levene's test choose between the classic F-test and Welch's ANOVA.

InsteadDecide in advance, from the design. Welch's ANOVA is a reasonable default because it doesn't assume equal variances (Delacre et al., 2019). The classic F suits designs where equal variances are defensible; equal group sizes make it more robust to moderate differences in spread, but don't remove the assumption.
Don'tRead a p-value as the chance that the null hypothesis is true.

InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance and Effect Size Guidelines

Benchmarks Significance (p Value)

p < .001
Very strong evidence against the null hypothesis.
.001 ≤ p < .01
Strong evidence against the null hypothesis.
.01 ≤ p < .05
Statistically significant at α = .05. Moderate evidence against the null hypothesis.
.05 ≤ p < .10
Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.
p ≥ .10
Not significant. Little evidence against the null, which is not the same as evidence of no effect.

These evidence labels are one common convention for describing p in graded language rather than as significant or not; the boundaries are rough guides, not rules, and other authors draw them differently. Muff et al. (2022) argue for this kind of language of evidence over a single cutoff. α is chosen before the analysis, and p never says how big or important an effect is.

More about statistical significance ↓

Benchmarks Effect Size (Eta Squared)

η² < .01 Below Small
Smaller than the conventional small benchmark. Whether an effect this size matters depends on the setting.
.01 ≤ η² < .06 Small
A sliver of the variation lines up with the groups. Hard to see without measuring it.
.06 ≤ η² < .14 Medium
A noticeable share of the variation lines up with the groups.
η² ≥ .14 Large
A large share of the variation lines up with the groups.

Benchmarks are conventions (Cohen, 1988), developed for simple comparisons between groups. They are not measures of practical importance.

More about effect size ↓

How It's Reported

Equation Form

Example

η² = .09, 95% CI [.00, .22], F(2, 72) = 3.62, p = .032; follow-up: HSD = 7.11

Full Text (APA Style)

Example

Students who used retrieval practice scored highest a week later (M = 76.00, SD = 10.99), followed by concept mapping (M = 72.00, SD = 10.50) and rereading (M = 68.00, SD = 10.00). Study technique was associated with 9% of the variation in scores, η² = .09, 95% CI [.00, .22]. A one-way ANOVA showed the differences were statistically significant, F(2, 72) = 3.62, p = .032. Because group sizes were equal and planned in advance, we report the classic F; Welch's ANOVA, which doesn't assume equal variances, gave the same conclusion, F(2, 47.93) = 3.58, p = .035. Tukey's HSD tests showed that retrieval practice outscored rereading by 8.00 points, 95% CI [0.89, 15.11], p = .024; concept mapping did not differ reliably from either (both p = .375).

Give every group's M and SD, so readers can see the pattern the F summarizes.
Report both degrees of freedom: between groups (k − 1) and within groups (N − k).
Follow a significant F with the comparisons that say which groups differ, and name the procedure.
Italicize Latin symbols (M, SD, F, p) but not Greek ones (η²), and report exact p to three decimals.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how surprising.

η² = .09, 95% CI [.00, .22]	Study technique accounts for about 9% of the variation in test scores. That's a medium effect by the benchmarks above, but the interval runs from essentially zero to a large effect, so 25 students per group pin its size down only loosely. The interval reaching zero doesn't contradict p = .032 (see Effect Size). Whether it matters is a question about learning, not statistics. Benchmarks ↑
F(2, 72) = 3.62	The group averages vary 3.62 times as much as we'd expect from how much students vary within a group. The 2 and 72 are the degrees of freedom: k − 1 and N − k.
p = .032	If the three techniques worked equally well, averages at least this far apart would turn up about 3% of the time. That's below .05, so the result is statistically significant. Benchmarks ↑
HSD = 7.11	Two techniques differ significantly if their averages are more than 7.11 points apart. Only retrieval practice versus rereading (8 points) clears that bar.

Calculations

The Running Example

Example

The study-technique study: 75 undergraduates, 25 per technique (rereading, concept mapping, retrieval practice), each with one test score out of 100.

Null Hypothesis:: $\mu_{\text{reread}} = \mu_{\text{map}} = \mu_{\text{retrieve}}$
Alternative:: $\text{not all three } \mu \text{ are equal}$

Each $\mu$ (“mu”) is the population mean test score for students who study that way, which our group averages of 68, 72 and 76 estimate. The alternative doesn't say which means differ or in what direction.

What Your Data Should Look Like

One row per person, with one column naming their group and one numeric column for their score. Each person appears once, in exactly one group.

Example

participant	group	score
1	rereading	54
2	retrieval	70
3	rereading	69
4	mapping	78
5	rereading	76
6	rereading	72
… 69 more rows

	n	M	SD
Rereading	25	68.00	10.00
Concept Mapping	25	72.00	10.50
Retrieval Practice	25	76.00	10.99
All	75	72.00	10.87

Simulated data.

Download all 75 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the study-technique study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Find Each Group's Average and the Overall Average

The idea behind Fisher's (1925) analysis of variance: every score's distance from the overall average has two parts, its group's distance from the overall average and its own distance from its group's average. Square those distances and add them up over everyone, and the two parts add up exactly to the total. The steps below split the variation that way, then compare the between-groups part with the within-groups part. It all starts from the overall average.

$\bar{x}_{\text{grand}} = \dfrac{\sum x_i}{N}$ = $\dfrac{5400}{75}$

= 72.00 points

$\sum x_i$ is all 75 scores added up, 5400, and $N$ is our 75 students. 5400 ÷ 75 = 72.00 points. With equal groups that's also the average of the three group means, (68 + 72 + 76) ÷ 3.
2. Measure How Far the Group Averages Spread from the Overall Average

If the techniques made no difference, the three averages would sit close to the overall one. This adds up how far they actually strayed, once for every student in the group.

$SS_{\text{between}} = \sum n_j (\bar{x}_j - \bar{x}_{\text{grand}})^2$ = $25(68 - 72)^2 + 25(72 - 72)^2 + 25(76 - 72)^2$

= 800

$n_j$ is the 25 students in each group and $\bar{x}_j$ is that group's average. Rereading is 4 below 72 and retrieval 4 above, so 25 × 16 + 0 + 25 × 16 = 800. Squaring makes every distance count as positive, so groups above and below the overall average don't cancel out, and it weights big gaps more than small ones.
3. Measure How Much People Vary within Their Own Group

Students using the same technique still score differently. That spread is the background noise any difference between groups has to stand out from.

$SS_{\text{within}} = \sum (n_j - 1)\, s_j^2$ = $24(10.00^2) + 24(10.50^2) + 24(10.992^2)$

= 7946

$s_j$ is each group's standard deviation and $n_j - 1 = 24$ . 2400 + 2646 + 2900 = 7946. A group's SD² × 24 gives back the sum of its students' squared distances from their group's average, so this is the same as adding up all 75 students' squared distances. (Retrieval's SD shows as 10.99 elsewhere; rounded that far, the last term comes to 2899.)
4. Turn the Between-Group Spread into an Average

Sums grow with every piece we add, so divide by the pieces of information behind them. An average of squared distances like this is a variance, and its square root is a standard deviation. Three group averages around a fixed overall average leave two free to vary.

$MS_{\text{between}} = \dfrac{SS_{\text{between}}}{k - 1}$ = $\dfrac{800}{3 - 1}$

= 400.00

$k$ is our 3 techniques, so $k - 1 = 2$ . 800 ÷ 2 = 400.
5. Turn the Within-Group Spread into an Average

The same move for the noise. Each group loses one piece of information to estimating its own average.

$MS_{\text{within}} = \dfrac{SS_{\text{within}}}{N - k}$ = $\dfrac{7946}{75 - 3}$

= 110.36

$N$ is our 75 students and $k$ the 3 groups, so $N - k = 72$ . 7946 ÷ 72 = 110.36, the average squared distance of a student from their group's average (its square root, 10.51 points, is the pooled SD). The degrees of freedom split the same way as the sums of squares, 74 in all (N − 1) = 2 between + 72 within.
6. Compare the Spread between Groups with the Spread within Them

If the techniques didn't matter, both averages would estimate the same noise and their ratio would hover around 1. F says how many times bigger the between-group spread is.

$F = \dfrac{MS_{\text{between}}}{MS_{\text{within}}}$ = $\dfrac{400}{110.36}$

= 3.62

$MS_{\text{between}}$ is our 400 and $MS_{\text{within}}$ is 110.36. 400 ÷ 110.36 = 3.62, so the group averages vary a little over three and a half times as much as noise alone would suggest.
7. Measure How Much of the Variation the Groups Explain

F grows with sample size; η² doesn't. It asks what share of all the variation in scores lines up with which technique students used.

$\eta^2 = \dfrac{SS_{\text{between}}}{SS_{\text{between}} + SS_{\text{within}}}$ = $\dfrac{800}{800 + 7946}$

= .09

$95\%\ \text{CI}$ = [.00, .22]

$SS_{\text{between}}$ is our 800, and the total, every student's squared distance from the overall average of 72, is 800 + 7946 = 8746. 800 ÷ 8746 = .09, so technique accounts for about 9% of the differences in scores.
8. Ask How Surprising the Result Would Be If the Techniques Worked Equally Well

Finally, ask how often chance alone would spread three group averages at least this far apart if all three techniques were equally good.

$p = P(F_{df_1,\,df_2} \ge F)$ = $P(F_{2,\,72} \ge 3.62)$

= .032

If the techniques worked equally well and we reran this study many times, the F values would average about 1, and 86% of them would fall below 2. That pattern is the F distribution with 2 and 72 df, $F_{2,\,72}$ , drawn under Vocabulary. Only 3.2% of those F values reach 3.62 or more, so p = .032, about 1 in 32. Only large F values count against the null, so there's one tail, but it catches differences in any direction.
9. Find Which Pairs of Groups Differ Reliably

A significant F says the averages aren't all equal, not which ones differ. Tukey's HSD sets one cutoff for every pair while keeping the overall chance of a false positive no higher than .05.

$HSD = q^{*}_{k,\,N-k} \sqrt{\dfrac{MS_{\text{within}}}{n}}$ = $3.384 \times \sqrt{\dfrac{110.36}{25}}$

= 7.11 points

$q^{*}$ = 3.384 is the studentized range cutoff for 3 groups and 72 df, and $\sqrt{110.36 / 25}$ = 2.101 points. 3.384 × 2.101 = 7.11 points. Retrieval practice beat rereading by 8, which clears it; concept mapping is 4 from each, which doesn't.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

SPSS's One-Way ANOVA needs a numeric factor, so first recode group: Transform → Recode into Different Variables, into a new variable technique with 1 = rereading, 2 = mapping, 3 = retrieval, and give it those value labels.
Analyze → Compare Means and Proportions → One-Way ANOVA.
Move score into Dependent List and technique into Factor. Check Estimate effect size for overall tests.
Post Hoc: check Tukey (and Games-Howell if the groups' spreads differ).
Options: check Descriptive and Welch, and click OK.
To check normality, save the residuals: Analyze → General Linear Model → Univariate, with score as Dependent Variable and technique as Fixed Factor; under Save, check Unstandardized residuals. Then Analyze → Descriptive Statistics → Explore on the new RES_1, with Normality plots checked under Plots.

Or paste this syntax:

RECODE group ('rereading'=1) ('mapping'=2) ('retrieval'=3) INTO technique.
VALUE LABELS technique 1 'rereading' 2 'mapping' 3 'retrieval'.
EXECUTE.

ONEWAY score BY technique
  /ES=OVERALL
  /STATISTICS DESCRIPTIVES WELCH
  /MISSING ANALYSIS
  /CRITERIA=CILEVEL(0.95)
  /POSTHOC=TUKEY GH ALPHA(0.05).

* Check the assumptions: save the residuals, then a Q-Q plot of them.
UNIANOVA score BY technique
  /SAVE=RESID.
EXAMINE VARIABLES=RES_1
  /PLOT NPPLOT.

Reading the Output

Read the tables in this order: the group Means in Descriptives, then Eta-squared, then F and Sig. in the ANOVA table, then Multiple Comparisons to see which pairs differ.

ANOVA (score)

	Sum of Squares	df	Mean Square	F	Sig.
Between Groups	800.000	2	400.000	3.624	.032
Within Groups	7946.000	72	110.361
Total	8746.000	74

ANOVA Effect Sizes (point estimates)

	Point Estimate
Eta-squared	.091
Omega-squared Fixed-effect	.065

SPSS also prints epsilon-squared, a random-effect omega-squared and 95% confidence intervals. For a design like this one, where the researcher chose the groups, read eta-squared or the fixed-effect omega-squared.

Robust Tests of Equality of Means (score)

	Statistic	df1	df2	Sig.
Welch	3.584	2	47.928	.035

Multiple Comparisons (Tukey HSD)

(I) technique	(J) technique	Mean Difference (I−J)	Std. Error	Sig.	95% CI Lower	Upper
retrieval	rereading	8.000*	2.971	.024	.89	15.11
retrieval	mapping	4.000	2.971	.375	-3.11	11.11
mapping	rereading	4.000	2.971	.375	-3.11	11.11

SPSS lists every pair twice, once in each direction; three rows are shown here. The asterisk marks differences significant at .05. The Games-Howell rows below them give p = .026, .394 and .360 for the same three pairs.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

study <- read.csv("data.csv")
study$group <- factor(study$group, levels = c("rereading", "mapping", "retrieval"))

fit <- aov(score ~ group, data = study)
summary(fit)

# Eta squared: the model's R-squared
summary.lm(fit)$r.squared

# Which pairs differ
round(TukeyHSD(fit)$group, 3)

# Welch's ANOVA, which doesn't assume equal variances
oneway.test(score ~ group, data = study)

# Check the assumptions: each group's SD, and a Q-Q plot of the residuals
tapply(study$score, study$group, sd)
qqnorm(residuals(fit)); qqline(residuals(fit))

            Df Sum Sq Mean Sq F value Pr(>F)
group        2    800   400.0   3.624 0.0316 *
Residuals   72   7946   110.4
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

[1] 0.09147039

                    diff    lwr    upr p adj
mapping-rereading      4 -3.111 11.111 0.375
retrieval-rereading    8  0.889 15.111 0.024
retrieval-mapping      4 -3.111 11.111 0.375

	One-way analysis of means (not assuming equal variances)

data:  score and group
F = 3.5836, num df = 2.000, denom df = 47.928, p-value = 0.03545

rereading   mapping retrieval 
 10.00000  10.50000  10.99242

R calls the within-groups row “Residuals.” Setting the factor levels puts rereading first, so each difference reads as the later technique minus the earlier one. The SDs are close, and the Q–Q plot (drawn in a plot window, not printed) shows the residuals near the line.

Output from a real run: R 4.6.1, run 2026-10-10.

Python

import pandas as pd
from scipy import stats

study = pd.read_csv("data.csv")
groups = [study.loc[study["group"] == g, "score"] for g in ["rereading", "mapping", "retrieval"]]

result = stats.f_oneway(*groups)
print(result.statistic, result.pvalue)

print(stats.tukey_hsd(*groups))

3.6244651396929273 0.03163786419943371
Tukey's HSD Pairwise Group Comparisons (95.0% Confidence Interval)
Comparison  Statistic  p-value  Lower CI  Upper CI
 (0 - 1)     -4.000     0.375   -11.111     3.111
 (0 - 2)     -8.000     0.024   -15.111    -0.889
 (1 - 0)      4.000     0.375    -3.111    11.111
 (1 - 2)     -4.000     0.375   -11.111     3.111
 (2 - 0)      8.000     0.024     0.889    15.111
 (2 - 1)      4.000     0.375    -3.111    11.111

The groups are numbered in the order you passed them: 0 is rereading, 1 mapping, 2 retrieval. So (2 − 0) is retrieval practice minus rereading, 8 points. SciPy doesn't print η²; compute it as the between-groups sum of squares over the total, or use pingouin.anova(data=study, dv="score", between="group", detailed=True), which reports it as np2 (0.0915), and pingouin.welch_anova for Welch's ANOVA.

Output from a real run: Python 3.9.6 (scipy 1.12.0, pandas 2.3.3), run 2026-10-03.

Impact

Effect Size

Eta squared (η²) is the share of the variation in the observed scores associated with differences between the groups. It describes this sample and tends to overestimate the population effect, especially in small samples (more below). It has no units, so it can be compared across scales, but it still depends on the design: which groups were compared, how far apart they were chosen to be, and how varied the sample is.

\eta^2 = \dfrac{SS_{\text{between}}}{SS_{\text{between}} + SS_{\text{within}}} = \dfrac{800}{800 + 7946} = .09

Of all the differences in students' test scores, about 9% line up with which technique they used, a medium effect by convention. The other 91% is everything else: ability, interest, sleep, luck.

The estimate is noisy. Its 95% confidence interval, from the noncentral F distribution, runs from .00 to .22: from no effect to a large one. Reaching .00 doesn't contradict p = .032. η² can't go below zero, so its interval isn't a test of whether the groups differ, and a two-sided 95% interval lines up with a stricter cutoff than .05. The 90% interval, [.005, .20], is the one whose lower limit clears zero exactly when p < .05. Even with no real differences, η² would average about 2 ÷ 74 ≈ .03 in a study this size. η² describes this sample, and as an estimate of the population it runs a little high. Omega squared corrects for that, though it is only less biased, not unbiased (Lakens, 2013). Here $\omega^2 = \dfrac{SS_{\text{between}} - (k - 1)\,MS_{\text{within}}}{SS_{\text{between}} + SS_{\text{within}} + MS_{\text{within}}} = .07$ . Report whichever your field expects, and say which.

η² covers all three groups at once, and it belongs to this design, not to retrieval practice itself. For the comparison readers care about, give its size in points: retrieval practice beat rereading by 8 points on a 100-point test, about 0.76 of the pooled standard deviation (Cohen's d = 0.76). Benchmarks are conventions. Whether 8 points is worth changing how students study is a question about learning, not statistics.

Benchmarks Eta Squared at a Glance

Below Small

η² < .01

Small

.01 ≤ η² < .06

Medium

.06 ≤ η² < .14

Large

η² ≥ .14

Full benchmarks and caveats ↑

Statistical Significance

p answers one narrow question: if every group's population mean were the same, how often would the sample means spread at least this far apart? Only a large F counts against the null, so p comes from one tail of the F distribution, but it responds to differences in any direction. Below the conventional α = .05, the result is called statistically significant.

If the three techniques worked equally well, averages at least as spread out as 68, 72 and 76 would turn up about 3.2% of the time, roughly 1 in 32. That's below .05, so it's significant.

p is not the probability that the null hypothesis is true, and it says nothing about how large the differences are or which groups differ. That's what η² and the follow-up comparisons are for.

Benchmarks p at a Glance

Very Strong

p < .001

Strong

.001 ≤ p < .01

Significant

.01 ≤ p < .05

Weak

.05 ≤ p < .10

Little

p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. A real effect doesn't guarantee a significant result: with too few people, it can easily come out non-significant. Power depends on how big the true effect is, how many people are in each group, how many groups there are, and α.

Power calculators for ANOVA use Cohen's f rather than η². It's the spread of the group means in standard deviations, and it carries the same information: $f = \sqrt{\eta^2 / (1 - \eta^2)}$ . Cohen's conventions are .10, .25 and .40 for small, medium and large (Cohen, 1992), which match η² of about .01, .06 and .14.

Suppose, before collecting any data, we expected a medium effect, f = 0.25 (η² ≈ .06). With 53 students per technique, 159 in all, the study would come out significant about 80% of the time, and miss the effect about 20% of the time.

Our observed η² of .09 works out to f = 0.32, but don't plan the next study from it: with 25 per group, that estimate is noisy, and η² runs a little high to begin with. Also remember that power for the F test isn't power for the pairwise comparisons, which is usually lower for any single pair, so plan the sample size for the comparison you care about.

Researchers usually aim for at least 80% power. Plan it before collecting data, using the smallest effect that would matter. Try it:

Other Info

Follow-Up and Post Hoc Tests

Planned Contrasts

Planned Contrasts: Use when theory predicted a specific comparison before you collected data, like retrieval practice versus the two other techniques combined. Test it directly, whether or not the F is significant; with several contrasts, correct with Holm. In the study-technique study that contrast gives t(72) = 2.33, p = .023. With ordered groups, such as increasing doses, a linear trend contrast tests whether scores rise or fall steadily across them; the ANOVA's F ignores the order (Maxwell et al., 2018).

Post Hoc Tests

Tukey's HSD: The standard choice for comparing every pair when group sizes and spreads are similar, as here. It's step 9 of the Napkin Math: only retrieval practice versus rereading differed, p = .024.
Games-Howell: Use instead of Tukey when the groups' variances differ (Games & Howell, 1976), especially with unequal group sizes. Tukey's HSD already handles unequal sizes when variances are similar (the Tukey-Kramer version). For our data it agrees: retrieval practice versus rereading, p = .026; the other two pairs, p > .35.
Bonferroni Correction: Works for any set of comparisons, but it's conservative, and Holm gives the same protection with more power.
Holm Correction: Holm's (1979) step-down correction is a good choice when you compare only some of the pairs, or run tests other than mean comparisons. For our data, retrieval practice versus rereading has a Holm-adjusted p = .026.

Related Methods

General CaseMultiple Linear Regression: Code the groups as k − 1 dummy variables and regress the outcome on them: the regression's F test is this F, and its R² is η². ANOVA is regression with categorical predictors.
General CaseMANOVA: the same comparison with two or more outcomes at once. With a single outcome, MANOVA's test reduces to the one-way ANOVA's F.
General CaseFactorial Between-Subjects ANOVA: A one-way ANOVA is a factorial ANOVA with a single factor. Add a second grouping variable and you get two main effects and their interaction.
Alternative MeasureKruskal-Wallis Test: The rank-based comparison of three or more groups, for when your outcome is ordered ratings or your question is whether some groups tend to score higher than others, rather than whether the means differ. It isn't ANOVA with the normality assumption removed: it answers a different question. If scores are skewed but the population means remain the question, consider Welch's ANOVA or another method for comparing means, with interval methods suited to the distributions and group sizes.
Related DesignBayesian ANOVA: Same design, same model, different question. Instead of asking whether the group means spread further apart than chance would allow, it weighs how much the data favor unequal means over equal ones, so it can also find evidence that they're equal.
Special CaseIndependent-Samples t-Test: With exactly two groups, the independent-samples t-test gives the same answer: the ANOVA's F is Student's t squared, and Welch's ANOVA's F is Welch's t squared, with the same p.
Related DesignOne-Way Repeated-Measures ANOVA: Both compare three or more means. Use the repeated-measures version when the same people are measured in every condition. It removes stable differences between people from the error term, so it usually needs far fewer people for the same power.
AlternativeMultilevel Regression (HLM): An alternative if “Independent Students” doesn't hold. A multilevel model, which accounts for the clustering.

References

Fisher, R. A. (1925). Statistical methods for research workers. Oliver and Boyd.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155
“The ES index is the standard deviation of the g population means divided by the common within-population standard deviation.” (p. 157)
Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38(3–4), 330–336. https://doi.org/10.1093/biomet/38.3-4.330
Games, P. A., & Howell, J. F. (1976). Pairwise multiple comparison procedures with unequal N's and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1(2), 113–125. https://doi.org/10.3102/10769986001002113
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
Blanca, M. J., Alarcón, R., Arnau, J., Bono, R., & Bendayan, R. (2017). Non-normal data: Is ANOVA still a valid option? Psicothema, 29(4), 552–557. https://doi.org/10.7334/psicothema2016.383
“The results showed that in terms of Type I error the F-test was robust in 100% of the cases studied, independently of the manipulated conditions.” (Abstract, Results)
Delacre, M., Leys, C., Mora, Y. L., & Lakens, D. (2019). Taking parametric assumptions seriously: Arguments for the use of Welch's F-test instead of the classical F-test in one-way ANOVA. International Review of Social Psychology, 32(1), 13. https://doi.org/10.5334/irsp.198
“we therefore recommend using the W-test by default when comparing means.” (Abstract)
Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018). Designing experiments and analyzing data: A model comparison perspective (3rd ed.). Routledge. https://doi.org/10.4324/9781315642956
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863
“Cohen (1988) has provided benchmarks to define small (η2 = 0.01), medium (η2 = 0.06), and large (η2 = 0.14) effects.” (Eta-squared in between and within-subjects comparisons)
Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
“we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)