Stats for
Menu

Nonparametric Group Comparisons

Friedman Test

The Friedman test compares ranked scores from the same people across conditions.

The Friedman test takes an ordinal or skewed outcome measured in the same people under three or more conditions and tests whether people tend to score higher in some conditions than in others.

Example

Does background music change how well students can concentrate? Twenty-four students each studied for 20 minutes in silence, with instrumental music and with music with lyrics, in counterbalanced order, rating their concentration from 1 to 7 after each session.

Same People, Several Conditions + Ordinal Outcome → Friedman Test
Tests
Differences among conditions in how each person ranks them
Observations
Related
Parametric
No
Headline Statistic
\chi^2_F
Effect Size
Kendall's W
Groups
3+

The Basics

When to Use It

Use it when the same people are measured under three or more conditions, or at three or more times, on an outcome that can be put in order, such as a rating scale or a skewed score, and you want to know whether some conditions tend to come out higher. It needs neither normality nor equal spreads across conditions. That safety can cost power: it uses only each person's order of the conditions, so when the repeated-measures ANOVA's assumptions are reasonable it finds a real difference more often. With skewed or outlier-prone scores, the rank test can do as well or better. And it reacts to any difference that changes how people order the conditions, not only a shift in level, so check the medians and each person's ratings to see what differs. For example:

  • Does background music change how well students can concentrate?
  • Do patients' pain ratings (0 to 10) differ across three sessions of a relaxation program?
  • Do people rate the attractiveness of the same faces differently under bright, dim and colored lighting?

Vocabulary You Need

Definition Rank

A score's position when all the scores are lined up from lowest to highest: 1 for the lowest, 2 for the next, and so on. Tied scores share the average of the positions they fill. Ranks keep the order of the scores but throw away the distances between them, so one extreme score counts only as the highest and can't dominate a rank-based test, and taking logs or square roots of the scores leaves the ranks unchanged. Glossary →

Definition Rank Sum (R)

Add up the ranks that belong to one group or condition. When the groups are separate people, everyone from all the groups is ranked together; when the same people are measured in every condition, each person's scores are ranked among themselves, and a condition's rank sum adds its rank from every person. If no group or condition tends to score higher, each rank sum is about its fair share of the total; one whose scores tend to be higher collects more than its share. Glossary →

Definition Chi-Square Statistic (\chi^2)

In a table test, for each cell, the squared gap between the observed and expected count, divided by the expected count, added up across all the cells. It is 0 when every count matches its expectation and grows as they drift apart. Other tests, such as likelihood-ratio tests that compare two models, also produce a χ², which grows with how much worse the more constrained model fits. Either way, it is compared with a chi-square distribution, the values it would take across many samples if the null hypothesis were true, to get a p-value; that distribution depends on the degrees of freedom, and its average equals them. It also grows with sample size, so it isn't a measure of how strong an effect is. Glossary →

Definition Kendall's W (W)

How consistently a set of people (or judges) order the same conditions or items, from 0 to 1 (everyone orders them identically). With no agreement beyond chance it still averages about 1 ÷ the number of people, so it runs high in small samples. After a Friedman test it serves as the effect size, W = χ² / (n(k − 1)), with n people and k conditions. It describes agreement on the order, not how far apart the conditions are on the original scale. Glossary →

Assumptions and Inference

1. Independent People (for the p-Value) Different people are independent of one another, though each person's own scores are meant to be related. The usual chi-square p-value is also a large-sample approximation; with few people, an exact or permutation p helps.
What It Means
Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To
Observations
How to Check
By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters
A lot. Each person's own scores are meant to be related, but different people must not influence each other. If students studied side by side and compared notes, their rankings are more alike than the test expects, and p comes out too small.

What to Do

  • Alternative Test Multilevel Regression (HLM). A multilevel (mixed) model with random effects for people and for the clusters they belong to.
  • Design Change Test each person on their own, so no one's ratings can sway anyone else's.
2. Ordered Outcome The scores can be put in order from lowest to highest.
What It Means
A higher value means more of something, so each person's scores across the conditions can be put in order. Ratings, rankings and continuous scores all qualify; unordered categories don't. Rank-based methods use only that order, not the distances between scores.
Applies To
Outcome
How to Check
By design. Ask whether a higher score means more of something. Then count how many people gave tied scores, since ties carry less information about the order.
How Much It Matters
Essential: without an order, ranks mean nothing. Ties are allowed and corrected for, but each tie leaves less information about the order. In the background-music study 14 of the 24 students gave at least two conditions the same rating, and 3 rated all three the same, which adds nothing to the test.

What to Do

  • Alternative Test Cochran's Q. For a yes/no outcome measured in three or more conditions.
  • Design Change Use a scale with more points, such as a 0-to-100 slider, so fewer people give two conditions the same score.
3. Complete Data Every person has a score in every condition.
What It Means
The test compares conditions within each person, so it needs each person's full set of scores. Anyone missing a condition is dropped from the analysis entirely.
Applies To
Observations
How to Check
Count the people with a missing condition and ask why they're missing. If dropouts are more likely in one condition, dropping them can bias the result.
How Much It Matters
Depends on why data are missing. A few people missing at random cost only power. If students who found music distracting skipped the lyrics session, dropping them would hide the very effect you're looking for.

What to Do

  • Alternative Test Multilevel Regression (HLM). A multilevel model uses everyone's available scores instead of dropping incomplete people.
  • Alternative Test Generalized Estimating Equations (GEE). Another model that keeps people with some missing conditions.
  • Proceed with Caveat Say how many people were dropped and from which condition, so readers can judge the risk.

Limitations

This Is Not Your Test If…

What It Doesn't Tell You

  • How much. Ranks keep only the order. The test can't say how many points the concentration ratings dropped; the medians and the ratings themselves do that.
  • Why. Because we set the conditions and balanced their order, we can say lyrics lowered how well students felt they concentrated, compared with silence, but not why: competing words, a worse mood, or simply liking silence.
  • Whether it matters. Significance doesn't make a one-point drop on a 7-point self-rating something that changes how much students learn.
  • That instrumental music is harmless. It didn't differ reliably from silence or lyrics, but with 24 students that's inconclusive, not evidence that it has no effect.

Common Mistakes

  • Don'tTreat a larger Friedman χ² as proof of a stronger effect.

    InsteadReport Kendall's W, which standardizes it for the number of people and conditions, and give each condition's median and the spread of its ratings: W describes how consistently people order the conditions, not differences in rating points.

  • Don'tUse the Friedman test when each condition has different people.

    InsteadUse the Kruskal-Wallis test, which compares independent groups by ranks.

  • Don'tRead W as how far apart the conditions are in rating points.

    InsteadRead it as how consistently people put the conditions in the same order, and give the medians for size.

  • Don'tFollow a significant result with Wilcoxon tests for every pair, each at .05.

    InsteadCorrect across the pairs with Holm, or plan one or two comparisons in advance.

  • Don'tRun a Friedman test on just two conditions.

    InsteadUse the Wilcoxon signed-rank test. Friedman with two conditions only counts who went up and who went down.

  • Don'tReport only means and SDs of the ratings, as if the test compared means.

    InsteadReport each condition's median and mean rank, which is what the test works with.

  • Don'tRead a p-value as the chance that the null hypothesis is true.

    InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance and Effect Size Guidelines

Benchmarks Significance (p Value)
  • p < .001

    Very strong evidence against the null hypothesis.

  • .001 ≤ p < .01

    Strong evidence against the null hypothesis.

  • .01 ≤ p < .05

    Statistically significant at α = .05. Moderate evidence against the null hypothesis.

  • .05 ≤ p < .10

    Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.

  • p ≥ .10

    Not significant. Little evidence against the null, which is not the same as evidence of no effect.

These evidence labels are one common convention for describing p in graded language rather than as significant or not; the boundaries are rough guides, not rules, and other authors draw them differently. Muff et al. (2022) argue for this kind of language of evidence over a single cutoff. α is chosen before the analysis, and p never says how big or important an effect is.

More about statistical significance ↓

Benchmarks Effect Size (Kendall's W)
  • W < .10 Below Small

    Below the lowest point on the scale. Little overall concordance in the orderings, by convention.

  • .10 ≤ W < .30 Very Weak

    Very weak agreement in the ordering, by convention.

  • .30 ≤ W < .50 Weak

    Weak agreement in the ordering, by convention.

  • .50 ≤ W < .70 Moderate

    Moderate agreement in the ordering, by convention.

  • .70 ≤ W < .90 Strong

    Strong agreement in the ordering, by convention.

  • W ≥ .90 Unusually Strong

    Very high concordance in the orderings, by convention.

Schmidt (1997) wrote these labels for how strongly a panel of experts agrees on a ranking, as a guide rather than exact cutoffs; each range starts at one of his points. They aren't established thresholds for clinical or educational importance. After a Friedman test, W is how consistently people order the conditions, an aggregate concordance, not the share of people who give the same order. It isn't a measure of how much scores change on the original scale.

More about effect size ↓

How It's Reported

Equation Form

Example

Mdn = 5, 5 and 4, W = .15, 95% CI [.04, .39], χ²(2, N = 24) = 7.01, p = .030

Full Text (APA Style)

Example

Students rated their concentration highest after studying in silence (Mdn = 5, mean rank = 2.33), then with instrumental music (Mdn = 5, mean rank = 2.00), and lowest with lyrics (Mdn = 4, mean rank = 1.67). Students showed very weak agreement in their ordering of the three conditions, Kendall's W = .15, 95% CI [.04, .39] (percentile bootstrap resampling students, 50,000 resamples, seeded). A Friedman test showed the differences were statistically significant, χ²(2, N = 24) = 7.01, p = .030. Wilcoxon signed-rank tests with a Holm correction showed that concentration was rated lower with lyrics than in silence (13 of the 16 students whose ratings differed rated silence higher), p = .022; the other two pairs did not differ reliably (p = .164 and .350).

  • Give each condition's median and mean rank. The medians say where ratings sit; the mean ranks show the order the test found.
  • Report N and the degrees of freedom (k − 1) with χ², and say it's a Friedman test, since several tests report a χ².
  • There's no single, widely preferred simple formula for W's interval, so give a bootstrap CI and say how it was made: the method (percentile), what was resampled (whole people), the number of resamples and that it was seeded. Another seed or program moves the limits slightly.
  • Follow a significant result with the pairwise comparisons, and name the correction.
  • Italicize Latin symbols (Mdn, W, N, p) but not Greek ones (χ²), and report exact p to three decimals.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how surprising.

W = .15 Students agreed only a little on how to order the three conditions: some consistency, with plenty of disagreement and ties. It's a standardized measure of the Friedman result, not how much concentration changed on the 1-to-7 scale. That's very weak agreement by the benchmarks above; whether a one-point drop in the median rating with lyrics matters is a question about studying, not statistics. Benchmarks ↑
95% CI [.04, .39] A seeded percentile bootstrap with 50,000 resamples of whole students. With 24 students, agreement could plausibly be anything from negligible (.04, about what chance alone gives with 24 students) to weak (.39), so the study can't say much about how strongly students share this order. Treat it as rough: W runs high in small samples, and the bootstrap carries that into the interval. R's rstatix with seed 2026 and its default 1,000 resamples gives a slightly different [.04, .38], because R's random numbers differ.
χ²(2, N = 24) = 7.01 The rank sums are further from an even split than chance usually produces: with no real difference, this χ² averages 2. The 2 in parentheses is the degrees of freedom, k − 1, and N = 24 is the number of students.
p = .030 If every student's ordering of the three conditions were down to chance, rank sums at least this uneven would turn up about 3% of the time. That's below .05, so the result is statistically significant. Benchmarks ↑

Calculations

The Running Example

Example

The background-music study: 24 students, each rating their concentration (1 to 7) after studying in silence, with instrumental music and with music with lyrics.

Null Hypothesis:
the three condition labels are interchangeable within each student
Alternative:
at least one condition systematically tends to get different within-student ranks

The null says background music makes no difference, so the condition labels are interchangeable within each student and no condition systematically gets higher or lower ranks; which condition a student happens to rate highest is down to chance. (With tied ratings a student has fewer distinct orders than the six strict ones, which is why the null is stated in terms of interchangeable labels.) The alternative doesn't say which conditions differ or in what direction. Because each student is ranked only against themselves, it doesn't matter that some students rate everything high and others everything low.

What Your Data Should Look Like

One score per person per condition. Most software wants it wide: one row per person, one column per condition. Ours is stored long, one row per person per condition with columns for the person, the condition and the score, and gets pivoted to wide for the test. Anyone missing a condition drops out.

Example
participantconditionconcentration
1silence6
1instrumental5
1lyrics5
2silence4
2instrumental5
2lyrics5
… 66 more rows
nMdnQ1–Q3Mean Rank
Silence2454–6.252.33
Instrumental2454–62.00
Lyrics2444–51.67

24 participants × 3 conditions: every participant has a score for every condition.

Q1–Q3 is where the middle half of the scores lies, from the 25th to the 75th percentile (R's default quantile definition).

Simulated data.

Download all 72 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the background-music study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

  1. 1. Rank Each Student's Three Ratings from Lowest to Highest

    The test compares conditions within each person, so each student's ratings are ranked only against each other. Ties share the average of the ranks they would take up.

    r_{ij} = \text{rank of } x_{ij} \text{ among person } i\text{'s } k \text{ scores}

    r_{1,\text{silence}} = 3.0

    r_{1,\text{instrumental}} = 1.5

    r_{1,\text{lyrics}} = 1.5

    x_{ij} is student i's rating in condition j. Student 1 rated silence 6 and both kinds of music 5, so silence gets the top rank, 3, and the two 5s share ranks 1 and 2, getting 1.5 each.

  2. 2. Add Up Each Condition's Ranks

    If one condition keeps coming out on top, its ranks pile up. A condition's rank sum adds its rank from every student.

    R_j = \sum_{i=1}^{n} r_{ij}

    R_{\text{silence}} = 56

    R_{\text{instrumental}} = 48

    R_{\text{lyrics}} = 40

    r_{ij} is each student's rank for condition j, and n is our 24 students. Silence collects 56, instrumental music 48 and lyrics 40. Every student hands out 1 + 2 + 3 = 6 ranks, so the three sums always total 24 × 6 = 144. Divided by our 24 students, they give mean ranks of 2.33, 2.00 and 1.67, against 2 if the conditions didn't matter.

  3. 3. Find What Each Rank Sum Would Be If the Conditions Didn't Matter

    If background music made no difference, each condition would get a fair share of the ranks, the same for all three.

    E(R_j) = \dfrac{n(k + 1)}{2} =\dfrac{24 \times (3 + 1)}{2}

    = 48

    n is our 24 students and k our 3 conditions. Each student's average rank is (3 + 1) ÷ 2 = 2, so over 24 students a condition should collect 24 × 2 = 48.

  4. 4. Measure How Far the Rank Sums Stray from That Fair Share

    Square each rank sum's gap from 48 and add them up, then scale the total so it can be judged against a chi-square distribution. This first version assumes no one gave two conditions the same rating.

    \chi^2_{\text{no ties}} = \dfrac{12}{nk(k + 1)} \sum_{j=1}^{k} \left(R_j - \dfrac{n(k + 1)}{2}\right)^2 =\dfrac{12}{24 \times 3 \times 4}\left(8^2 + 0^2 + (-8)^2\right)

    \chi^2_{\text{no ties}} = 5.33

    Silence's 56 is 8 above 48, instrumental's 48 is right on it, and lyrics' 40 is 8 below. 64 + 0 + 64 = 128, and 12 ÷ (24 × 3 × 4) = 12 ÷ 288, so 128 × 12 ÷ 288 = 5.33.

  5. 5. Adjust for Students Who Gave Two Conditions the Same Rating

    Tied ratings pull ranks toward the middle, so the rank sums vary less than they would without ties, even when the conditions don't matter. The correction rescales χ² to the spread the ranks could actually have, so it can still be judged against the same chi-square distribution.

    \chi^2_F = \dfrac{\chi^2_{\text{no ties}}}{1 - \dfrac{\sum (t^3 - t)}{nk(k^2 - 1)}} =\dfrac{5.33}{1 - \dfrac{138}{24 \times 3 \times 8}}

    = 7.01

    \text{divisor} = 0.760

    t is the size of each tied set. Eleven students tied two conditions (2³ − 2 = 6 each, 66 in all) and three tied all three (3³ − 3 = 24 each, 72), so \sum (t^3 - t) = 138. The divisor is 1 − 138 ÷ 576 = 0.760, and 5.33 ÷ 0.760 = 7.01.

  6. 6. Account for How Much Information We Had

    The three rank sums always add up to 144, so once two are known the third is fixed. df tells us which chi-square distribution to judge χ² against.

    df = k - 1 =3 - 1

    = 2

    k is our 3 conditions, not our 24 students. 3 − 1 = 2.

  7. 7. Measure How Consistently Students Ordered the Conditions

    χ² grows with the number of people; W doesn't. Dividing χ² by the largest value it could reach gives a score up to 1 (everyone ranks the conditions identically). With no shared order it still averages about 1/n, .04 with our 24 students, rather than 0.

    W = \dfrac{\chi^2_F}{n(k - 1)} =\dfrac{7.01}{24 \times (3 - 1)}

    = .15

    95\%\ \text{CI} = [.04, .39]

    \chi^2_F is our 7.01, n our 24 students and k - 1 = 2. The biggest χ² possible is 24 × 2 = 48, reached only if every student ranked the conditions the same way. 7.01 ÷ 48 = .15. There's no single, widely preferred simple formula for its interval, so we bootstrap it, redrawing 24 students (all three ratings together) with replacement 50,000 times (seeded); the middle 95% of those W values runs from .04 to .39.

  8. 8. Ask How Surprising the Result Would Be If Music Made No Difference

    Finally, ask how often chance alone would push the rank sums this far from an even split if all three conditions were really alike.

    p = P(\chi^2_{df} \ge \chi^2) =P(\chi^2_{2} \ge 7.01)

    = .030

    \chi^2_{2} is a chi-square distribution with 2 df. The chance it lands at or above 7.01 is .030, about 1 in 33. Only large values count against the null, so there's one tail, but it catches differences in any direction.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)
  1. SPSS wants one column per condition. Our file is long, so first use Data → Restructure → Restructure selected cases into variables, with participant as the Identifier Variable and condition as the Index Variable. You get one row per student with columns silence, instrumental and lyrics. Check first that each student has exactly one row per condition: a duplicated or missing row breaks the restructure or silently drops the student.
  2. Analyze → Nonparametric Tests → Legacy Dialogs → K Related Samples.
  3. Move silence, instrumental and lyrics into Test Variables. Under Test Type, check Friedman and Kendall's W.
  4. Statistics: check Quartiles, and click OK.
  5. For the pairwise comparisons: Analyze → Nonparametric Tests → Legacy Dialogs → 2 Related Samples, add the three pairs, check Wilcoxon, and click OK. Then apply the Holm correction by hand (below).

Or paste this syntax:

NPAR TESTS
  /FRIEDMAN=silence instrumental lyrics
  /KENDALL=silence instrumental lyrics
  /STATISTICS QUARTILES
  /MISSING LISTWISE.

NPAR TESTS
  /WILCOXON=silence silence instrumental WITH instrumental lyrics lyrics (PAIRED)
  /MISSING ANALYSIS.

Reading the Output

Read the tables in this order: the medians (the 50th percentiles in Descriptive Statistics) and the Mean Rank of each condition, then Kendall's W, then Chi-Square, df and Asymp. Sig., then the pairwise Wilcoxon tests.

Ranks

Mean Rank
silence 2.33
instrumental 2.00
lyrics 1.67

Test Statistics (Kendall's W)

N Kendall's W Chi-Square df Asymp. Sig.
24 .146 7.014 2 .030

The Friedman test prints the same Chi-Square, df and Asymp. Sig. in its own Test Statistics table; the Kendall's W table adds W. SPSS corrects both for ties.

Test Statistics (Wilcoxon Signed Ranks Test)

instrumental − silence lyrics − silence lyrics − instrumental
Z -.934 -2.689 -1.740
Asymp. Sig. (2-tailed) .350 .007 .082

SPSS's Wilcoxon Z is always negative, because it's based on the smaller of the two rank sums, so its sign says nothing about direction. Read direction from the Ranks table (how many Negative and Positive Ranks each pair has) or from the footnote under Z, “Based on positive ranks” or “Based on negative ranks.” Here all three pairs lean the same way: the later condition in each pair was usually rated lower. Pairs with no difference (ties) are left out. SPSS doesn't adjust these p-values. For Holm, sort them from smallest to largest and multiply by 3, 2 and 1, keeping each at least as large as the one before: .007 × 3 = .022, .082 × 2 = .164, .350 × 1 = .350. (Use the unrounded .00718 for the first.)

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R 
music <- read.csv("data.csv")

friedman.test(concentration ~ condition | participant, data = music)

# Kendall's W
chi <- friedman.test(concentration ~ condition | participant, data = music)$statistic
unname(chi) / (24 * (3 - 1))

# 95% CI for W: percentile bootstrap of whole students, 1,000 resamples, seeded
# (install.packages("rstatix") once)
set.seed(2026)
rstatix::friedman_effsize(music, concentration ~ condition | participant, ci = TRUE)

# Which pairs differ: Wilcoxon signed-rank tests with a Holm correction.
# Rows are sorted by participant, so each condition's scores line up by student.
pairwise.wilcox.test(music$concentration, music$condition, paired = TRUE,
                     exact = FALSE, correct = FALSE, p.adjust.method = "holm")

	Friedman rank sum test

data:  concentration and condition and participant
Friedman chi-squared = 7.0137, df = 2, p-value = 0.02999

[1] 0.1461187
# A tibble: 1 × 7
  .y.               n effsize conf.low conf.high method    magnitude
* <chr>         <int>   <dbl>    <dbl>     <dbl> <chr>     <ord>
1 concentration    24   0.146     0.04      0.38 Kendall W small

	Pairwise comparisons using Wilcoxon signed rank test

data:  music$concentration and music$condition

        instrumental lyrics
lyrics  0.164        -
silence 0.350        0.022

P value adjustment method: holm

friedman.test corrects for ties, like SPSS. rstatix::friedman_effsize resamples whole students (rows of the wide data) and gives a percentile bootstrap 95% CI for W of [.04, .38], rounded to two decimals. This page's [.04, .39] comes from the same kind of bootstrap with 50,000 resamples and different random numbers, so the upper limit differs slightly; set a seed, or R's interval changes every run. Its magnitude column uses Cohen-style cutoffs (.1, .3, .5) that we couldn't trace to a source, so this page labels W with Schmidt's (1997) agreement terms instead.

exact = FALSE, correct = FALSE makes the pairwise tests match SPSS, Python and this page: zero differences dropped, normal approximation, no continuity correction. Leave them out and the result depends on your R version. Since R 4.6.0, R runs exact tests that allow for ties and keep the zero differences in the ranking, giving Holm-adjusted p = .233, .298 and .027. Older versions warn about ties and zeros and use the normal approximation with a continuity correction, giving .17, .36 and .023. Silence versus lyrics is the only pair below .05 every way.

Output from a real run: R 4.6.1 (rstatix 1.1.0, boot 1.3.32), run 2026-09-30.

Python 
import pandas as pd
from scipy import stats
from statsmodels.stats.multitest import multipletests

music = pd.read_csv("data.csv")
wide = music.pivot(index="participant", columns="condition", values="concentration")

result = stats.friedmanchisquare(wide["silence"], wide["instrumental"], wide["lyrics"])
result.statistic   # 7.0137
result.pvalue      # 0.02999

# Kendall's W
n, k = wide.shape
result.statistic / (n * (k - 1))   # 0.1461

# Pairwise Wilcoxon signed-rank tests, Holm-corrected
pairs = [("silence", "instrumental"), ("silence", "lyrics"), ("instrumental", "lyrics")]
p = [stats.wilcoxon(wide[a], wide[b]).pvalue for a, b in pairs]
p                                      # [0.3501, 0.00718, 0.0819]
multipletests(p, method="holm")[1]     # [0.3501, 0.0215, 0.1638]

SciPy wants one array per condition, so pivot turns the long file into one column per condition first. With zeros or ties in the differences and more than a handful of pairs, stats.wilcoxon uses the normal approximation without a continuity correction, matching SPSS. pingouin.friedman(data=music, dv="concentration", within="condition", subject="participant") works on the long file directly and prints W alongside the tie-corrected χ² (which it calls Q) and p.

Output from a real run: Python 3.9.6 (scipy 1.13.1, statsmodels 0.14.6, pandas 2.3.3), run 2026-09-29.

Impact

Effect Size

Kendall's W (Kendall & Babington Smith, 1939) expresses the result as agreement: how consistently people put the conditions in the same order, from 0 to 1. It's χ² divided by the largest value χ² could reach (Tomczak & Tomczak, 2014), so unlike χ² it doesn't keep growing as more people take part. It isn't 0 when there's no shared order, though: chance alone gives W of about 1/n, .04 with our 24 students and .10 with 10, so W from a small study runs high. Keep that in mind before comparing W across studies of different sizes.

W = \dfrac{\chi^2_F}{n(k - 1)} = \dfrac{7.01}{24 \times 2} = .15

If all 24 students had ranked silence first, instrumental second and lyrics last, χ² would have hit its ceiling of 48 and W would be 1. Ours reached 7.01 of that 48, so W = .15: there's a shared lean, but many students ordered the conditions differently or tied them. That's very weak agreement by convention.

There's no single, widely preferred simple formula for a confidence interval for W in this setting, so we use a percentile bootstrap that resamples whole students. Our percentile bootstrap redrew 24 students with replacement, keeping each student's three ratings together, recomputed W, repeated that 50,000 times from a fixed seed, and kept the middle 95%: 95% CI [.04, .39]. Agreement could be anything from negligible to weak; the lower limit, .04, is about what chance alone gives with 24 students, and 24 students can't narrow it further. Treat the interval as rough: W can't go below 0 and runs high in small samples, and a percentile bootstrap carries both quirks into the interval (our resampled Ws averaged .18, against the observed .15). For the same reason, and because the interval isn't built as an inversion of the Friedman test, whether it excludes 0 isn't a significance test; the χ² test is. Report the method, the number of resamples and the seed; another seed moves the limits slightly. R's rstatix (friedman_effsize(..., ci = TRUE)) with set.seed(2026) and its default 1,000 resamples gives [.04, .38], because R's random numbers differ.

Another way to see W: rank each person's scores, run a repeated-measures ANOVA on those ranks, and W is exactly the partial η² for conditions. It's the share of the within-person variation in ranks that lines up with the condition.

W says nothing about distance on the rating scale, so give that too. Students' median concentration was 5 in silence and with instrumental music, and 4 with lyrics. Benchmarks are conventions, and these were written for panels of judges agreeing on a ranking (Schmidt, 1997). Whether a one-point drop in self-rated concentration is worth turning the lyrics off is a question about studying, not statistics.

Benchmarks Kendall's W at a Glance
Below Small
W < .10
Very Weak
.10 ≤ W < .30
Weak
.30 ≤ W < .50
Moderate
.50 ≤ W < .70
Strong
.70 ≤ W < .90
Unusually Strong
W ≥ .90

Full benchmarks and caveats ↑

Statistical Significance

p answers one narrow question: if the conditions made no difference, so that every student's ordering of them was down to chance, how often would the rank sums spread at least this far from an even split? Only a large χ² counts against the null, so p comes from one tail of the chi-square distribution, but it responds to differences in any direction. Below the conventional α = .05, the result is called statistically significant.

If every student's ordering of the three conditions were down to chance, rank sums at least as uneven as 56, 48 and 40 would turn up about 3.0% of the time, roughly 1 in 33. That's below .05, so it's significant.

p is not the probability that the null hypothesis is true, and it says nothing about how large the differences are or which conditions differ. That's what W, the medians and the pairwise comparisons are for. The p here comes from the chi-square approximation, which is what SPSS, R and Python report by default. With only a handful of people, look for software that computes an exact p instead.

Benchmarks p at a Glance
Very Strong
p < .001
Strong
.001 ≤ p < .01
Significant
.01 ≤ p < .05
Weak
.05 ≤ p < .10
Little
p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. A real effect doesn't guarantee a significant result: with too few people, it can easily come out non-significant. Power depends on how big the true effect is, how many people you have, how many conditions there are, α, and, for rank tests on rating scales, how many ties the scale produces.

There's no simple formula for the Friedman test's power, so this page has no calculator. The dependable way is simulation: write down what you expect the ratings to look like, including how much people differ from each other and how coarse the scale is, generate thousands of pretend studies, and count how often the test comes out significant.

We did that with the model behind our simulated data. Each student's underlying concentration is 4.3 plus their own general level (SD 1 across students), plus 0.35 in silence, 0.15 with instrumental music or −0.5 with lyrics, plus session-to-session noise (SD 1); it's then rounded to a whole number and kept within 1 to 7. With our 24 students, about 66% of studies came out significant, so a significant result was likely but far from assured. With 36 students it's about 84%. Using only the order has a price: on the same model without rounding, so the scores are normal, a repeated-measures ANOVA came out significant in about 78% of studies of 24 students and the Friedman test in 63%.

Our observed W of .15 is a noisy estimate from 24 students, so don't plan the next study from it alone. A finer scale, with fewer ties, is worth trying in the simulation too: it may buy power without more people.

Researchers usually aim for at least 80% power. Plan it before collecting data, using the smallest effect that would matter.

Other Info

Follow-Up and Post Hoc Tests

Planned Contrasts

  • Planned Contrasts: Use when you predicted a specific comparison before collecting data, like lyrics versus silence. Test it directly with a Wilcoxon signed-rank test, whether or not the Friedman test is significant; with more than one planned comparison, correct with Holm. In the background-music study that single comparison gives p = .007.

Post Hoc Tests

  • Pairwise Wilcoxon Signed-Rank Tests: The standard follow-up for comparing every pair after a significant result. Each pair gets its own Wilcoxon (1945) signed-rank test on the same students, then a Holm correction across the three. Only silence versus lyrics differed, Holm-adjusted p = .022; instrumental versus lyrics, p = .164; silence versus instrumental, p = .350. Tests built on the Friedman mean ranks (Nemenyi's, or the one in Conover, 1999) are also common, but their verdict on one pair can change with which other conditions were in the study (Benavoli et al., 2016).

References

  • Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200), 675–701. https://doi.org/10.1080/01621459.1937.10503522
  • Kendall, M. G., & Babington Smith, B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275–287. https://doi.org/10.1214/aoms/1177732186
  • Tomczak, M., & Tomczak, E. (2014). The need to report effect size estimates revisited. An overview of some recommended measures of effect size. Trends in Sport Sciences, 21(1), 19–25. https://tss.awf.poznan.pl/The-need-to-report-effect-size-estimates-revisited-An-overview-of-some-recommended,188960,0,2.html
    “Also, for the Friedman test, a non-parametric statistical test employed to compare three or more paired measurements (repeated measures), an effect size estimate can be calculated (and is referred to as W)” (p. 24)
  • Schmidt, R. C. (1997). Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences, 28(3), 763–774. https://doi.org/10.1111/j.1540-5915.1997.tb01330.x
  • Conover, W. J. (1999). Practical nonparametric statistics (3rd ed.). Wiley.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
  • Benavoli, A., Corani, G., & Mangili, F. (2016). Should we really use post-hoc tests based on mean-ranks? Journal of Machine Learning Research, 17(5), 1–10. https://jmlr.org/papers/v17/benavoli16a.html
    “we suggest instead to perform the multiple comparison using a test whose outcome only depends on the two algorithms being compared, such as the sign-test or the Wilcoxon signed-rank test” (Abstract)
  • Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
    “we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)
  • Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. https://doi.org/10.2307/3001968

Spot an error? Something still confusing? Tell us at support@statsforpsychology.com.