Cochran's Q

• You measured the same people only twice McNemar's Test →
• Each condition has different people Pearson Chi-Square Test →
• Your outcome is a rating or rank, not yes/no Friedman Test →
• You need to adjust for covariates Generalized Estimating Equations (GEE) →
• You want each person's odds, have clustered people, or need to keep people who missed a time point Multilevel Logistic Regression →

• Don'tTreat a larger Q as proof of a larger change.

  InsteadReport the percentage answering yes at each time and the percentage-point differences between the times you care about, with intervals. Q is an omnibus test statistic; it doesn't say how large any particular change was.

• Don'tRun a chi-square test on the time-by-answer table as if each answer came from a different person.

  InsteadUse Cochran's Q, which keeps each person's answers together. The ordinary chi-square test counts the same students three times.

• Don'tStop at a significant Q.

  InsteadFollow it with pairwise comparisons (exact McNemar tests with a discrete Holm adjustment) and report the proportions and differences with their intervals.

• Don'tJudge W against Cohen's benchmarks for η² or Kendall's labels for agreement.

  InsteadJudge the size from the proportions and their differences. W also shrinks as more people give the same answer throughout.

• Don'tDrop people who answered the same way every time before running the test.

  InsteadLeave them in. They don't change Q, but they count in N, the proportions and every interval.

• Don'tCut a score into yes/no just to use Cochran's Q.

  InsteadAnalyze the scores themselves (a Friedman test or a repeated-measures model), unless the cutoff itself is the question. Cutting throws away information and power.

• Don'tCompare W between studies where very different shares of people never changed their answer.

  InsteadCompare the proportions and their differences. W counts people who never changed in N, so it drops as they become more common.

• Don'tRead a p-value as the chance that the null hypothesis is true.

  InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Equation Form

Table 1. Students Who Often Felt Lonely at Each Time Point

Table 2. Pairwise McNemar Tests

Full Text (APA Style)

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

What Your Data Should Look Like

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the loneliness study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

NPAR TESTS
  /COCHRAN=september december april
  /MISSING LISTWISE.

NPAR TESTS
  /MCNEMAR=september september december WITH december april april (PAIRED)
  /MISSING ANALYSIS.

library(DescTools)
library(coin)

lonely <- read.csv("data.csv")
lonely$time <- factor(lonely$time, levels = c("september", "december", "april"))
lonely$student <- factor(lonely$student)   # coin needs the person as a factor

# Share answering yes at each time point
tapply(lonely$lonely, lonely$time, mean)

# Cochran's Q
CochranQTest(lonely ~ time | student, data = lonely)

# Effect size: Q / (N(k - 1))
q <- CochranQTest(lonely ~ time | student, data = lonely)$statistic
unname(q) / (80 * (3 - 1))

# Permutation p (Monte Carlo, so it varies a little with the seed)
set.seed(2026)
pvalue(symmetry_test(lonely ~ time | student, data = lonely, teststat = "quadratic",
                     distribution = approximate(nresample = 100000)))

# Which pairs differ: exact McNemar tests with a discrete Holm adjustment
wide <- xtabs(lonely ~ student + time, data = lonely)   # one row per student
pairs <- list(c("september", "december"), c("september", "april"), c("december", "april"))
yes_no <- sapply(pairs, function(p) sum(wide[, p[1]] == 1 & wide[, p[2]] == 0))
no_yes <- sapply(pairs, function(p) sum(wide[, p[1]] == 0 & wide[, p[2]] == 1))
changed <- yes_no + no_yes
p_exact <- mapply(function(b, n) binom.test(b, n)$p.value, yes_no, changed)

# Chance, under the null, that a pair with n changers gives an exact p at or below `cut`
p_at_or_below <- function(n, cut) {
  attainable <- sapply(0:n, function(x) binom.test(x, n)$p.value)
  sum(dbinom(0:n, n, 0.5)[attainable <= cut * (1 + 1e-7)])
}
ord <- order(p_exact); p_adj <- numeric(3); running <- 0
for (j in seq_along(ord)) {
  still_in <- ord[j:3]
  step <- sum(sapply(still_in, function(i) p_at_or_below(changed[i], p_exact[ord[j]])))
  running <- max(running, min(1, step)); p_adj[ord[j]] <- running
}
data.frame(pair = sapply(pairs, paste, collapse = "-"), yes_no, no_yes, p_exact, p_adj)

september  december     april
     0.45      0.35      0.30

	Cochran's Q test

data:  lonely and time and student
Q = 8, df = 2, p-value = 0.01832

[1] 0.05
[1] 0.01662
99 percent confidence interval:
 0.01559668 0.01768961

                pair yes_no no_yes    p_exact      p_adj
1 september-december     14      6 0.11531830 0.19213104
2    september-april     16      4 0.01181793 0.02781677
3     december-april     10      6 0.45449829 0.45449829

import pandas as pd
from scipy.stats import binom
from statsmodels.stats.contingency_tables import cochrans_q, mcnemar

lonely = pd.read_csv("data.csv")
wide = lonely.pivot(index="student", columns="time", values="lonely")
wide = wide[["september", "december", "april"]]

print(wide.mean())

result = cochrans_q(wide)
print(result)

n, k = wide.shape
print(result.statistic / (n * (k - 1)))

# Which pairs differ: exact McNemar tests with a discrete Holm adjustment
pairs = [("september", "december"), ("september", "april"), ("december", "april")]
tables = [pd.crosstab(pd.Categorical(wide[a], [0, 1]), pd.Categorical(wide[b], [0, 1]), dropna=False)
          for a, b in pairs]
changed = [t.iloc[1, 0] + t.iloc[0, 1] for t in tables]
p_exact = [mcnemar(t, exact=True).pvalue for t in tables]

def p_at_or_below(m, cut):
    """Chance, under the null, that a pair with m changers gives an exact p at or below cut."""
    return sum(binom.pmf(x, m, 0.5) for x in range(m + 1)
               if min(1, 2 * binom.cdf(min(x, m - x), m, 0.5)) <= cut * (1 + 1e-7))

order = sorted(range(len(pairs)), key=lambda i: p_exact[i])
p_adj, running = [0.0] * len(pairs), 0.0
for j, i in enumerate(order):
    step = sum(p_at_or_below(changed[r], p_exact[i]) for r in order[j:])
    running = max(running, min(1.0, step))
    p_adj[i] = running
print(pd.DataFrame({"p_exact": p_exact, "p_adj": p_adj}, index=[f"{a}-{b}" for a, b in pairs]))

time
september    0.45
december     0.35
april        0.30
dtype: float64
df          2
pvalue      0.018315638888734182
statistic   8.0
0.05
                     p_exact     p_adj
september-december  0.115318  0.192131
september-april     0.011818  0.027817
december-april      0.454498  0.454498

There are no established benchmarks for this index specific to Cochran's Q, so there are none here; neither Cohen's conventions for η² nor Kendall's W's agreement labels carry over. Judge the size from the proportions and the differences between them.

Planned Contrasts

• Planned Contrasts: Use when you predicted one specific comparison before collecting data, such as the first time point against the last. Test it with a single McNemar test, whether or not Q is significant; with more than one planned comparison, correct with Holm. In the loneliness study, September against April gives exact p = .012. If the conditions are ordered, like time points, and you expect change in one direction, a trend test (a GEE logistic model with time entered as a number) asks that one question, usually with more power than Q.

Post Hoc Tests

• Pairwise McNemar Tests: Use to compare every pair after a significant Q. Each pair gets its own exact (binomial) McNemar test on the same people, then Westfall et al.'s (2010) discrete Holm adjustment, which counts how small a p each pair's exact test could actually produce. It keeps the familywise error rate at α in any sample size and, in their simulations, had more power than the common shortcut of ordinary Holm on large-sample McNemar p-values, which controls the error rate only approximately. They also describe a bootstrap step-down method, which they prefer in large samples because it accounts for how the pairs are correlated. In the loneliness study only September versus April differed, adjusted p = .028; September versus December, p = .192; December versus April, p = .454. The pairwise 95% intervals in Table 2 aren't adjusted for the three comparisons, so they won't always agree with the adjusted p-values.

• General CaseFriedman Test: Cochran's Q is the Friedman test run on answers coded 0 and 1: with the usual correction for tied ranks, Friedman's χ² on 0/1 data equals Q exactly, so Q ÷ N(k − 1) is Kendall's W computed on those answers. Friedman handles ratings and ranks; Cochran's Q is the yes/no case.
• Related DesignGeneralized Estimating Equations (GEE): Both compare a yes/no outcome across repeated occasions in the same people. GEE adds covariates and between-person groups, keeps people with some missing occasions (unbiased only if they're missing completely at random, unless weighted), and reports each change as an odds ratio.
• Related DesignMultilevel Logistic Regression: A multilevel logistic model with a random intercept for each person also compares a yes/no outcome across occasions, keeps incomplete cases and takes covariates, but its odds ratios describe a typical person rather than the whole group.
• Special CaseMcNemar's Test: the special case with exactly two occasions: a yes/no outcome measured twice in the same people. Its uncorrected χ² equals Cochran's Q for those two occasions.

• Cochran, W. G. (1950). The comparison of percentages in matched samples. Biometrika, 37(3–4), 256–266. https://doi.org/10.1093/biomet/37.3-4.256
• Tate, M. W., & Brown, S. M. (1970). Note on the Cochran Q test. Journal of the American Statistical Association, 65(329), 155–160. https://doi.org/10.1080/01621459.1970.10481069

  “rows containing only 1's or only 0's, i.e., only successes or only failures, do not affect the value of Q” (Abstract)

• Patil, K. D. (1975). Cochran's Q test: Exact distribution. Journal of the American Statistical Association, 70(349), 186–189. https://doi.org/10.1080/01621459.1975.10480285

  “A relatively simple method for computing the exact probability and distribution of the Q statistic is proposed.” (Abstract)

• McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157. https://doi.org/10.1007/BF02295996
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
• Westfall, P. H., Troendle, J. F., & Pennello, G. (2010). Multiple McNemar tests. Biometrics, 66(4), 1185–1191. https://doi.org/10.1111/j.1541-0420.2010.01408.x

  “Bonferroni-Holm with the ordinary McNemar (large sample) test and with the exact McNemar p-values are inferior methods.” (Section 10, Conclusion)

• Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

  “This approach, which appears in various forms, is fundamentally flawed.” (Abstract)

• Tomczak, M., & Tomczak, E. (2014). The need to report effect size estimates revisited. An overview of some recommended measures of effect size. Trends in Sport Sciences, 21(1), 19–25. https://tss.awf.poznan.pl/The-need-to-report-effect-size-estimates-revisited-An-overview-of-some-recommended,188960,0,2.html

  “Also, for the Friedman test, a non-parametric statistical test employed to compare three or more paired measurements (repeated measures), an effect size estimate can be calculated (and is referred to as W)” (p. 24)

• Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009

  “we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)