Fisher's Exact Test

Use Fisher's exact test (also called the Fisher–Irwin test) when you have two independent groups and a yes/no outcome (or any two yes/no variables), counted into a 2 × 2 table, and the sample is small. Because it's exact and never too liberal, it's a safe default for small tables. The traditional rule, chi-square only when every expected frequency is 5 or more, is stricter than it needs to be: for 2 × 2 tables, the “N − 1” chi-square holds its error rate whenever every expected count is at least 1 (Campbell, 2007). Choose one test in advance and report it. Fisher's test is valid for large samples too. To compare one group's rate with a fixed value in a small sample, the exact counterpart is the exact binomial test (binom.test in R). For example:

• Do booster sessions of cognitive therapy lower the chance of a depression relapse?
• Among 28 students with test anxiety, are those who tried expressive writing before the exam more likely to pass it?
• Of 32 volunteers, are those told to count basketball passes less likely to notice the gorilla in the video?

• How big. The test says a split this uneven would be rare if boosters made no difference; the interval (an odds ratio anywhere from 1.27 to 118.36) says the data can't pin down how big the benefit is.
• Whether it matters. Significance doesn't make one relapse in 15 versus seven in 15 a dependable clinical benefit. That takes a larger trial and a judgment about cost and burden.
• Who else. Thirty volunteers from one pilot represent people like them, recently recovered and willing to join a trial, not everyone with depression.
• That there's no effect, if p had been above .05. With 15 per group, even a sizable real benefit often comes out non-significant (see Power and Sample Size below), so a non-significant pilot would have been inconclusive.

The odds ratio compares one group's odds of an outcome with another's, and it's what logistic regression reports. It doesn't depend directly on the total sample size, but its value depends on which group and which outcome are the reference: flip either and it becomes its reciprocal (12.25 becomes 0.08). Odds ratios from other studies are comparable only when those definitions and the populations are, and different baseline risks can still complicate the comparison.

Usual care's odds of relapse, 7 to 8, are about 12 times the boosters' 1 to 14. Read the other way, boosters cut the odds of relapse to 0.08 of usual care's.

Odds aren't chances. Relapse was 7 times as likely under usual care (47% vs. 7%), not 12 times. When an outcome is common, the odds ratio sits further from 1 than the ratio of the percentages, so say “odds” and give the percentages too.

Zero cells. A single zero cell, with the other counts positive, makes the odds ratio 0 or infinite; more than one zero can leave it undefined. If the one booster patient who relapsed had stayed well, the boosters' odds would be 0 and the odds ratio infinite, with no log-scale interval. Fisher's p still works (it would be .006). A common fix for the estimate is to add 0.5 to every cell (Haldane, 1956), which would give 27.35 here; say so if you do.

Which interval. The headline interval uses Woolf's method because it can be worked by hand, but it's a large-sample approximation, which sits awkwardly on a page about avoiding them: with a cell holding a single patient it's especially imprecise and its coverage may be unreliable. R also prints a different point estimate, the conditional maximum-likelihood odds ratio (11.26), not the sample odds ratio (12.25). The exact (conditional) interval R prints, [1.13, 588.26], uses the same hypergeometric probabilities as Fisher's test, but it's built from two one-sided tests, so it doesn't always agree with the two-sided p: with small or unequal groups it can include 1 while p is below .05, or the reverse. Here every interval excludes 1. Report the interval you planned to, and if it disagrees with p, say so.

Odds ratios have no generally accepted small, medium and large cutoffs. The ones sometimes quoted (1.68, 3.47 and 6.71; Chen, Cohen & Chen, 2010) were matched to Cohen's d for an outcome that 1% of the comparison group has, so they don't carry over to a common outcome like relapse. An odds ratio further from 1, counted multiplicatively (4 and 0.25 are equally far, in opposite directions), means a stronger difference in odds, not necessarily a bigger difference in risk: here 12 times the odds is 7 times the risk. The percentages, 7% against 47%, are the clearer guide to size. Whether cutting relapse from about half to 1 in 15 is worth monthly sessions is a clinical question, and with 30 patients the interval says the true benefit could be far smaller.

p answers one narrow question: if group and outcome were unrelated, and the marginal totals were fixed as observed, how often would the counts split at least this unevenly? Fisher's test is an exact test: it answers the question by listing every table with those totals and adding up their probabilities from the hypergeometric distribution, instead of approximating with the chi-square distribution.

With 15 patients per group and 8 relapses in all, the relapse study could have produced 9 tables:

The four tables no more likely than ours add up to .035. If boosters made no difference, the 8 relapses would split between the groups at least this unevenly (in either direction) about 1 time in 28. That's below .05, so it's significant.

Which two-sided p. This page, R, SPSS and scipy all use the same rule: count every table whose probability is no bigger than the observed table's. Other rules exist and can disagree when the distribution is lopsided (Agresti, 1992). Here the groups are the same size, so the distribution is symmetric and the two-sided p is exactly twice the one-sided .018.

A cautious test. Only 9 tables are possible, so p can only take a handful of values, and the real chance of a false positive is usually below .05. That makes Fisher's test conservative: safe, but a little less likely to catch a real effect. The ordinary chi-square test on the same table gives p = .013, and the “N − 1” version, which holds its error rate well in 2 × 2 tables with expected counts as small as 1 (Campbell, 2007), gives .015. With expected counts of 4, either is defensible; the point is to pick one test before seeing the results.

p is not the probability that the null hypothesis is true, and it says nothing about how big the benefit is. That's what the odds ratio and its interval are for.

Power is the chance that a study detects an effect that's really there. For Fisher's test it depends on the two true proportions (not just the odds ratio between them), the group sizes and α. Because the test is exact and cautious, its power is lower than a chi-square or z-test formula suggests, and in small samples the gap is large: in the scenario below, the usual formula promises 58% power with 15 per group, where Fisher's test has 43%. There's no simple formula of its own: power comes from working through every outcome the study could produce, or from simulation.

Suppose boosters truly cut relapse from 45% to 10%. With 15 per group, as in our pilot, Fisher's test would come out significant only about 43% of the time. It takes about 29 per group to reach 80%.

Researchers usually aim for at least 80% power. Plan it before collecting data, from the two proportions you expect and the smallest difference that would matter. For small samples, compute power for Fisher's test itself rather than borrowing a large-sample formula. Its steps are uneven: adding a person can even lower power slightly (with these rates, 25 per group has a little less than 24). There's no calculator for this test yet.