Stats for
Menu

Categorical Data and Proportions

Binomial Test

The binomial test checks a yes/no proportion against a set value.

The exact binomial test takes a yes/no outcome from one group and tests whether the share answering yes differs from a fixed value, using exact probabilities that hold in small samples.

Example

Can people pick out a liar from four speakers more often than guessing would? Fifty adults each watched four short videos of people describing their weekend, one of whom was lying, and picked the one they thought was lying.

One Sample + Binary Outcome + Comparison Value → Binomial Test
Tests
Population proportion
Observations
Independent
Parametric
No
Headline Statistic
p
Effect Size
Cohen's h
Groups
1

The Basics

When to Use It

Use it when you have one group, a yes/no outcome with one answer per person, and a proportion worth comparing the group's share of yeses against, fixed before you look at the data: the rate guessing would give, a published rate or a norm. Because it computes p from the binomial distribution itself rather than the normal curve, it's an exact test and works at any sample size, including the small samples where a z-test's approximation is rough. For example:

  • Can people pick out a liar from four speakers more often than guessing would?
  • Do more than the 20% of clients a counseling center planned around miss their second session?
  • Do infants shown a puppet helping and a puppet hindering reach for the helper more often than half the time?

The test starts from the null hypothesis: here, that the population's share of yeses equals the comparison proportion. It asks how strongly the data argue against that.

Exact test or z-test? With large samples the two agree closely, and the one-sample z-test is a fine shortcut when the expected numbers of yeses and nos, nπ₀ and n(1 − π₀), are both comfortably large. Prefer the exact test when either is small (say, under 10), when π₀ is near 0 or 1, or simply whenever your software offers it, which is nearly always. In the lie-spotting study the expected 12.5 hits (picks of the liar) isn't small, yet the uncorrected z-test's p (.014) is still noticeably smaller than the exact .021.

Vocabulary You Need

Definition Comparison Proportion (\pi_0)

A proportion set before looking at the data, from the design (the 1 in 4 a guess would get right with four options, if a guesser is equally likely to pick any of them), a published rate or a norm, that the population proportion is compared with. It plays the part a comparison value plays for a mean. Glossary →

Definition Binomial Distribution

The probabilities of getting 0, 1, 2, … up to n “yes” answers when n independent tries each have the same chance π of a yes, like counting heads in n coin flips. Its average is nπ and its SD is √(nπ(1 − π)), largest at π = .5 and shrinking toward 0 and 1. Exact tests for a proportion use it to say how likely each possible count would be if π were the value the null hypothesis names. When nπ and n(1 − π) are both large it looks like a normal curve, which is why z-tests for proportions work in big samples; with small samples or π near 0 or 1 it's lumpy and lopsided. Glossary →

Definition Exact Test

A test that computes its p-value directly, by listing every outcome the study could have produced under the null hypothesis and adding up the probabilities of those at least as extreme as the one observed. Approximate tests, like the chi-square test, instead compare a statistic with a smooth distribution that is only accurate for large samples. Glossary →

Definition Clopper–Pearson Interval

A confidence interval for a population proportion built from binomial probabilities: its ends are the population shares that would make a count like yours, or one further out, just 2.5% likely from each side (for 95%). Equivalently, it holds every value of π that neither one-sided exact test at α/2 would reject. It's “exact” in the sense that it uses binomial probabilities, and as long as the tries are independent with one shared chance of yes, it covers the true proportion at least 95% of the time, but because counts are whole numbers it usually covers more often than that, so it's wider than it needs to be. Score intervals such as Wilson's are narrower and cover close to 95% on average (Agresti & Coull, 1998). Glossary →

Definition Power

The probability that a test comes out significant when a true effect of a given size exists. It depends on the effect size, the sample size, α and whether the test is one- or two-sided. Glossary →

Assumptions and Inference

1. Independent Responses (for the p-Value and Interval) The people (or other units) in your study don't influence each other.
What It Means
Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To
Observations
How to Check
By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters
A lot. The binomial distribution assumes each yes/no answer is a separate, unconnected try. If each person gives several answers (say, 10 lie judgments each), their answers resemble each other, the count varies more than the binomial allows, and p comes out too small. The same happens if people talk to each other between judgments.

What to Do

  • Design Change Collect one yes/no answer from each person, as the lie-spotting study did.
  • Alternative Test One-Sample t-Test. If each person gave several answers, turn each person's answers into their own share correct and test the mean of those shares against the comparison value with a one-sample t-test.
  • Alternative Test Multilevel Logistic Regression. With many answers per person, a multilevel logistic regression models each person's own rate and can add predictors.
2. Same Chance for Every Response (for the p-Value and Interval) The responses share one underlying chance of a yes. For the test, the null sets that chance to the comparison value; the interval considers other possible values.
What It Means
The binomial distribution adds up independent tries that each have the same chance of a yes. With one answer from each of a random sample of people, that holds even if people differ in skill: each response has the population's average chance, and the null says that average is the comparison value. It can fail when the tries differ in a fixed way, such as a set of trials with some much easier than others. Then the count no longer follows the ordinary binomial distribution, and an exact p is exact only under the model.
Applies To
Observations
How to Check
By design. Did each person give one answer, and were people sampled from one population? If responses come from fixed trials of different difficulty, or from the same people answering several times, think about whether one shared chance describes them.
How Much It Matters
Usually little, and in the safe direction. One answer from each of 50 sampled adults, all seeing the same four speakers, gives every response the same chance under the null, even if people differ in skill. If the chance varies in a fixed way across the tries (some much easier than others), the count varies less than the binomial allows, so p tends to come out too large, not too small. Several answers from the same people are the opposite problem; see Independent Responses.

What to Do

  • Design Change Sample people from one population and take one answer from each, as the lie-spotting study did.
  • Alternative Test Multilevel Logistic Regression. If responses come from trials of different difficulty or from the same people, model the trials and people instead of assuming one shared chance.
3. Sample Size Not Set by the Results (for the p-Value) When you stopped collecting data didn't depend on the results.
What It Means
The binomial distribution counts yeses out of a number of tries that doesn't depend on the answers. A sample size set in advance, or one set by something unrelated to the results (everyone who signed up this term), is fine. If you keep adding people until the count looks convincing, or stop as soon as p dips below .05, the real chance of a false positive rises above α, and far above it if you look often, even though each p you computed looked exact.
Applies To
Design
How to Check
By design. Was n set before data collection (ideally preregistered) or by something unrelated to the results, and did you analyze the data once, at the end? If you looked at the results along the way, say so and how often.
How Much It Matters
A lot, in the risky direction. Checking p after every few people and stopping once it falls below .05 can push the real false-positive rate far above 5%. The binomial probabilities themselves stay exact; it's the stopping rule that breaks them.

What to Do

  • Design Change Decide the sample size from a power analysis before collecting data, and analyze once.
  • Correction If you need to look early, plan the looks in advance with a group-sequential design that spends α across them, so the overall false-positive rate stays near α.
  • Proceed with Caveat If you already looked along the way, say how often, and treat p as smaller than it should be.

Limitations

This Is Not Your Test If…

What It Doesn't Tell You

  • Who else. The result describes adults like these 50 judging clips like these. Some clips may simply be easier than others, so a new set of liars and truth-tellers could give a different hit rate.
  • How. It doesn't say what people noticed. They may have read real signs of lying, or something about these particular speakers that happens to go with lying here. Everyone saw the same four speakers, so a hit rate above .25 could come from a cue that happens to go with this liar (looking the most nervous, say), not from spotting lies; students could even share a preference for a kind of speaker, whoever is lying. The study shows that people picked this liar, from this group of speakers, more often than guessing predicts. To establish a general lie-detection ability, researchers would need several independently chosen sets of speakers, with who lies and how each speaker comes across varied or counterbalanced.
  • Whether it matters. Significance doesn't make 40% useful in practice: most picks (60%) were still wrong. Studies where people judge one statement at a time as a lie or the truth find they average 54% correct, where guessing gives 50% (Bond & DePaulo, 2006).
  • That people can't do it, if p had been above .05. A non-significant result would have meant the data couldn't tell the hit rate from guessing, not that it equals .25.

Common Mistakes

  • Don'tSwitch to a one-sided p because the result went the way you hoped.

    InsteadUse a one-sided test only if you specified the direction before collecting data and have a defensible reason why an effect in the opposite direction wouldn't count toward your claim. Otherwise use a two-sided test, as here: doing systematically worse than guessing would be interesting too.

  • Don'tReport SPSS's one-tailed binomial p as a two-sided one when the comparison value isn't .5.

    InsteadReport a two-sided exact p and name the rule: R's and Python's (every count no more likely than ours, .021 here), or the one-sided p doubled (.028), which matches the Clopper–Pearson interval. Pick the rule before you see the data, not after seeing which one gets under .05.

  • Don'tCount every trial as a separate try when each person gave several answers.

    InsteadAnalyze one share per person, or use a model for repeated answers.

  • Don'tPick the comparison proportion after looking at the data.

    InsteadFix it in advance from the design (with four options, .25, if a guesser is equally likely to pick any of them), a published rate or a norm.

  • Don'tReport only p.

    InsteadGive the count and n (20 of 50), the percentage and the interval, so readers can judge the size.

  • Don'tRead a p-value as the chance that the null hypothesis is true.

    InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance and Effect Size Guidelines

Benchmarks Significance (p Value)
  • p < .001

    Very strong evidence against the null hypothesis.

  • .001 ≤ p < .01

    Strong evidence against the null hypothesis.

  • .01 ≤ p < .05

    Statistically significant at α = .05. Moderate evidence against the null hypothesis.

  • .05 ≤ p < .10

    Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.

  • p ≥ .10

    Not significant. Little evidence against the null, which is not the same as evidence of no effect.

These evidence labels are one common convention for describing p in graded language rather than as significant or not; the boundaries are rough guides, not rules, and other authors draw them differently. Muff et al. (2022) argue for this kind of language of evidence over a single cutoff. α is chosen before the analysis, and p never says how big or important an effect is.

More about statistical significance ↓

Benchmarks Effect Size (Cohen's h)
  • |h| < 0.2 Below Small

    Smaller than the conventional small benchmark. Whether an effect this size matters depends on the setting.

  • 0.2 ≤ |h| < 0.5 Small

    A relatively modest standardized difference, by convention.

  • 0.5 ≤ |h| < 0.8 Medium

    A moderate standardized difference, by convention.

  • |h| ≥ 0.8 Large

    A relatively substantial standardized difference, by convention.

The sign shows direction; the benchmarks apply to its size. They are conventions (Cohen, 1988, 1992), not measures of practical or clinical importance. As illustrations only, 50% against 75% is h ≈ 0.52 and 50% against 88% is h ≈ 0.86; whether a difference is noticeable depends on the baseline rate, the sample and the setting.

More about effect size ↓

How It's Reported

Equation Form

Example

20 of 50 (40%) vs. 25% by chance, 95% CI [.26, .55], h = 0.32, exact binomial p = .021

Full Text (APA Style)

Example

Twenty of the 50 participants (40%) picked out the liar, compared with the 25% expected from guessing among four speakers, 95% CI [.26, .55] (Clopper–Pearson), h = 0.32. A two-sided exact binomial test against .25 showed the difference was statistically significant, p = .021.

  • Give the count and the sample size as well as the percentage (20 of 50, 40%), and name the comparison proportion and where it came from (guessing among four speakers).
  • Name the interval method. Clopper–Pearson's is the one R's binom.test prints; with independent answers it never covers less than 95% of the time, at the cost of being wide; Wilson's, here [.28, .54], is narrower and covers closer to 95% on average (Agresti & Coull, 1998).
  • Avoid the textbook interval, \hat{p} \pm 1.96\sqrt{\hat{p}(1 - \hat{p})/n} (the Wald interval): its coverage swings erratically with n and \pi and can fall well below 95%, even in samples the usual rules of thumb call large enough (Brown et al., 2001).
  • Say the p is exact and two-sided. SPSS's legacy Binomial Test prints a one-sided p whenever the test proportion isn't .5 (.014 here); don't report that as two-sided.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

40% (20 of 50) Two in every five participants picked out the liar.
h = 0.32 Small by the benchmarks above. h has no units, so it can be compared across studies with different comparison proportions, but it still depends on the design (here, these clips and one pick each). Whether picking the liar 40% of the time instead of 25% matters is a question about lie detection, not statistics. Benchmarks ↑
95% CI [.26, .55] The true share who would pick out the liar is plausibly anywhere from 26% to 55%. The whole range is above the 25% guessing gives, but its low end barely is. In Cohen's h, the range runs from 0.03 to 0.62, from below small to medium by the benchmarks above.
p = .021 If the population's hit rate were .25, a count at least this unlikely, in either direction (20 or more hits, or 5 or fewer), would turn up about 2.1% of the time. That's below .05, so the result is statistically significant. Benchmarks ↑

Calculations

The Running Example

Example

The lie-spotting study: 50 adults, one pick each from four speakers, compared against the 1 in 4 that guessing would get right.

Null Hypothesis:
\pi = .25
Alternative (Two-Sided):
\pi \neq .25

The null hypothesis is the claim the test looks for evidence against; the alternative is any difference. \pi (“pi”; here a population proportion, not 3.14) is the share of all adults like ours who would pick the liar out of a set like these. The null says \pi = .25, the hit rate of pure guessing with four choices; the sample's .40 estimates \pi. The question points one way (better than guessing), but we test two-sided, the usual default, so a result below chance would count too; a one-sided test is only for a direction predicted before collecting data.

What Your Data Should Look Like

One row per person and one column coding the outcome 1 (yes) or 0 (no). The comparison proportion comes from outside the data: the design, a published rate or a norm. Some software wants only the counts instead: the number of yeses and the total (here 20 and 50).

Example
participantspotted_liar
11
21
31
41
51
60
… 44 more rows
nYes%
All502040.0%

Simulated data.

Download all 50 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the lie-spotting study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

  1. 1. Find the Share Who Picked Out the Liar

    A yes/no outcome boils down to one number, the proportion who said yes. Here a yes is picking the speaker who was lying.

    \hat{p} = \dfrac{x}{n} =\dfrac{20}{50}

    = .40

    x is the 20 participants who picked the liar and n is all 50. 20 ÷ 50 = .40, so 40% picked out the liar. \hat{p} (“p-hat”) is the sample's share; don't confuse it with the p-value, plain p, at the end. The population share it estimates is \pi, and the comparison value is \pi_0.

  2. 2. Work Out What Guessing Alone Would Give

    With four speakers to choose from, someone with no idea who's lying still picks the liar 1 time in 4, but only if a guesser is equally likely to pick any of the four. Shuffling the order of the videos for each participant takes care of position; it can't stop a liar who happens to stand out for some other reason from drawing extra guesses. That 1 in 4 is the comparison proportion, set by the design before anyone was tested.

    E = n\pi_0 =50 \times .25

    = 12.5

    \hat{p} - \pi_0 = .15

    n is our 50 participants and \pi_0 is the .25 chance of picking the liar by guessing. 50 × .25 = 12.5, so guessing alone would pick the liar about 12 or 13 times on average. We saw 20, which as shares is .40 − .25 = .15, 15 percentage points above guessing.

  3. 3. Measure How Far the Share Is from Guessing

    The same 15-point gap is easier to detect when the comparison proportion is near 0 or 1, where proportions vary less, than when it's near .5. Cohen's h stretches the scale near 0 and 1 (an arcsine transformation) so that equal values of h are about equally easy to detect at any baseline (Cohen, 1988).

    h = 2\arcsin\sqrt{\hat{p}} - 2\arcsin\sqrt{\pi_0} =2\arcsin\sqrt{.40} - 2\arcsin\sqrt{.25}

    = 0.32

    2\arcsin\sqrt{\hat{p}} turns our .40 into 1.369, and 2\arcsin\sqrt{\pi_0} turns guessing's .25 into 1.047 (both in radians). 1.369 − 1.047 = 0.32.

  4. 4. Work Out How Likely Each Count Is If People Were Only Guessing

    If the population's hit rate (the share picking the liar) were .25, each participant would pick the liar with chance 1 in 4, independently of everyone else, so the number of hits out of 50 would follow the binomial distribution. Any count from 0 to 50 could happen; counts near 12 or 13 are the most likely.

    P(X = k) = \binom{n}{k}\,\pi_0^{\,k}\,(1 - \pi_0)^{\,n - k} =\binom{50}{20}\,(.25)^{20}\,(.75)^{30}

    \text{counts} = 51

    P(X = 20) = .0077

    \binom{50}{20} counts the ways to choose which 20 of the 50 participants were the ones who picked the liar, about 47 trillion. Each of those ways has chance .25^{20} (20 hits) times .75^{30} (30 misses), a tiny number. Multiplied together, P(X = 20) = .0077, about 1 in 131. Capital X is the number of hits a study like ours could turn up; lowercase x = 20 is the one we got, so P(X = 20) reads “the chance of exactly 20 hits”.

  5. 5. Add Up Every Count at Least as Unlikely as Ours

    p is the chance of a count as surprising as ours or more so. “As surprising” means no more likely than ours, on either side of 12.5. Because the binomial distribution with π₀ = .25 is lopsided, the two sides don't match.

    p = \sum_{P(X = k) \,\le\, P(X = x)} P(X = k) =\underbrace{P(0) + \dots + P(5)}_{.0070} + \underbrace{P(20) + \dots + P(50)}_{.0139}

    = .021

    P(X \ge 20) = .0139

    The counts no more likely than our .0077 are 20 hits or more (.0139 together) and 5 hits or fewer (.0070 together). 6 hits is left out because, at .0123, it's more likely than our 20. .0139 + .0070 = .021, about 1 in 48.

  6. 6. Give a Range of Plausible Values for the True Share

    Instead of a single best guess, give the range of population shares that fit the data. The exact interval uses the same tail sums as the last step, run backwards: instead of fixing the share at .25 and asking how likely 20 or more hits is, it looks for the shares that make a count like ours just 2.5% likely from either side (Clopper & Pearson, 1934).

    P(X \ge x \mid \pi_L) = \dfrac{\alpha}{2}, \quad P(X \le x \mid \pi_U) = \dfrac{\alpha}{2} =P(X \ge 20 \mid \pi_L) = 2.5\%, \quad P(X \le 20 \mid \pi_U) = 2.5\%

    = [.26, .55]

    \text{Wilson} = [.276, .538]

    The bar “\mid” reads “if the population share were”, so P(X \ge 20 \mid \pi_L) is the chance of 20 or more hits if the share were \pi_L. \pi_L is the lowest population share that would still make 20 or more hits out of 50 at least 2.5% likely, and \pi_U is the highest that would make 20 or fewer at least 2.5% likely. Software finds them by trial and error (or from the beta distribution) at .264 and .548, so the range is .26 to .55. Wilson's score interval, which builds the interval from the normal curve using the standard error at each candidate share rather than the sample's, is a little narrower, .276 to .538 (Agresti & Coull, 1998).

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)
  1. Analyze → Nonparametric Tests → Legacy Dialogs → Binomial.
  2. Move spotted_liar into Test Variable List.
  3. Under Define Dichotomy, leave Get from data. Set Test Proportion to .25 and click OK.

The test proportion applies to the first category, which SPSS takes from the first value it meets in the data. The first participant in data.csv picked the liar, so the first category is 1. If your first case is a 0, sort the file so a 1 comes first, or enter 1 − π₀ as the test proportion.

For the Clopper–Pearson interval, use Analyze → Compare Means and Proportions → One-Sample Proportions (SPSS 27 and later): the second command below.

Or paste this syntax:

NPAR TESTS
  /BINOMIAL (.25)=spotted_liar
  /MISSING ANALYSIS.

PROPORTIONS
  /ONESAMPLE spotted_liar TESTVAL=0.25 TESTTYPES=EXACT CITYPES=CLOPPER_PEARSON WILSON_SCORE
  /SUCCESSVALUE=LEVEL(1)
  /CRITERIA CILEVEL=95
  /MISSING SCOPE=ANALYSIS USERMISSING=EXCLUDE.

Reading the Output

Read Observed Prop. first, then Exact Sig. (1-tailed). With a test proportion other than .5, SPSS's legacy Binomial Test reports only a one-sided p, in the direction the data fell: here the chance of 20 or more hits (picks of the liar), .014. Don't report it as two-sided. For a two-sided p, use R or Python below (.021, adding up every count no more likely than 20), or double it (.028, the rule that matches the Clopper–Pearson interval) and say you did. Depending on the version and sample size, the column may read Asymp. Sig. (1-tailed) with a footnote saying "Based on Z Approximation"; check the value against .014.

Binomial Test

Category N Observed Prop. Test Prop. Exact Sig. (1-tailed)
spotted_liar Group 1 1 20 .40 .25 .014
Group 2 0 30 .60
Total 50 1.00

The PROPORTIONS command adds the Clopper–Pearson interval, [.264, .548], and Wilson's, [.276, .538]. Its exact test's two-sided p wasn't checked against SPSS here, so compare it with .021 (or .028) before reporting it. Left to its defaults, PROPORTIONS runs the mid-p and score tests instead of the exact one; TESTTYPES=EXACT asks for the exact test. SPSS doesn't print Cohen's h; compute it as 2 × ARSIN(SQRT(.40)) − 2 × ARSIN(SQRT(.25)) = 0.322. The syntax was written from IBM's NPAR TESTS and PROPORTIONS references, not pasted from SPSS; the dialog's Paste button gives the exact form for your version.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R 
liars <- read.csv("data.csv")
x <- sum(liars$spotted_liar)
n <- nrow(liars)
binom.test(x, n, p = 0.25)

# Cohen's h
2 * asin(sqrt(x / n)) - 2 * asin(sqrt(0.25))

	Exact binomial test

data:  x and n
number of successes = 20, number of trials = 50, p-value = 0.02096
alternative hypothesis: true probability of success is not equal to 0.25
95 percent confidence interval:
 0.2640784 0.5482060
sample estimates:
probability of success
                   0.4

[1] 0.3222409

binom.test gives the two-sided p by adding up every count no more likely than 20 (.021) and the Clopper–Pearson interval. prop.test(x, n, p = 0.25, correct = FALSE) runs the z-test instead (p = .014) with Wilson's interval; leave out correct = FALSE and it adds a continuity correction.

Output from a real run: R 4.6.1, run 2026-10-10.

Python 
import numpy as np
import pandas as pd
from scipy import stats

liars = pd.read_csv("data.csv")
x = int(liars["spotted_liar"].sum())  # 20
n = len(liars)                        # 50

result = stats.binomtest(x, n, p=0.25)
result.pvalue                         # 0.02096
result.proportion_ci(confidence_level=0.95, method="exact")
# ConfidenceInterval(low=0.2641, high=0.5482)   Clopper-Pearson

# Cohen's h
2 * np.arcsin(np.sqrt(x / n)) - 2 * np.arcsin(np.sqrt(0.25))  # 0.3222

scipy's binomtest uses the same two-sided rule as R. method="wilson" gives Wilson's interval, (0.2761, 0.5382).

Output from a real run: Python 3.9.6 (scipy 1.13.1, pandas 2.3.3), run 2026-10-10.

Impact

Effect Size

The plainest size is the gap itself: 40% picked the liar where guessing gives 25%, 15 percentage points above chance. To compare across studies with different comparison proportions, Cohen's h converts both proportions with an arcsine transformation that stretches the scale near 0 and 1, then takes the difference (Cohen, 1988). Equal values of h are about equally easy to detect at any baseline, which is why power tables use it.

h = 2\arcsin\sqrt{\hat{p}} - 2\arcsin\sqrt{\pi_0} = 2\arcsin\sqrt{.40} - 2\arcsin\sqrt{.25} = 0.32

Going from guessing's 25% to 40% is 15 percentage points, and h = 0.32, small by convention. The interval's ends, 26% and 55%, are h = 0.03 and 0.62: anywhere from below small to medium.

Cohen's benchmarks were set for comparing two groups' proportions; for one proportion against a fixed value they're a rough guide. h has no units, but equal h doesn't mean equal importance: going from 1% to 11% is h = 0.48, from 40% to 50% only 0.20. Whether picking the liar 40% of the time instead of 25% matters depends on what the judgment is for, and 60% of picks were still wrong.

Benchmarks Cohen's h at a Glance
Below Small
|h| < 0.2
Small
0.2 ≤ |h| < 0.5
Medium
0.5 ≤ |h| < 0.8
Large
|h| ≥ 0.8

Full benchmarks and caveats ↑

Statistical Significance

p answers one narrow question: if the population's share of yeses were exactly the comparison proportion, how often would a count at least as unlikely as ours turn up, in either direction? The exact binomial test answers it by listing every count that could have happened, 0 to 50 here, working out each one's probability from the binomial distribution, and adding up those no more likely than the observed count. The chart under Exact Test, in Vocabulary You Need, draws all 51. Twenty hits has probability .0077. Every count from 6 to 19 is more likely than that (12, the most likely, has .1294), so only the two ends count: 0 to 5 hits on the low side and 20 to 50 on the high side.

Twenty or more hits has probability .0139, and 5 or fewer .0070, so p = .021. If the population's hit rate were .25, a count at least this unlikely, in either direction, would turn up about 1 time in 48. That's below .05, so it's significant.

Which two-sided p. With π₀ = .25 the distribution is lopsided, so the two tails differ. R's binom.test, scipy's binomtest and this page add up every count no more likely than ours (.021). Doubling the one-sided p gives .028 instead; that's another accepted rule (the "central" one), and it's the one that matches the Clopper–Pearson interval. Either is fine if you choose it before seeing the data and say which you used. Because the usual test and the usual interval follow different rules, they can disagree near .05: the test can reject while the interval still contains π₀ (Fay, 2010). Here both agree. SPSS's legacy Binomial Test prints only a one-sided p (.014) whenever the test proportion isn't .5; don't report that as two-sided.

A cautious test. Because counts come in whole numbers, p can only take certain values, and the real chance of a false positive at α = .05 is usually a little below .05 (4.8% at n = 50 and π₀ = .25). The mid-p, which counts the observed count's own probability only half, is less cautious and is what SPSS's PROPORTIONS command runs by default; its false-positive rate is closer to α on average but can go above it.

The z-test on the same data gives z = 2.45, p = .014, smaller than the exact .021. The normal curve treats the count as continuous and symmetric, and both shrink its right tail here (.007, against the exact .014). A continuity correction, which treats a count of 20 as starting at 19.5, fixes most of that: p = .022 (R's prop.test default).

p is not the probability that the null hypothesis is true, and it says nothing about how big or useful the hit rate is. That's what the proportion, its interval and h are for.

Benchmarks p at a Glance
Very Strong
p < .001
Strong
.001 ≤ p < .01
Significant
.01 ≤ p < .05
Weak
.05 ≤ p < .10
Little
p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. For the binomial test it depends on the true proportion, the comparison proportion, the sample size, α and whether the test is one- or two-sided. Because p comes in steps, power doesn't rise smoothly with n: adding a person can lower it. Exact power adds up, under the true rate, the probabilities of every count the test would call significant.

Suppose, before collecting any data, we'd decided that a true hit rate of 40% against guessing's 25% would matter. With 50 participants, as here, the exact test would come out significant only about 66% of the time. 73 participants first reach 80% power, but 74 drop back to 77%; from 81 on, power stays above 80%. The normal-curve formula, using h = 0.32, says 76.

Researchers usually aim for at least 80% power. Plan it before collecting data, from the smallest true proportion that would matter. For small samples, compute exact power (in R, by running binom.test on every possible count, or with G*Power's exact test for one proportion), and choose an n past the sawtooth rather than on a lucky peak. The calculator below does the exact sum, and gives the n from which power stays at the target. Try it:

Other Info

Follow-Up and Post Hoc Tests

None needed. This test answers a single question, so there are no planned contrasts or post hoc tests to run.

References

  • Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413. https://doi.org/10.1093/biomet/26.4.404
  • Fay, M. P. (2010). Two-sided exact tests and matching confidence intervals for discrete data. The R Journal, 2(1), 53–58. https://doi.org/10.32614/RJ-2010-008
    “This can lead to inconsistencies where the exact test rejects but the exact confidence interval contains the null parameter value.” (Abstract)
  • Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16(2), 101–133. https://doi.org/10.1214/ss/1009213286
  • Bond, C. F., Jr., & DePaulo, B. M. (2006). Accuracy of deception judgments. Personality and Social Psychology Review, 10(3), 214–234. https://doi.org/10.1207/s15327957pspr1003_2
    “people achieve an average of 54% correct lie-truth judgments” (Abstract)
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
  • Agresti, A., & Coull, B. A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. The American Statistician, 52(2), 119–126. https://doi.org/10.1080/00031305.1998.10480550
    “coverage probabilities tend to be too large for “exact” confidence intervals based on inverting the binomial test” (Abstract)
  • Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155
    “To detect a small difference between two independent population proportions (h = .20 in Table 1) at α = .05 requires N = 392 cases in each group.” (p. 158)
  • Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
    “we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)

Spot an error? Something still confusing? Tell us at support@statsforpsychology.com.