Distribution Fitting: Interpretation and APA Reporting

The Basics

When to Use It

Three questions about a distribution are easy to blur. What does it look like? Describe it with a histogram, skewness and a Q-Q plot. Can we reject a named distribution? That is a goodness-of-fit test, such as Shapiro-Wilk, and the main subject of this page. Which distribution should we use? Compare candidate shapes, which the end of Effect Size covers briefly.

Use it when you have one continuous variable, or one set of residuals, and want to know whether its scores could have come from a named distribution, usually the normal curve. Pair the test with a Q-Q plot and a judgment about whether any departure matters for what you'll do next. For example:

Are people's average reaction times on a simple task normally distributed?
Are the residuals from a regression predicting depression scores from sleep roughly normal?
Are scores on a new anxiety questionnaire roughly normal in a community sample, as its norms assume?

This page shows the normal curve. For counts, such as panic attacks per month checked against a Poisson distribution, the same observed-versus-expected idea is Pearson's chi-square goodness-of-fit test (see Related Methods).

Vocabulary You Need

Definition Normal Distribution

A symmetric, bell-shaped distribution fixed entirely by its mean and standard deviation: about 68% of scores fall within one standard deviation of the mean and about 95% within two, and the mean, median and mode coincide. Many tests assume that something is normal, usually the residuals or the sampling distribution of a mean, not every variable in the study. Real measures are never exactly normal; the useful question is whether they are close enough for the analysis at hand. Also called the Gaussian distribution or the bell curve. Glossary →

Definition Q-Q Plot

A quantile-quantile plot. Sort the scores, and pair each with the value the named distribution (usually the normal) expects for a score in that position, then plot the pairs. If the data follow the distribution, the points lie close to a straight line. The shape of any bend is informative, but the picture depends on two conventions. R, Python and this site put the expected values across the bottom, while SPSS puts them up the side; and programs draw the reference line differently (through the quartiles, by least squares, or from the mean and SD). So rather than memorize bend shapes, compare the sorted scores with the expected values directly. Where the scores run further from the center than expected, that tail is longer than a normal curve's; where they fall short, it is shorter. Right skew shows as the highest scores running further out than expected and the lowest falling short. Both tails running further out means heavier tails than normal, and a few points far from the rest at either end are possible outliers. With small samples some wiggle is normal, so look for a clear, systematic bend. Glossary →

Definition Shapiro-Wilk W (

W

)

The test statistic of the Shapiro-Wilk test for normality. It compares two measures of spread: one built from the sorted scores, weighted as they would be in a normal sample, and their ordinary variance. Shapiro and Wilk derived the weights from the expected normal scores and from how the sorted values of a normal sample vary together; software uses Royston's close approximation of them. The statistic can be written as the squared correlation between the sorted scores and those weights. The related Shapiro-Francia statistic uses the expected normal scores themselves instead. The two are closely related but need not agree, especially for non-normal samples. Both are near 1 when the normal Q-Q plot is nearly straight and fall as it bends. It is 1 for a perfectly straight plot and falls as the data depart from normality. Even normal samples give values below 1, and how far below depends on n (in simulated normal samples the median W is about .93 with 10 scores but about .98 with 60), so W is read through its p-value, which depends on the sample size, not against a fixed cutoff. Glossary →

Definition Goodness of Fit

How closely observed data match what a named distribution or model predicts. A goodness-of-fit test turns the mismatch into one number and asks whether it is bigger than chance would produce if the data really came from that distribution. The null hypothesis is that the fit is right, so a significant result says the data don't fit, and a non-significant one only says the test couldn't tell. Normality tests and the chi-square goodness-of-fit test for counts are the common examples. Glossary →

Assumptions

1. Independent Observations The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: A lot. Related scores, such as several trials from the same person entered as separate rows, make the sample look more informative than it is, and the p-value comes out wrong.

If It's Violated

Design Change Summarize each person's trials into one score (a mean or median), or test the residuals of a model that accounts for the clustering.
Alternative Test Multilevel Regression (HLM). Fit a model with a random effect for person and check its residuals instead.

2. Continuous, Finely Measured Scores Your scores can take many different values, not just a handful.

What It Means: Normality tests compare the data with a smooth curve, so they expect a continuous measure recorded finely enough that few people share exactly the same score. Heavily rounded scores or short rating scales produce many ties and stair-step Q-Q plots, and a large sample of them can fail a normality test even when the underlying quantity is roughly normal.
Applies To: Observations
How to Check: Count the distinct values and look for long runs of identical scores in the sorted data or flat steps in the Q-Q plot. A 1–5 rating is not a continuous measure however many people answer it.
How Much It Matters: Moderate. A few ties barely matter. Many ties, as with scores rounded to a handful of values, can make the test reject normality because of the rounding rather than the shape, and more so as n grows.

If It's Violated

Proceed with Caveat Look at the histogram and Q-Q plot and ask whether the shape, apart from the steps, is close enough for your analysis.
Design Change Keep the measurement's full precision (milliseconds rather than tenths of a second, or a total rather than a single item).

Limitations

This Is Not Your Test If…

You want to describe the shape, not test it Descriptive Statistics →
Your data are counts in categories Pearson Chi-Square Test →
You want the full set of regression or ANOVA diagnostics Linear Regression →
You need a group comparison that doesn't assume normality Nonparametric Group Comparisons →

What It Doesn't Tell You

That the data are normal. The log-scale p of .500 means the test found no clear departure in 60 people, not that log reaction times are exactly normal. No real measure is.
Whether it matters. The skew here (0.82) was clear enough for the test to reject normality, but whether it would distort a t-test or regression on these scores depends on the analysis and the sample size, not on this p.
Why. The test doesn't say whether the long tail comes from a few distracted participants, a real slow subgroup, or the way reaction times always behave.
Which distribution is right. Rejecting the normal doesn't identify the true shape, and passing on the log scale doesn't prove the scores are lognormal; other skewed shapes would pass too.

Common Mistakes

Don'tUse a normality test as a gatekeeper that decides between a t-test and a nonparametric test.

InsteadChoose the analysis from the design, earlier data and the plots, and report a sensitivity analysis if in doubt.
Don'tRead a non-significant normality test as proof that the data are normal.

InsteadSay the test found no clear departure, and show the Q-Q plot. Small samples often pass even when the shape is clearly off.
Don'tTest the raw outcome when the analysis assumes normal residuals.

InsteadTest (or better, plot) the residuals, or each group's scores separately in a group comparison.
Don'tTransform the data, retest, and report only the version that passed.

InsteadChoose a transformation for a reason (log for reaction times) and interpret the results on that scale.
Don'tName the skew after the side where the scores pile up.

InsteadName it after the long tail: a tail stretching toward high scores is positive (right) skew.
Don'tDescribe high kurtosis as a tall, pointy peak.

InsteadDescribe it as how prone the scores are to extreme values; high kurtosis usually comes from a few unusually extreme scores (Westfall, 2014).
Don'tUse the plain Kolmogorov-Smirnov p-value after estimating the mean and SD from the same data.

InsteadUse the Lilliefors-corrected p-value (SPSS's Explore does) or the Shapiro-Wilk test.
Don'tRead a p-value as the chance that the null hypothesis is true.

InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance Guidelines

Benchmarks Significance (p Value)

p < .001
Very strong evidence against the null hypothesis.
.001 ≤ p < .01
Strong evidence against the null hypothesis.
.01 ≤ p < .05
Statistically significant at α = .05. Moderate evidence against the null hypothesis.
.05 ≤ p < .10
Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.
p ≥ .10
Not significant. Little evidence against the null, which is not the same as evidence of no effect.

Here the null hypothesis is that the population is normal, so a small p is evidence that it isn't, not evidence of an effect. A large p doesn't show the data are normal; with a small sample the test can miss a real departure, and with a large one it flags trivial departures. α is chosen before the analysis.

More about statistical significance ↓

How It's Reported

Equation Form

Example

skewness = 0.82, W = .94, p = .008; log scale: skewness = 0.08, W = .98, p = .500

Table 1. Shape and Normality Tests for Reaction Time, Raw and Log-Transformed

Example

	Skewness	Kurtosis	$D$	$W$	$p$
Milliseconds	0.82	0.54	.13	.94	.008
Log milliseconds	0.08	0.37	.08	.98	.500

Note. N = 60. Skewness and kurtosis are the bias-adjusted values SPSS reports (excess kurtosis, 0 for a normal curve); each has a standard error of about 0.31 and 0.61 at this sample size. D is the Kolmogorov-Smirnov statistic with the mean and SD estimated from the data (Lilliefors); its p-value differs a little between programs: .010 to .015 for the raw scores and above .20 for the log scores (see Run It in Software). p is for the Shapiro-Wilk test.

Full Text (APA Style)

Example

Because reaction times are characteristically right-skewed, we planned before collecting data to analyze them in log milliseconds. The raw scores were right-skewed (M = 310.62 ms, Mdn = 288.50 ms, SD = 79.78; skewness = 0.82), on the normal Q-Q plot the slowest scores ran further out than a normal curve predicts, and a Shapiro-Wilk test rejected normality, W = .94, p = .008. The log-transformed scores were close to symmetric (skewness = 0.08), and the Shapiro-Wilk test found no clear departure from normality, W = .98, p = .500, consistent with that choice.

Describe the shape in words and numbers (skewness, the Q-Q plot) before giving the test, and say which test you used.
Say what you tested: the raw scores, each group's scores, or a model's residuals.
Never write that the data "were normal." A non-significant result means no clear departure was found.
If you transformed the data, say so, say why, and report results on the scale you analyzed. A mean of log scores, transformed back, is the geometric mean: here the mean log of 5.708 is about 301 ms, near the median rather than the mean. Differences in log units are ratios: 0.10 log units is about 10% slower.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how surprising.

skewness = 0.82	Clearly right-skewed: a longer tail of slow participants than of fast ones. The standard error at n = 60 is about 0.31, so 0.82 is more than two and a half standard errors from the 0 of a normal curve: more lopsided than a normal sample of 60 usually comes out. Dividing by the standard error makes this a test, which flags smaller skews as samples grow; the size of the skew is 0.82 itself.
D = .13	At its worst point, the share of people at or below a reaction time is 13 percentage points off what a normal curve predicts. Judged with the Lilliefors correction, that is a significant misfit (p between .010 and .015, depending on the program).
W = .94	A W of .94 means the sorted reaction times line up closely, but not perfectly, with a normal curve's expected values. That sounds high, but samples of 60 from a truly normal population almost always come out higher, which is what the p-value checks.
p = .008	If reaction times in the population were normal, a sample of 60 at least this far from normal (as W measures it) would turn up less than 1% of the time. That's below .05, so the test rejects normality. It says nothing about whether the skew is big enough to matter for a later analysis. Benchmarks ↑

Calculations

The Running Example

Example

The reaction-time study: 60 adults, one average reaction time each, checked against a normal distribution, then checked again on the log scale.

Null Hypothesis:: The population of reaction times is normal
Alternative:: The population of reaction times is not normal

Here the null hypothesis is “normal,” so a non-significant result can't confirm it; it only fails to reject it. The alternative has no direction, since any departure (skew, heavy tails, gaps) counts. Only a W below what normal samples produce counts against the null, so the p-value comes from one tail of W's distribution, but that isn't a one-sided test in the usual sense: there is no direction to predict.

What Your Data Should Look Like

One row per person and one numeric column holding the scores to check. If the analysis you're preparing for assumes normal residuals, save the residuals from that model as a new column and test that column instead of the raw outcome. For a group comparison, check each group's scores separately. With one group and nothing predicting the scores, as in the reaction-time study, each residual is just a score minus the mean, so testing the scores is the same as testing the residuals.

Example

participant	rt_ms
1	212
2	352
3	331
4	336
5	274
6	356
… 54 more rows

	n	M	SD	Mdn	Min–Max	Skewness
All	60	310.62	79.78	288.50	147–524	0.82

Skewness is the bias-adjusted G₁: 0 for a symmetric distribution, positive when the long tail is on the right.

Simulated data.

Download all 60 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the reaction-time study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Describe the Scores' Center and Spread

A normal curve is fixed by just two numbers, its mean and standard deviation. We take both from the data, and every later step compares the scores with the normal curve that has them.

$s = \sqrt{\dfrac{\sum (x_i - \bar{x})^2}{n - 1}}$

= 79.78 ms

$\bar{x}$ = 310.62 ms

$\bar{x}$ is our 60 participants' average, 310.62 ms. Each person's distance from it, squared, added up, divided by $n - 1 = 59$ and square-rooted gives $s$ = 79.78 ms.
2. Measure How Lopsided the Scores Are

A normal curve is symmetric, so its mean and median match and its skewness is 0. A long tail of slow responders pulls the mean above the median and makes skewness positive.

$G_1 = \dfrac{n}{(n - 1)(n - 2)} \sum z_i^3$

= 0.82

$Mdn$ = 288.50 ms
$\text{kurtosis}$ = 0.54

Cubing each distance from 310.62 ms keeps its sign, so the few slow participants (up to 524 ms) outweigh the fast ones. The mean sits 22 ms above the median of 288.50 ms, and $G_1$ = 0.82.
3. Line Each Score Up with Where a Normal Curve Would Put It

Sort the scores. If they were normal, the smallest, the second smallest and so on would sit at predictable distances from the mean. Plotting each actual score against its predicted place is the Q-Q plot.

$m_i = \Phi^{-1}\!\left(\dfrac{i - 3/8}{n + 1/4}\right)$ = $\Phi^{-1}\!\left(\dfrac{60 - 3/8}{60 + 1/4}\right) = \Phi^{-1}(0.990)$

= 2.31

$z_{(60)}$ = 2.67

For our slowest participant, $i$ = 60 of $n$ = 60, so a normal sample would put them 2.31 SDs above the mean. They're actually (524 − 310.62) ÷ 79.78 = 2.67 SDs above it. Points that run further out than predicted at the slow end make the Q-Q plot bend upward.
4. Find the Biggest Gap between Our Scores and the Normal Curve

At every score, compare the share of people at or below it with the share the normal curve predicts. The Kolmogorov-Smirnov test uses only the single biggest gap. The normal curve here uses our mean and SD, with the SD dividing by n − 1 as SPSS, R's nortest and statsmodels do. A maximum-likelihood fit divides by n instead, which gives a slightly different D (.1333 rather than .1330 here). That choice is separate from the Lilliefors correction, which adjusts the p-value for having estimated the mean and SD at all.

$D = \max_i \max\!\left\{ \dfrac{i}{n} - \Phi(z_{(i)}),\; \Phi(z_{(i)}) - \dfrac{i - 1}{n} \right\}, \quad z_{(i)} = \dfrac{x_{(i)} - \bar{x}}{s}$ = $\tfrac{36}{60} - \Phi\!\left(\tfrac{304 - 310.62}{79.78}\right) = 0.600 - 0.467$

= .13

$F_n(304)$ = .600
$\Phi$ = .467

The biggest gap is at 304 ms, our 36th score. $F_n$ = 36 of 60 people, .600, were at or below it, but the normal curve $\Phi$ puts only .467 there, because most people bunch below the mean. The gap is .133. It's the first kind of gap in the formula (the sample ahead of the curve); the largest gap of the other kind, just below a score, is smaller here.
5. Measure How Straight the Q-Q Plot Is

The Shapiro-Wilk statistic uses every point, not just the worst one. It compares the spread implied by the sorted scores, weighted the way a normal sample would be, with their ordinary spread. That can be written as the squared correlation between the sorted scores and the weights, so W is near 1 when the Q-Q plot is nearly straight.

$W = \dfrac{\left(\sum a_i\, x_{(i)}\right)^2}{\sum (x_i - \bar{x})^2}$

= .94

$r^2_{Q\text{-}Q} \text{ (Shapiro-Francia)}$ = .944

The weights $a_i$ come from the expected normal scores, with the two most extreme pairs adjusted, and software computes them. Correlating our sorted scores with the 60 expected normal scores themselves, the points of the Q-Q plot, gives a closely related statistic, Shapiro-Francia's $r^2$ = .944. $W$ also comes out at .944, but the two are different calculations that agree closely here, not one statistic. A perfectly straight plot would give 1 on either.
6. Ask How Surprising That Bend Would Be If the Population Were Normal

This step is software's: Royston's formula for samples of 12 or more turns W into a z-score using how W behaves in normal samples of this size. A small W gives a large z, and only that upper tail counts.

$z = \dfrac{\ln(1 - W) - \mu_n}{\sigma_n}, \quad p = P(Z \ge z)$ = $P\!\left(Z \ge \dfrac{\ln(1 - 0.9442) - (-3.996)}{0.464}\right) = P(Z \ge 2.39)$

= .008

$z$ = 2.39

For normal samples of $n$ = 60, $\ln(1 - W)$ averages $\mu_n$ = −3.996 with an SD of $\sigma_n$ = 0.464. Ours is −2.887, which is $z$ = 2.39 SDs higher, and $P(Z \ge 2.39)$ = .008, about 1 in 119.
7. Check Again on the Log Scale

Right-skewed measures such as reaction times often become roughly normal after a log transformation, which pulls in the long right tail more than the short left one. Repeat the check on the transformed scores.

$y_i = \ln(x_i)$ = $\ln(212) = 5.357, \; \ln(352) = 5.864, \; \ldots$

$\text{skewness}$ = 0.08
$W$ = .98
$p$ = .500

The log squeezes the slow end. A 124 ms gap near the top (400 to 524 ms) becomes 0.27 log units, less than half the 0.60 the same gap becomes near the bottom (150 to 274 ms). On that scale skewness drops to 0.08, $W$ rises to .98, and $p$ = .500, no clear departure from normality.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

Analyze → Descriptive Statistics → Explore.
Move rt_ms into Dependent List.
Click Plots, check Normality plots with tests (and Histogram), and click Continue, then OK.
For the log scale: Transform → Compute Variable, name it log_rt with the expression LN(rt_ms), then run Explore again on log_rt.

Or paste this syntax:

EXAMINE VARIABLES=rt_ms
  /PLOT HISTOGRAM NPPLOT
  /STATISTICS DESCRIPTIVES
  /MISSING LISTWISE
  /NOTOTAL.

COMPUTE log_rt = LN(rt_ms).
EXECUTE.
EXAMINE VARIABLES=log_rt
  /PLOT HISTOGRAM NPPLOT
  /STATISTICS DESCRIPTIVES
  /NOTOTAL.

Reading the Output

Look at the Normal Q-Q Plot and histogram first, then Skewness and Kurtosis in the Descriptives table, then the Tests of Normality table. Read the Shapiro-Wilk columns; the Kolmogorov-Smirnov columns already carry the Lilliefors correction.

Descriptives (rt_ms, selected rows)

	Statistic	Std. Error
Mean	310.62	10.300
Median	288.50
Std. Deviation	79.780
Skewness	.821	.309
Kurtosis	.538	.608

Tests of Normality

	Kolmogorov-Smirnov Statistic	df	Sig.	Shapiro-Wilk Statistic	df	Sig.
rt_ms	.133	60	.010	.944	60	.008
log_rt	.085	60	.200*	.982	60	.500

a. Lilliefors Significance Correction. *. This is a lower bound of the true significance.

SPSS puts your scores on the horizontal axis and the normal curve's expected values on the vertical one, the opposite of R, Python and the chart on this page, and draws its line from the mean and SD, so the same scores make a different picture. In the Normal Q-Q Plot of rt_ms, the slowest participants sit below the line at the right end, the fastest dip below it at the left, and the middle rises above it, so the points form an arch. The plot of log_rt follows the line, apart from the single fastest participant, whom the log pushes a little further out at the low end. SPSS's "Detrended Normal Q-Q Plot" shows the same thing as distances from the line.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

rt <- read.csv("data.csv")
shapiro.test(rt$rt_ms)
nortest::lillie.test(rt$rt_ms)   # K-S with the Lilliefors correction

qqnorm(rt$rt_ms); qqline(rt$rt_ms)

rt$log_rt <- log(rt$rt_ms)
shapiro.test(rt$log_rt)

	Shapiro-Wilk normality test

data:  rt$rt_ms
W = 0.94423, p-value = 0.008414

	Lilliefors (Kolmogorov-Smirnov) normality test

data:  rt$rt_ms
D = 0.13305, p-value = 0.01002

	Shapiro-Wilk normality test

data:  rt$log_rt
W = 0.9816, p-value = 0.4998

lillie.test is in the nortest package. Don't use ks.test(x, "pnorm", mean(x), sd(x)): it treats the mean and SD as known in advance and gives a p-value that is too large. shapiro.test accepts 3 to 5000 scores; above that, a test will flag almost any departure anyway, so rely on the Q-Q plot.

Output from a real run: R 4.6.1 (nortest 1.0.4), run 2026-09-29.

Python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats
from statsmodels.stats.diagnostic import lilliefors

rt = pd.read_csv("data.csv")

stats.shapiro(rt["rt_ms"])
# ShapiroResult(statistic=0.9442321951975545, pvalue=0.008413718086650694)

lilliefors(rt["rt_ms"], dist="norm")
# (0.13304885459187438, 0.015256240029405812)

stats.skew(rt["rt_ms"], bias=False), stats.kurtosis(rt["rt_ms"], bias=False)
# (0.8214441372760163, 0.5378020784691575)

stats.shapiro(np.log(rt["rt_ms"]))
# ShapiroResult(statistic=0.9815998588821956, pvalue=0.4998430061354107)

stats.probplot(rt["rt_ms"], dist="norm", plot=plt)   # the Q-Q plot
plt.show()

statsmodels' lilliefors looks its p-value up in a simulated table by default (.015); pvalmethod="approx" uses the Dallal-Wilkinson formula instead and gives .010, the value R's nortest shows. Both agree the fit is poor. Don't use stats.kstest(x, "norm", args=(x.mean(), x.std())) for this: it assumes the mean and SD were known in advance.

Output from a real run: Python 3.9.6 (scipy 1.13.1, statsmodels 0.14.6, pandas 2.3.3), run 2026-09-29; the Q-Q plot lines need matplotlib.

Impact

Effect Size

A normality test has no standard effect size. W and D both say how far the data are from normal, but neither has agreed benchmarks, and the same W means a smaller departure in a small sample than in a large one. The closest thing to a size is the shape itself: skewness and kurtosis, and above all the Q-Q plot.

G_1 = 0.82 \quad \text{(raw)}, \qquad G_1 = 0.08 \quad \text{(log scale)}

Reaction times had a clearly longer slow tail than fast one: the mean sat 22 ms above the median, and skewness was 0.82. On the log scale the skew all but disappeared, 0.08.

Whether a skew that size matters is a question about the analysis that comes next. For comparing means, moderate skew matters less and less as the sample grows (Ghasemi & Zahediasl, 2012), though heavy tails can matter more than skew (Field & Wilcox, 2017); for predicting individual scores, or for tail-sensitive measures such as the proportion of very slow responders, it can matter a lot.

If it does matter, there are four ways out. Transform the scores (log, or the reciprocal, for reaction times) and interpret the results on that scale; this changes the question, since comparing mean logs compares geometric means rather than ordinary ones. Keep the scale but use a method that leans less on normality, such as a bootstrapped confidence interval or a trimmed mean (Field & Wilcox, 2017). Switch to a rank-based test. Or model the skewed shape directly, for example with a regression whose outcome follows a gamma distribution (Lo & Andrews, 2015).

To see which skewed shape fits best, such as the lognormal (normal once you take logs) or the gamma, let software find each shape's best-fitting version (by maximum likelihood) and compare them with AIC, where lower means a better fit after a penalty for extra parameters, along with a Q-Q plot for each. In R: fitdistrplus::fitdist(rt$rt_ms, "lnorm") and "gamma", then gofstat(); in Python, scipy.stats.lognorm.fit(). gofstat() also prints the Anderson-Darling statistic for each shape, a goodness-of-fit test that came second only to Shapiro-Wilk for normality in Razali and Wah's (2011) power comparison.

Statistical Significance

p answers one narrow question: if the population really were normal, how often would a sample of this size look at least this far from normal? The null hypothesis here is the comfortable one, so a small p is evidence of non-normality, and a large p is only an absence of evidence.

If reaction times in the population were normal, a sample of 60 would look at least this far from normal (a W this low or lower) about 0.8% of the time, so the test rejects normality (p = .008). On the log scale, p = .500: no sign of a departure, which is not the same as proof of normality.

The catch is that p tracks sample size as much as shape. With small samples, normality tests often have low power against moderate departures, so many non-normal samples pass; how often depends on the kind of departure, and a severe one can still be caught (Ghasemi & Zahediasl, 2012; Razali & Wah, 2011). With samples in the hundreds or thousands, a departure far too small to affect any analysis comes out significant. And the tests most analyses lean on, t-tests and ANOVA, become more tolerant of non-normality as samples grow, exactly when normality tests become harsher. That's why the plot and the analysis you're planning should decide, not p: preliminary normality testing to choose a test is formally incorrect, because it distorts the error rate of whichever test it picks, even though in their simulations the two-stage procedure's overall error rate stayed acceptable (Rochon, Gondan & Kieser, 2012), and the usual assumption is about the residuals, not the raw variable (Ernst & Albers, 2017).

p is not the probability that the data are normal, and it says nothing about how far from normal they are.

Benchmarks p at a Glance

Very Strong

p < .001

Strong

.001 ≤ p < .01

Significant

.01 ≤ p < .05

Weak

.05 ≤ p < .10

Little

p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. For a normality test, the "effect" is a departure from normality, and power depends on the sample size, α, the test and the kind of departure. Skew is easier to detect than slightly heavy tails.

Razali and Wah (2011) simulated samples from a gamma distribution with skewness 1.00, a little more lopsided than our reaction times. At α = .05, Shapiro-Wilk caught it 44% of the time with 30 people, 69% with 50 and 96% with 100. The Lilliefors-corrected Kolmogorov-Smirnov test caught it only 25%, 40% and 70% of the time.

That is the whole problem with normality tests as gatekeepers. The test is weakest in small samples, where some classical procedures are more sensitive to non-normality, and strongest in large samples, where many of them are more tolerant. Whether a departure matters depends on the planned analysis and the kind of departure, not on sample size alone: a few extreme scores can distort a mean at any n. There is no calculator here, because the right question isn't how many people you need to detect non-normality; it's whether your planned analysis is robust to the departures your measure is known to have. Decide that before collecting data, from earlier studies with the same measure (for reaction times, plan on right skew).

Other Info

Follow-Up and Post Hoc Tests

None needed. This test answers a single question, so there are no planned contrasts or post hoc tests to run.

Related Methods

Related DesignPearson Chi-Square Test: Pearson's chi-square goodness-of-fit test is the same idea for counts: compare the observed count in each category with the count a named distribution predicts. The chi-square test of independence applies the same observed-versus-expected logic to two variables.
Related DesignDescriptive Statistics: Skewness, kurtosis, the histogram and the Q-Q plot describe a distribution's shape; distribution fitting adds a test of whether that shape could have come from a named distribution.
Related DesignOne-Sample t-Test: The exact small-sample p-values of t-tests, ANOVA and regression assume normally distributed residuals (for a one-sample t-test, the scores themselves), which is the usual reason psychologists check a distribution. With larger samples their p-values stay close to right without it, under conditions such as independent observations and no extreme outliers. With moderate samples they tolerate moderate skew (Ghasemi & Zahediasl, 2012), so a failed normality test isn't by itself a reason to abandon them.
AlternativeMultilevel Regression (HLM): An alternative if “Independent Observations” doesn't hold. Fit a model with a random effect for person and check its residuals instead.

References

Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3–4), 591–611. https://doi.org/10.1093/biomet/52.3-4.591
Royston, P. (1995). Remark AS R94: A remark on Algorithm AS 181: The W-test for normality. Applied Statistics, 44(4), 547–551. https://doi.org/10.2307/2986146
Lilliefors, H. W. (1967). On the Kolmogorov-Smirnov test for normality with mean and variance unknown. Journal of the American Statistical Association, 62(318), 399–402. https://doi.org/10.1080/01621459.1967.10482916
Wilk, M. B., & Gnanadesikan, R. (1968). Probability plotting methods for the analysis of data. Biometrika, 55(1), 1–17. https://doi.org/10.1093/biomet/55.1.1
Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(302), 157–175. https://doi.org/10.1080/14786440009463897
Razali, N. M., & Wah, Y. B. (2011). Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. Journal of Statistical Modeling and Analytics, 2(1), 21–33.
Ghasemi, A., & Zahediasl, S. (2012). Normality tests for statistical analysis: A guide for non-statisticians. International Journal of Endocrinology and Metabolism, 10(2), 486–489. https://doi.org/10.5812/ijem.3505
“For small sample sizes, normality tests have little power to reject the null hypothesis and therefore small samples most often pass normality tests.” (Section 3, Normality Tests)
Rochon, J., Gondan, M., & Kieser, M. (2012). To test or not to test: Preliminary assessment of normality when comparing two independent samples. BMC Medical Research Methodology, 12, 81. https://doi.org/10.1186/1471-2288-12-81
“From a formal perspective, preliminary testing for normality is incorrect and should therefore be avoided.” (Conclusions section (main text))
Field, A. P., & Wilcox, R. R. (2017). Robust statistical methods: A primer for clinical psychology and experimental psychopathology researchers. Behaviour Research and Therapy, 98, 19–38. https://doi.org/10.1016/j.brat.2017.05.013
Lo, S., & Andrews, S. (2015). To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. Frontiers in Psychology, 6, 1171. https://doi.org/10.3389/fpsyg.2015.01171
Ernst, A. F., & Albers, C. J. (2017). Regression assumptions in clinical psychology research practice—a systematic review of common misconceptions. PeerJ, 5, e3323. https://doi.org/10.7717/peerj.3323
“Most commonly, researchers incorrectly assume that Xi, or both Xi and Y, should be normally distributed, rather than the errors of the model.” (p. 3, Common misconceptions about assumptions)
Joanes, D. N., & Gill, C. A. (1998). Comparing measures of sample skewness and kurtosis. Journal of the Royal Statistical Society: Series D (The Statistician), 47(1), 183–189. https://doi.org/10.1111/1467-9884.00122
“Comparisons are made between those measures adopted by well-known statistical computing packages” (Abstract)
Pearson, K. (1895). Contributions to the mathematical theory of evolution. II. Skew variation in homogeneous material. Philosophical Transactions of the Royal Society of London. A, 186, 343–414. https://doi.org/10.1098/rsta.1895.0010
Westfall, P. H. (2014). Kurtosis as peakedness, 1905–2014. R.I.P. The American Statistician, 68(3), 191–195. https://doi.org/10.1080/00031305.2014.917055
“Kurtosis tells you virtually nothing about the shape of the peak - its only unambiguous interpretation is in terms of tail extremity” (Abstract)
Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
“we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)