Independent-Samples t-Test: Interpretation and APA Reporting

The Basics

When to Use It

Use it when you have two separate groups of different people, one continuous outcome, and you want to know whether the groups' averages differ. For example:

Does a mindfulness app lower college students' scores on the Perceived Stress Scale?
Do students who study with practice tests score higher on the final exam than students who reread?
Do bilingual and monolingual children differ in reaction time on a Stroop task?

Vocabulary You Need

Definition Standard Deviation (

s

)

How spread out individual scores are around their mean. For roughly bell-shaped scores, about two-thirds fall within one standard deviation of the mean. Its square, s², is the variance: the average squared distance from the mean (dividing by n − 1). Glossary →

Definition Standard Error (

SE

)

How much a statistic, such as a mean, a proportion or a difference between two of them, would vary from one sample to the next. SD asks how much individuals vary; SE asks how sure we are about the estimate. It shrinks as the sample grows. Glossary →

Definition Pooled Standard Deviation (

s_p

)

The typical distance of a score from its own group's mean, found by averaging the groups' variances (each weighted by its size minus one) and taking the square root. It's the natural yardstick for a difference between groups, and it makes most sense when the groups are about equally spread out. Glossary →

Definition Degrees of Freedom (

df

)

How much independent information is left for estimating variability, after the quantities estimated from the same data are accounted for. Each estimated quantity uses one up: with a single mean, deviations from it always sum to zero, so once you know n − 1 of them the last is fixed; two group means use up two (N − 2), and a regression with k predictors and an intercept uses k + 1. Degrees of freedom pick which t, F or χ² distribution a result is judged against, and fewer of them mean heavier tails (more extreme values), so a result needs a bigger statistic to count as significant. Some tests, such as Welch's t or a sphericity-corrected F, use fractional values. In a table of counts, it's the number of cells free to vary once the row and column totals are fixed. Glossary →

Assumptions and Inference

1. Independent Students (for the p-Value and Intervals) The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: A lot. Scores that share a cluster within a group (classmates, couples, one therapist's clients) make the standard error too small, so p comes out too small. Random assignment doesn't prevent this: students randomized one by one can still share a class or a therapist. People who appear in both groups break the test differently: their two scores are related, so they need a paired or mixed design instead.

What to Do

Alternative Test Multilevel Regression (HLM). A multilevel model, which accounts for the clustering.
Design Change Make sure each person contributes one score, to one group only.

2. Normality in Each Group (for the p-Value and Intervals) Each group's scores should be reasonably well behaved. With normal scores the t procedures have their strongest footing; bigger groups tolerate moderate departures, but heavy skew or extreme outliers can still matter, so look at each group rather than relying on a sample-size cutoff.

What It Means: The test judges the gap between means against a t distribution, which fits best when each group's scores are roughly normal. Even then Welch's version is an approximation, because it estimates the two spreads separately, but a very accurate one. With other shapes it leans on each mean being roughly normal across repeated samples, which arrives sooner for mild skew than for severe skew or heavy tails. If the scores are clearly skewed and your question really is about means, Welch's test with a bootstrap interval may still do; switching to a rank test or trimmed means changes the question.
Applies To: Residuals
How to Check: Look at a histogram or Q–Q plot of each group's scores for strong skew or extreme outliers. Don't rely on a normality test: with small samples it misses real problems, and with large ones it flags trivial ones. In the stress-app study, both groups look roughly symmetric, with no extreme scores.
How Much It Matters: Check each group's scores separately (or the residuals, each score minus its group's mean): a histogram of everyone together mixes two groups and can look skewed or two-humped when each group is fine. Less as the groups grow. In Fagerland's (2012) simulations of Welch's t-test, with both groups skewed the same way (skewness from 1 to 4) and 25 or more per group, a true null was rejected 4.7% to 5.0% of the time on average, close to the 5% intended. Severe skew in small groups matters more, and extreme outliers pull the means at any size.

What to Do

Robust Estimation Bootstrap a confidence interval for the difference between the means.
Robust Estimation Compare 20% trimmed means with Yuen's test (R: WRS2::yuen), which extreme scores barely move.
Alternative Test Mann-Whitney U Test. It asks whether scores in one group tend to be higher than in the other, rather than comparing means, so the question changes.
Transformation Log-transform right-skewed outcomes such as reaction times. Results are then on the log scale.

3. Equal Variances (Student's Version Only) Student's version assumes the two populations are equally spread out. Welch's version, used on this page, doesn't, which is why it's the safer default. The usual interval for d still leans on it.

What It Means: The outcome varies about as much within each group's population. Tests that pool the groups' variances into one estimate, like Student's t-test and the classic ANOVA, rely on it.
Applies To: Variances
How to Check: Compare the groups' standard deviations and look at side-by-side box plots. Don't use Levene's test as a gatekeeper: it misses real differences in small samples and flags trivial ones in large samples. In the stress-app study, the waitlist's SD (7.00) is about 1.3 times the app's (5.50): a modest difference, which Welch's version allows for anyway.
How Much It Matters: Very little for Welch's version, which this page uses: it keeps each group's own variance (its SD squared). Student's version can give a p that's too small or too large when the variances differ, especially when the groups also differ in size (Delacre, Lakens & Leys, 2017).

What to Do

Correction Use Welch's t-test, as this page does. It's R's default and costs almost nothing when the variances happen to be equal.
Design Change Plan equal group sizes. Student's version is much less thrown by unequal variances when the groups are the same size.

Limitations

This Is Not Your Test If…

Your outcome isn't continuous (yes/no, categories, ranks) Categorical Data and Proportions →
You measured the same people twice, or matched pairs Paired-Samples t-Test →
You have one group and a fixed comparison value One-Sample t-Test →
You're comparing three or more groups ANOVA Family →
Your people come in clusters, like classrooms or clinics Multilevel Regression (HLM) →
Your outcome is ordered ratings, or your question is which group tends to score higher rather than the difference in means Mann-Whitney U Test →

What It Doesn't Tell You

Why. Random assignment lets us credit the app rather than pre-existing differences, but a waitlist can't tell us which part did the work: the meditations, the daily reminders, or simply expecting to feel better.
Whether it matters. Significance doesn't make 3.5 points on a 40-point stress scale something students would notice in their lives.
Who else. The result describes students like these 70 volunteers, over four weeks; it says nothing about long-term use or about people who wouldn't sign up for a stress study.
That there's no difference, if p had been above .05. A non-significant result would have meant the data were inconclusive, not that the app does nothing. To show two groups are about the same, set the smallest difference that would matter and run an equivalence test (two one-sided tests).

Common Mistakes

Don'tSwitch to a one-sided p because the result went the way you hoped.

InsteadUse a one-sided test only if you specified the direction before collecting data and have a defensible reason why an effect in the opposite direction wouldn't count toward your claim. Otherwise, use a two-sided test.
Don'tRun Levene's test to choose between Student's and Welch's t-test.

InsteadUse Welch's t-test by default. It costs almost nothing when the variances are equal.
Don'tTest each group's change (or each group against a norm) separately and compare the two p-values.

InsteadCompare the groups directly, with this test on the outcome or the change scores.
Don'tRun a t-test on every pair when you have three or more groups.

InsteadRun a one-way ANOVA, then planned contrasts or post hoc tests.
Don'tClaim the grouping caused the difference when people weren't randomly assigned.

InsteadSave causal language for randomized groups. Otherwise, describe how the groups differ.
Don'tRead a p-value as the chance that the null hypothesis is true.

InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Result Decoder

Significance and Effect Size Guidelines

Benchmarks Significance (p Value)

p < .001
Very strong evidence against the null hypothesis.
.001 ≤ p < .01
Strong evidence against the null hypothesis.
.01 ≤ p < .05
Statistically significant at α = .05. Moderate evidence against the null hypothesis.
.05 ≤ p < .10
Not significant. Weak evidence; sometimes called a “trend.” Treat it as inconclusive.
p ≥ .10
Not significant. Little evidence against the null, which is not the same as evidence of no effect.

These evidence labels are one common convention for describing p in graded language rather than as significant or not; the boundaries are rough guides, not rules, and other authors draw them differently. Muff et al. (2022) argue for this kind of language of evidence over a single cutoff. α is chosen before the analysis, and p never says how big or important an effect is.

More about statistical significance ↓

Benchmarks Effect Size (Cohen's d)

|d| < 0.2 Below Small
Smaller than the conventional small benchmark. Whether an effect this size matters depends on the setting.
0.2 ≤ |d| < 0.5 Small
A relatively modest standardized difference
0.5 ≤ |d| < 0.8 Medium
A moderate standardized difference
|d| ≥ 0.8 Large
A relatively substantial standardized difference

The sign shows direction. Cohen (1988) named 0.2, 0.5 and 0.8 small, medium and large; the “below small” band under 0.2 is this site's label, not Cohen's. Benchmarks are conventions, not measures of practical or clinical importance.

More about effect size ↓

How It's Reported

Equation Form

Example

M_diff = −3.50, 95% CI [−6.52, −0.48], d = −0.56, 95% CI [−1.03, −0.08], t(62.62) = −2.32, p = .024

Full Text (APA Style)

Example

Students who used the mindfulness app reported less stress (M = 17.00, SD = 5.50, n = 36) than students on the waitlist (M = 20.50, SD = 7.00, n = 34), a difference of −3.50 points, 95% CI [−6.52, −0.48], d = −0.56 (pooled SD), 95% CI [−1.03, −0.08]. Welch's t-test showed the difference was statistically significant, t(62.62) = −2.32, p = .024.

Say you used Welch's t-test, so readers know why df isn't a whole number.
Give each group's M, SD and n, and say which group was subtracted from which, so the sign makes sense.
Give the size and interval before the significance test, and say that d uses the pooled SD.
Italicize statistical symbols (M, SD, d, t, p) and report exact p to three decimals.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

M_diff = −3.50	Students who used the app scored 3.50 points lower on the 40-point stress scale than students on the waitlist.
d = −0.56	The app group's stress was 0.56 standard deviations lower. That's a medium effect by the benchmarks above; whether 3.5 points of stress matters is a judgment about students' lives, not statistics. Benchmarks ↑
95% CI [−6.52, −0.48]	The true difference is plausibly anywhere from about half a point to six and a half points lower with the app. The whole range is below zero, though its near end is close to it.
95% CI [−1.03, −0.08]	The true effect is plausibly anywhere from about 0.1 to about 1 standard deviation, from below small to large by the benchmarks, so this study can't pin down its size. Software finds it from the noncentral t distribution, using Student's t, so like the pooled SD it assumes equal spreads; SPSS prints it in the effect-size table.
t(62.62) = −2.32	The difference is 2.32 standard errors below zero. The 62.62 is Welch's degrees of freedom, which is why it isn't a whole number.
p = .024	If the app made no difference on average, a Welch t at least 2.32 from zero (a gap at least 2.32 of its own standard errors), in either direction, would turn up about 2.4% of the time. That's below .05, so the result is statistically significant. Benchmarks ↑

Calculations

The Running Example

Example

The stress-app study: 70 students randomly assigned to a mindfulness app (36) or a waitlist (34), compared on stress scores after four weeks.

Null Hypothesis:: $\mu_1 = \mu_2$
Alternative (Two-Sided):: $\mu_1 \neq \mu_2$

$\mu_1$ and $\mu_2$ (“mu one” and “mu two”) are the population means, the average stress of all students like ours after four weeks with the app ( $\mu_1$ ) or on the waitlist ( $\mu_2$ ). The samples' 17.00 and 20.50 estimate them.

What Your Data Should Look Like

One row per person, with a column saying which group they're in and a numeric column for the outcome. Each person appears once, in one group only.

Example

group	score
app	23
waitlist	31
waitlist	25
waitlist	17
app	22
app	24
… 64 more rows

	n	M	SD
App	36	17.00	5.50
Waitlist	34	20.50	7.00
All	70	18.70	6.47

Simulated data.

Download all 70 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the stress-app study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Find the Difference between the Two Groups' Averages

Everything starts with the gap between the two groups. A negative gap means the app group came in lower.

$\bar{x}_1 - \bar{x}_2$ = $17.00 - 20.50$

= −3.50 points

$\bar{x}_1$ is the app group's average stress, 17.00 points, and $\bar{x}_2$ is the waitlist's, 20.50. 17.00 − 20.50 = −3.50 points.
2. Figure Out How Much Scores Vary within Each Group

Before we can judge whether a gap is big, we need to know how much scores normally bounce around. We do this separately in each group.

$s = \sqrt{\dfrac{\sum (x_i - \bar{x})^2}{n - 1}}$

$s_1$ = 5.50 points
$s_2$ = 7.00 points

$x_i$ is each student's score and $\bar{x}$ is their own group's average. Find each distance, square it, add them up, divide by $n - 1$ and take the square root. That's 5.50 points in the app group ( $s_1$ ) and 7.00 on the waitlist ( $s_2$ ).
3. Figure Out How Precisely We've Estimated the Difference

Each group's average would wobble from sample to sample, and the difference inherits the wobble from both. A spread squared, $s^2$ , is called the variance. Because the groups are independent, their squared wobbles (the variances of their means) add, so the difference wobbles more than either mean but less than the two wobbles added together. Welch's version lets each group keep its own spread instead of assuming the two match.

$SE = \sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}$ = $\sqrt{\dfrac{5.50^2}{36} + \dfrac{7.00^2}{34}}$

= 1.51 points

$s_1$ = 5.50 and $s_2$ = 7.00 are the two groups' spreads, and $n_1$ = 36 and $n_2$ = 34 are their sizes. $s_1^2/n_1$ = 0.84 is the app mean's squared standard error (0.92²), and $s_2^2/n_2$ = 1.44 is the waitlist's (1.20²). 0.84 + 1.44 = 2.28, and its square root is 1.51 points.
4. Compare the Difference with That Uncertainty

A gap only means something relative to that wobble. t says how many standard errors the gap is.

$t = \dfrac{\bar{x}_1 - \bar{x}_2}{SE}$ = $\dfrac{-3.50}{1.51}$

= −2.32

Our −3.50-point gap divided by the 1.51-point $SE$ puts the app group 2.32 standard errors below the waitlist.
5. Account for How Much Information We Had

We judge t against a t distribution: the spread of t values that studies like this one would give if the two groups' true averages were equal. Its exact shape depends on df. Welch's version works df out from the two groups' spreads and sizes, so it's rarely a whole number. It falls between 33 (the smaller group's n − 1) and 68, the 70 students minus the two means we estimated, which is what assuming equal spreads would give.

$df = \dfrac{\left(\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}\right)^2}{\dfrac{(s_1^2/n_1)^2}{n_1 - 1} + \dfrac{(s_2^2/n_2)^2}{n_2 - 1}}$ = $\dfrac{(0.8397 + 1.4407)^2}{\dfrac{0.8397^2}{35} + \dfrac{1.4407^2}{33}}$

= 62.62

$s_1^2/n_1$ = 0.8397 and $s_2^2/n_2$ = 1.4407 are the two pieces under the square root in step 3, and 35 and 33 are each group's $n - 1$ . 5.2002 ÷ 0.08304 = 62.62.
6. Put Both Groups' Spread on One Scale

To say how big the gap is, we need one typical spread for a student's score. Pooling averages the two groups' variances, giving the bigger group a little more weight.

$s_p = \sqrt{\dfrac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}$ = $\sqrt{\dfrac{35 \times 5.50^2 + 33 \times 7.00^2}{36 + 34 - 2}}$

= 6.27 points

$s_1$ = 5.50 and $s_2$ = 7.00 are weighted by $n_1 - 1$ = 35 and $n_2 - 1$ = 33. 2675.8 ÷ 68 = 39.35, and its square root is 6.27 points.
7. Measure How Big the Difference Actually Is

t grows with sample size; d doesn't. Dividing by the usual spread among students in the same group instead of the standard error gives a size you can compare across studies.

$d = \dfrac{\bar{x}_1 - \bar{x}_2}{s_p}$ = $\dfrac{-3.50}{6.27}$

= −0.56

$\text{CI}_{d}$ = [−1.03, −0.08]

The same −3.50-point gap, divided by the 6.27-point pooled spread $s_p$ instead, is 0.56 of a standard deviation.
8. Give a Range of Plausible Values for the True Difference

Instead of a single best guess, give the range of true differences that fit the data.

$(\bar{x}_1 - \bar{x}_2) \pm t^{*}_{df} \times SE$ = $-3.50 \pm 1.999 \times 1.51$

= [−6.52, −0.48] points

$t^{*}_{62.62}$ = 1.999 is the cutoff that holds the middle 95% of a t distribution with 62.62 df. 1.999 × 1.51 = 3.02 points, so the range is −3.50 ± 3.02, from −6.52 to −0.48 points.
9. Ask How Surprising the Result Would Be If There Were No Real Difference

Finally, ask how surprising our gap would be if the two groups' true averages were the same. If they were, and we ran this study again and again, t would land near 0 most of the time and far from it only rarely. The t distribution is the shape of those t values, and p is the share of it at least as far from 0 as ours, in either direction.

$p = 2 \times P(T_{df} \le -|t|)$ = $2 \times P(T_{62.62} \le -2.32)$

= .024

$P(T_{62.62} \le -2.32)$ is the chance that a t with 62.62 df lands at or below −2.32. Doubling it counts both directions, .024, about 1 in 42.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

Analyze → Compare Means and Proportions → Independent-Samples T Test.
Move score into Test Variable(s) and group into Grouping Variable.
Click Define Groups and enter app for Group 1 and waitlist for Group 2.
Leave Estimate effect sizes checked, and click OK.

Or paste this syntax:

T-TEST GROUPS=group('app' 'waitlist')
  /MISSING=ANALYSIS
  /VARIABLES=score
  /ES DISPLAY(TRUE)
  /CRITERIA=CI(.95).

Reading the Output

SPSS prints three tables and runs both versions of the test. Read the Equal variances not assumed row: that's Welch's t-test. Skip Levene's test rather than using it to pick a row (Delacre, Lakens & Leys, 2017). Then read in this order: Mean Difference, Cohen's d, the 95% Confidence Interval, then Two-Sided p.

Group Statistics

group	N	Mean	Std. Deviation	Std. Error Mean
app	36	17.00	5.498	.916
waitlist	34	20.50	6.999	1.200

Independent Samples Test (score)

	Levene's F	Sig.	t	df	One-Sided p	Two-Sided p	Mean Difference	Std. Error Difference	95% CI Lower	Upper
Equal variances assumed	1.668	.201	-2.334	68	.011	.023	-3.500	1.500	-6.493	-.507
Equal variances not assumed			-2.318	62.620	.012	.024	-3.500	1.510	-6.518	-.482

Independent Samples Effect Sizes

	Standardizer	Point Estimate	95% CI Lower	Upper
Cohen's d	6.271	-.558	-1.034	-.078
Hedges' correction	6.342	-.552	-1.023	-.077

SPSS's Cohen's d divides by the pooled SD (the Standardizer), the same d as on this page. “Hedges' correction” adjusts it for small-sample bias; at 70 students the two barely differ. SPSS also prints a Glass's delta row, which divides by the waitlist's SD alone (6.999), giving −.500.

The two rows barely differ here because the spreads are similar (5.50 vs 7.00). When they differ more, especially with unequal group sizes, the rows can disagree, and the Welch row is the one to trust.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

stress <- read.csv("data.csv")
t.test(score ~ group, data = stress)  # Welch's version is R's default

# Cohen's d, using the pooled SD
m <- tapply(stress$score, stress$group, mean)
s <- tapply(stress$score, stress$group, sd)
n <- table(stress$group)
sp <- sqrt(((n[1] - 1) * s[1]^2 + (n[2] - 1) * s[2]^2) / (sum(n) - 2))
unname((m[1] - m[2]) / sp)

	Welch Two Sample t-test

data:  score by group
t = -2.3177, df = 62.62, p-value = 0.02375
alternative hypothesis: true difference in means between group app and group waitlist is not equal to 0
95 percent confidence interval:
 -6.5180599 -0.4819401
sample estimates:
     mean in group app mean in group waitlist
                  17.0                   20.5

[1] -0.5580861

R subtracts the second group (alphabetically, waitlist) from the first (app), so the difference and its interval are negative. Add var.equal = TRUE only if you have a specific reason to assume equal variances.

Output from a real run: R 4.6.1, run 2026-09-29.

Python

import pandas as pd
from scipy import stats

stress = pd.read_csv("data.csv")

app = stress.loc[stress["group"] == "app", "score"]
waitlist = stress.loc[stress["group"] == "waitlist", "score"]

result = stats.ttest_ind(app, waitlist, equal_var=False)  # Welch's version
result.statistic             # -2.3177
result.pvalue                # 0.02375
result.df                    # 62.62
result.confidence_interval() # (-6.5181, -0.4819)

# Cohen's d, using the pooled SD
n1, n2 = len(app), len(waitlist)
sp = (((n1 - 1) * app.var() + (n2 - 1) * waitlist.var()) / (n1 + n2 - 2)) ** 0.5
(app.mean() - waitlist.mean()) / sp  # -0.5581

equal_var=False makes it Welch's test. SciPy's default is equal_var=True, Student's version, so set it explicitly.

Output from a real run: Python 3.9.6 (scipy 1.13.1, pandas 2.3.3), run 2026-09-29.

Impact

Effect Size

Cohen's d expresses the difference in standard deviations, so it can be compared across studies that used different scales, but it still depends on the design, how varied the sample is, and how reliably things were measured. With two groups, the standard deviation is the pooled standard deviation, the typical spread of a score around its own group's mean (Lakens, 2013, calls this version $d_s$ ).

d = \dfrac{\bar{x}_1 - \bar{x}_2}{s_p} = \dfrac{17.00 - 20.50}{6.27} = -0.56

Students who used the app scored 3.50 points lower than the waitlist, which is 0.56 of the usual spread among students in the same group, a medium effect by convention. With groups this size, though, the true effect could be anywhere from below small to large (95% CI [−1.03, −0.08]).

Welch's test doesn't pool the variances, but d still needs one yardstick, and the pooled SD is the standard choice. Choosing Welch doesn't make d's interval robust too: the usual noncentral-t interval for d comes from Student's equal-variance model. Here the spreads are close (7.00 vs. 5.50), so it's a fair guide. When the two spreads differ a lot, say which SD you divided by. You can also report Glass's Δ, which divides by the control group's SD alone, or divide by $\sqrt{(s_1^2 + s_2^2)/2}$ , the square root of the two variances' plain average, which weights the two spreads equally whatever the group sizes. Either is a different yardstick, not a fix for the interval: each needs its own interval method, so say how you got it.

Benchmarks are conventions. Whether 3.5 fewer points of stress matters is a question about students' lives, not statistics.

Benchmarks Cohen's d at a Glance

Below Small

|d| < 0.2

Small

0.2 ≤ |d| < 0.5

Medium

0.5 ≤ |d| < 0.8

Large

|d| ≥ 0.8

Full benchmarks and caveats ↑

Statistical Significance

p answers one narrow question: if the two populations had the same average, how often would Welch's t land at least this far from zero, in either direction? That's a gap at least this many standard errors wide, with the standard error estimated from each sample's own spreads. Below the conventional α = .05, the result is called statistically significant.

If the app made no difference on average, a Welch t at least 2.32 from zero, in either direction, would turn up about 2.4% of the time, roughly 1 in 42. That's below .05, so it's significant.

This page uses Welch's version of the test, which doesn't assume the two groups are equally spread out. Delacre, Lakens and Leys (2017) recommend it as the default for psychology, and it's R's default. When the spreads match, it gives almost the same answer as Student's version (here, p = .024 vs. .023); when they don't, Welch's p stays close to accurate and Student's can come out too small or too large.

p is not the probability that the null hypothesis is true, and it says nothing about how large or important the difference is. That's what the effect size and interval are for.

Benchmarks p at a Glance

Very Strong

p < .001

Strong

.001 ≤ p < .01

Significant

.01 ≤ p < .05

Weak

.05 ≤ p < .10

Little

p ≥ .10

Strength of evidence against the null hypothesis.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. A real effect doesn't guarantee a significant result: with too few people, it can easily come out non-significant. Power depends on four things: how big the true effect is, how many people you have in each group, α, and whether the test is one- or two-sided.

Suppose, before collecting any data, we thought the app might lower stress by half a standard deviation (d = 0.5). With 64 students per group, the study would come out significant about 80% of the time. With the 35 or so per group we actually had, it would have been about 54%, not much better than a coin flip.

These figures, and the calculator below, use the standard planning model for two equal-sized groups with equal spreads. That's a close guide for Welch's test when the spreads and group sizes are similar; with clearly unequal spreads or deliberately unequal groups, plan with a calculation (or simulation) that builds those in.

Researchers usually aim for at least 80% power. Plan it before collecting data, using the smallest effect that would matter. Try it:

Other Info

Follow-Up and Post Hoc Tests

None needed. This test answers a single question, so there are no planned contrasts or post hoc tests to run.

Related Methods

General CaseSimple Linear Regression: Code group as 0 and 1 and regress the outcome on it. The slope is the difference between the means, and its t-test is Student's version of this test; Welch's version estimates the same difference without pooling the variances.
General CaseOne-Way Between-Subjects ANOVA: With just two groups, a one-way ANOVA's F is Student's t squared, and Welch's ANOVA's F is Welch's t squared, with the same p.
Related DesignPaired-Samples t-Test: Compares two means too, but when the same people are measured twice. Pairing removes stable differences between people, so when the two measurements are positively correlated, as they usually are, it needs fewer people.
Related DesignBayesian Independent-Samples t-Test: Same design, same t statistic, different question. Instead of asking how surprising the difference would be if the population means were equal, it weighs how much the data favor a difference over none, so it can also find evidence that they're equal.
Alternative MeasureMann-Whitney U Test: The rank-based test for the same two-group design. It asks whether scores in one group tend to be higher than in the other rather than comparing means, so it suits ordered ratings, or questions about which group tends to score higher when comparing means isn't the goal.
Related DesignMixed Factorial ANOVA: With two groups measured twice, a Student's (pooled-variance) independent-samples t on the change scores (after minus before) is the mixed ANOVA's time × group interaction: its square is the interaction's F, with the same p.
Related Designz-Test for Proportions: answers the same two-group question when the outcome is yes/no rather than a score.
AlternativeMultilevel Regression (HLM): An alternative if “Independent Students” doesn't hold. A multilevel model, which accounts for the clustering.

References

Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.2307/2331554
Welch, B. L. (1947). The generalization of ‘Student's’ problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. https://doi.org/10.1093/biomet/34.1-2.28
Fagerland, M. W. (2012). t-tests, non-parametric tests, and large studies—a paradox of statistical practice? BMC Medical Research Methodology, 12, 78. https://doi.org/10.1186/1471-2288-12-78
“When the sample size increases, so does the robustness of the t-tests to deviations from normality.” (Discussion; Table 4)
Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology, 30(1), 92–101. https://doi.org/10.5334/irsp.82
“We argue that Welch’s t-test should be used as a default strategy.” (Abstract)
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863
“The denominator is the pooled standard deviation.” (section “Cohen's d in between-subjects designs”)
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009
“we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)