Stats for
Menu

ANOVA Family · General Linear Model

Bayesian ANOVA

A Bayesian ANOVA weighs the evidence for and against group means differing.

A Bayesian ANOVA takes one continuous outcome from three or more separate groups and weighs how much the data favor different population means over equal ones, or the reverse.

Example

Does it matter which version of a mindfulness app students use, or do all three leave them about equally stressed? Ninety undergraduates were randomly assigned, 30 each, to a guided-audio, written or gamified version of the same app for four weeks, then rated their stress on a 0–40 questionnaire.

Three or More Groups + Continuous Outcome → Bayesian ANOVA
Tests
Evidence for differences among population means versus none
Observations
Independent
Parametric
Yes
Headline Statistic
BF_{01}
Effect Size
Eta Squared
Groups
3+

The Basics

When to Use It

Use it when you have three or more separate groups of different people, one continuous outcome, and you want to weigh the evidence for differences between their averages against the evidence for none. It earns its keep when "no difference" is a real possibility you'd like to be able to support, not just fail to reject. For example:

  • Does it matter which version of a mindfulness app students use, or do all three leave them about equally stressed?
  • Is working-memory span the same after 10, 20 or 30 minutes of a video game, or does it change with play time?
  • Do three wordings of a survey question produce different average trust ratings, or equivalent ones?

Vocabulary You Need

Definition Bayes Factor (BF_{10})

A ratio comparing how well two hypotheses predicted the observed data: each hypothesis's marginal likelihood (how well it predicted the data, averaged over its prior distribution), divided by the other's. For continuous data these are predictive densities (heights of a curve at the observed data), not probabilities of those exact observations. BF₁₀ puts the alternative on top and BF₀₁ = 1 / BF₁₀ puts the null on top, so BF₁₀ = 5 and BF₀₁ = 5 are equally strong, in opposite directions. Values near 1 mean the data can't tell the hypotheses apart. Unlike a p-value, it can show evidence for the null, not just a failure to reject it. It depends on the prior distributions, so report which ones you used. Glossary →

Definition Prior Distribution

A probability distribution that says which values of a parameter, such as the size of an effect, a hypothesis considers plausible before the data are in. In a Bayes factor it's how each hypothesis makes its predictions: a narrow prior bets on small effects, a wide one spreads its bets over large effects too. The Bayes factor depends on it, so the prior is chosen before the analysis, usually a published default, and results are checked with other reasonable priors. Glossary →

Definition Posterior Probability (P(H \mid \text{data}))

The probability of a hypothesis after the data are in. It combines the Bayes factor with the prior odds, how probable the hypotheses seemed beforehand: posterior odds = Bayes factor × prior odds. With equal prior odds and two hypotheses, the null's posterior probability is BF₀₁ / (1 + BF₀₁). Unlike a p-value, it is a probability about the hypothesis, but it's only as sensible as the prior odds behind it. Glossary →

Definition F Statistic (F)

The variance due to an effect divided by the error variance it is tested against: in a between-subjects ANOVA, the between-groups variance over the within-groups variance; in a repeated-measures ANOVA, the conditions' variance over the inconsistency in how people change across them. It is near 1 when there's no real effect and grows as the effect stands out from the error. It is compared with an F distribution, which has two degrees of freedom, to get a p-value. Glossary →

Assumptions and Inference

1. Independent Students (for the Model) Each student's score is independent of everyone else's, within and between groups. Random assignment doesn't ensure this if students share classrooms, teams or other clusters.
What It Means
Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To
Observations
How to Check
By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters
A lot. If students used the app together or were recruited in clusters, their scores are more alike than the model expects, and the Bayes factor overstates the evidence in whichever direction it points.

What to Do

  • Alternative Test Multilevel Regression (HLM). A multilevel model, Bayesian or classical, which accounts for the clustering.
  • Design Change Assign and measure each person separately, or analyze cluster averages.
2. Normality Within Groups (for the Model) The default Bayesian ANOVA models each group's scores as normal around its mean. It doesn't become distribution-free because it uses Bayes factors: the Bayes factor compares how well the specified models predicted the data, so a poor likelihood changes the evidence itself. Larger samples make estimates more stable, but they don't make a misspecified model irrelevant.
What It Means
The test's model assumes the scores it relies on are roughly normal: the scores within each group (or the scores themselves, for a one-sample test), the difference scores in a paired design, or the residuals (the errors left around a model's predictions) in a regression. For p-values and confidence intervals, what really matters is that the estimates (such as a mean, a difference between means or a model's coefficients) are roughly normal across repeated samples. A large enough sample usually provides that; how large depends on how skewed the scores are and how extreme their outliers.
Applies To
Residuals
How to Check
Here a residual is each student's score minus their group's average. Plot those in a histogram or Q–Q plot, not the raw scores pooled across groups, which mix three different centers. Look for strong skew, heavy tails or extreme scores. Don't rely on a normality test: with small samples it misses real problems, and with large ones it flags trivial ones.
How Much It Matters
The Bayes factor uses the same normal model as the F-test, so outliers pull the group means and inflate the within-groups variance here too, and the evidence shifts with them. With 30 per group and no extreme scores, as in our data, moderate skew is a smaller worry than a few wild scores; check whether one or two of them drive the result. Here the residuals' skewness runs from −0.17 to 0.88 (the text group leans right), and no score is more than 2.5 SDs from its group's mean.

What to Do

  • Alternative Test Kruskal-Wallis Test. A classical rank-based comparison, for ordered ratings or a question about which groups tend to score higher. It doesn't test equal means, gives no Bayes factor and can't give evidence for no difference. If means are still the question and scores are skewed or heavy-tailed, fit a robust Bayesian model instead, such as one with t-distributed errors (in R, brms with family = student()).
  • Transformation Log-transform right-skewed outcomes such as reaction times. Results are then on the log scale.
  • Proceed with Caveat Rerun the analysis without the extreme scores and report both Bayes factors if they differ.
3. Equal Variances (for the Model) The default Bayesian ANOVA assumes the groups share one population variance. Equal group sizes help with moderate differences, but substantial differences in spread call for a Bayesian model that gives each group its own variance.
What It Means
The outcome varies about as much within each group's population. Tests that pool the groups' variances into one estimate, like Student's t-test and the classic ANOVA, rely on it.
Applies To
Variances
How to Check
Compare the groups' standard deviations and look at side-by-side box plots. Don't use Levene's test as a gatekeeper: it misses real differences in small samples and flags trivial ones in large samples.
How Much It Matters
Matters most when groups differ in both size and spread; with equal groups, like our 30 per version with SDs from 6.00 to 7.00 (Levene's F(2, 87) = 0.94, p = .40), it matters less. The default Bayesian ANOVA estimates one shared variance for all the groups, just as the classical F-test does.

What to Do

  • Design Change Keep the groups the same size, which makes a pooled variance far less risky.
  • Alternative Test Fit a model that gives each group its own SD (in R, brms with bf(stress ~ app_version, sigma ~ app_version)) and read the posteriors for the differences. A Bayes factor from such a model needs its own priors and method, so its number isn't comparable with the default one.
  • Proceed with Caveat Run Welch's ANOVA alongside. It can flag a difference the pooled model hides, but a non-significant Welch p doesn't back up the Bayes factor's evidence for equal means.

Limitations

This Is Not Your Test If…

What It Doesn't Tell You

  • That the versions are identical. A BF₀₁ of 5.74 says equal means predicted the data better than the default alternative did. Tiny differences, too small for 90 students to detect, are still possible.
  • Anything prior-free. The evidence depends on the prior. With wider priors the same data favor equal means more strongly (BF₀₁ = 9.55 at r = 0.707, 16.98 at r = 1; see the prior-sensitivity table in How It's Reported); a prior concentrated on very small effects would favor them less.
  • Whether the app works at all. All three groups used an app. With no untreated comparison group, the study says nothing about whether any version lowers stress.
  • Who else. The result describes students like these 90, over four weeks, and only as well as a self-report questionnaire measures stress.

Common Mistakes

  • Don'tRead a non-significant p as evidence that the groups are the same.

    InsteadReport a Bayes factor, which can show evidence for no difference, or show that the data can't tell.

  • Don'tCall a Bayes factor of 2 or 0.5 evidence for anything.

    InsteadCall it inconclusive. Between 1/3 and 3, the data barely favor either hypothesis.

  • Don'tReport a Bayes factor without its prior.

    InsteadName the prior and its scale (for example, a Cauchy prior with r = 0.5), and show how the result changes with a wider one.

  • Don'tChoose the prior scale after seeing which one gives the result you want.

    InsteadFix it before the analysis, usually the default, and report a robustness check with other scales.

  • Don'tRead a Bayes factor as the probability that a hypothesis is true.

    InsteadRead it as how many times better one hypothesis predicted the data than the other. Turning it into a probability needs prior odds too.

Result Decoder

Evidence and Effect Size Guidelines

Benchmarks Strength of Evidence (Bayes Factor)
  • BF₁₀ < 1/100 Extreme for H₀

    Extreme evidence for the null hypothesis.

  • 1/100 ≤ BF₁₀ < 1/30 Very Strong for H₀

    Very strong evidence for the null hypothesis.

  • 1/30 ≤ BF₁₀ < 1/10 Strong for H₀

    Strong evidence for the null hypothesis.

  • 1/10 ≤ BF₁₀ < 1/3 Moderate for H₀

    Moderate evidence for the null hypothesis.

  • 1/3 ≤ BF₁₀ < 1 Anecdotal for H₀

    Barely worth mentioning. Treat it as inconclusive.

  • 1 ≤ BF₁₀ < 3 Anecdotal for H₁

    Barely worth mentioning. Treat it as inconclusive.

  • 3 ≤ BF₁₀ < 10 Moderate for H₁

    Moderate evidence for the alternative hypothesis.

  • 10 ≤ BF₁₀ < 30 Strong for H₁

    Strong evidence for the alternative hypothesis.

  • 30 ≤ BF₁₀ < 100 Very Strong for H₁

    Very strong evidence for the alternative hypothesis.

  • BF₁₀ ≥ 100 Extreme for H₁

    Extreme evidence for the alternative hypothesis.

Labels from Lee and Wagenmakers, adjusted from Jeffreys (1961). They are a rough heuristic, not cutoffs; Kass and Raftery (1995) proposed coarser ones. For BF₀₁, flip the scale: BF₀₁ = 5 is BF₁₀ = 1/5.

More about strength of evidence ↓

Benchmarks Effect Size (Eta Squared)
  • η² < .01 Below Small

    Smaller than the conventional small benchmark. Whether an effect this size matters depends on the setting.

  • .01 ≤ η² < .06 Small

    A sliver of the variation lines up with the groups. Hard to see without measuring it.

  • .06 ≤ η² < .14 Medium

    A noticeable share of the variation lines up with the groups.

  • η² ≥ .14 Large

    A large share of the variation lines up with the groups.

Benchmarks are conventions (Cohen, 1988), developed for simple comparisons between groups. They are not measures of practical importance.

More about effect size ↓

How It's Reported

Equation Form

Example

BF01 = 5.74 (JZS prior, r = 0.5), F(2, 87) = 0.71, p = .496, η² = .02, 95% CI [.00, .09]

Table 1. Prior Sensitivity of the Bayes Factor for App Version

Example
BF_{10}BF_{01}P(H_0 \mid \text{data})
Medium, r = 1/2 0.175.74.85
Wide, r = \sqrt{2}/2 0.109.55.91
Ultrawide, r = 1 0.0616.98.94

Note. r is the scale of the Cauchy prior on the standardized effects; the names are R's BayesFactor presets, and “medium” is the default reported above. Posterior probabilities assume even prior odds. Wider priors spread the alternative's bets over larger effects, so these small differences fit it worse and the evidence for equal means grows. It favors equal means at every scale.

Full Text (APA Style)

Example

Stress after four weeks was similar across the three versions of the app: audio (M = 17.00, SD = 6.00), text (M = 18.00, SD = 6.50) and gamified (M = 16.00, SD = 7.00). App version accounted for under 2% of the variation in stress, η² = .02, 95% CI [.00, .09] (a classical confidence interval). A Bayesian one-way ANOVA with the default prior (Cauchy, r = 0.5; Rouder et al., 2012) favored equal means: the data were about 5.74 times better predicted by equal population mean stress scores than by the alternative model, BF01 = 5.74, moderate evidence for the null hypothesis. With wider priors (r = 0.707 and 1), BF01 rose to 9.55 and 16.98. A classical one-way ANOVA was not significant, F(2, 87) = 0.71, p = .496.

  • Name the prior and its scale, and the software, so readers can reproduce the Bayes factor.
  • Say which way the Bayes factor points: BF₀₁ for evidence for the null, BF₁₀ for evidence for a difference. Give the evidence label too, as a rough guide.
  • Report a robustness check with at least one wider prior; conclusions that flip with the prior deserve a caveat.
  • Give every group's M and SD, and the effect size with its CI, so readers can see how small the differences are, not just that the evidence favors none.
  • If you give the posterior probability of the null, say which prior odds it assumes (.85 here assumes 50–50).

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how surprising.

η² = .02 App version accounts for under 2% of the variation in stress scores in this sample. With F below 1, that's mostly noise: the 95% CI runs from 0 to .09, so don't read it as a small effect. On its own, a sample η² can't tell you whether the true effect is small or zero. Benchmarks ↑
F(2, 87) = 0.71 The three averages vary less than we'd expect from how much students vary within a version: 0.71 times as much. The 2 and 87 are the degrees of freedom: k − 1 and N − k.
BF01 = 5.74 Equal population mean stress across the three versions predicted the data 5.74 times better than averages differing by the amounts the default prior expects. That's moderate evidence for the null by the conventional labels, which are rough guides, not cutoffs. It's about the data, not yet about how likely the null is. Benchmarks ↑
P(H0 | data) = .85 Starting from 50–50, there's now about an 85% chance that the population mean stress scores are exactly equal, and a 15% chance they differ. That isn't an 85% chance the versions are practically equivalent, which would need a margin set in advance. A skeptic who started at 20% for the null would end near 59%, so say which prior odds you used.
p = .496 Not significant. But the same p could come from versions that truly work alike or from a study too small to tell them apart; p can't distinguish the two.

Calculations

The Running Example

Example

The mindfulness-app study: 90 undergraduates, 30 per app version (audio, text, gamified), each with one stress score from 0 to 40.

Null Hypothesis:
\mu_{\text{audio}} = \mu_{\text{text}} = \mu_{\text{game}}
Alternative:
\text{not all three } \mu \text{ are equal}

Each \mu (“mu”) is the population mean stress score for students using that version, which our group averages of 17, 18 and 16 estimate. Unlike a classical test, the alternative has to say how big the differences might be: its prior distribution on the standardized differences is a Cauchy with scale r = 0.5, the default in R's BayesFactor package. A standardized effect is a rescaled version of a group's distance from the overall mean, in standard deviations (with two groups, r = 0.5 here is the same prior as r = 0.707 on the standardized difference δ in a Bayesian t-test). The Cauchy is a bell-shaped curve with heavier tails: with r = 0.5, the alternative expects half of those effects to lie within ±0.5 SD but leaves room for much larger ones. This default is called the JZS (Jeffreys–Zellner–Siow) prior.

What Your Data Should Look Like

One row per person, with one column naming their group and one numeric column for their score. Each person appears once, in exactly one group. It's the same layout as a classical one-way ANOVA; the difference is in the analysis, not the data.

Example
participantapp_versionstress
1game12
2text21
3game19
4text34
5text25
6text23
… 84 more rows
nMSD
Audio3017.006.00
Text3018.006.50
Gamified3016.007.00
All9017.006.49

Simulated data.

Download all 90 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the mindfulness-app study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

  1. 1. Find Each Version's Average and the Overall Average

    Everything starts with the group averages, and the easiest way to compare them is to measure each against the average of everyone.

    \bar{x}_{\text{grand}} = \dfrac{\sum x_i}{N} =\dfrac{1530}{90}

    = 17.00 points

    \bar{x}_{\text{audio}} = 17.00 points

    \bar{x}_{\text{text}} = 18.00 points

    \bar{x}_{\text{game}} = 16.00 points

    \sum x_i is all 90 stress scores added up, 1530, and N is our 90 students. 1530 ÷ 90 = 17.00 points. The three versions averaged 17 (audio), 18 (text) and 16 (gamified).

  2. 2. Split the Variation into Differences between Versions and Differences within Them

    Some of the spread in stress lines up with which version a student used; the rest is students differing from others on the same version. Both are sums of squared distances.

    SS_{\text{between}} = \sum n_j (\bar{x}_j - \bar{x}_{\text{grand}})^2 =30(17 - 17)^2 + 30(18 - 17)^2 + 30(16 - 17)^2

    = 60

    SS_{\text{within}} = 3690

    n_j is the 30 students per version and \bar{x}_j each version's average. Text sits 1 above 17 and gamified 1 below, so 0 + 30 + 30 = 60. Within the versions, the squared distances from each group's own average add up to 3690, far more.

  3. 3. Compare the Spread between Versions with the Spread within Them

    Turn each sum into an average by dividing by its degrees of freedom, then take the ratio. If the versions don't matter, it should hover around 1.

    F = \dfrac{MS_{\text{between}}}{MS_{\text{within}}} =\dfrac{60 / 2}{3690 / 87} = \dfrac{30}{42.41}

    = 0.71

    MS_{\text{between}} is 60 over k - 1 = 2, which is 30. MS_{\text{within}} is 3690 over N - k = 87, which is 42.41. 30 ÷ 42.41 = 0.71, less than the 1 we'd expect from noise alone.

  4. 4. Measure How Much of the Variation the Versions Explain

    F grows with sample size; η² doesn't. It's the share of all the variation in stress that lines up with the app version, and the Bayes factor formula uses it too.

    \eta^2 = \dfrac{SS_{\text{between}}}{SS_{\text{between}} + SS_{\text{within}}} =\dfrac{60}{60 + 3690}

    = .02

    95\%\ \text{CI} = [.00, .09]

    SS_{\text{between}} is our 60 and the total is 60 + 3690 = 3750. 60 ÷ 3750 = .016, so the version accounts for 1.6% of the differences in stress.

  5. 5. See What the Classical Test Can and Can't Say

    The classical ANOVA asks how often chance alone would spread three averages at least this far apart if the versions worked equally well and the model's assumptions held. A large p says the data aren't surprising under the null; it can't say whether they support it.

    p = P(F_{df_1,\,df_2} \ge F) =P(F_{2,\,87} \ge 0.71)

    = .496

    F_{2,\,87} is an F distribution with 2 and 87 df. The chance it lands at or above 0.71 is .496, about 1 in 2.

  6. 6. Get a Rough Bayes Factor by Hand

    The Bayesian Information Criterion (BIC) scores each model for fit, with a penalty for every extra parameter. Lower is better, so a positive \Delta BIC_{10} means the equal-means model wins. Half the difference between the two scores, exponentiated (exp undoes the natural log, ln), approximates the Bayes factor (Wagenmakers, 2007). It's a quick large-sample approximation, not the Bayes factor under another prior. It doesn't use the JZS prior at all, so it can differ a lot from the default analysis. Report the prior-based Bayes factor and treat the BIC value as a rough check.

    BF_{01} \approx \exp\!\left(\dfrac{\Delta BIC_{10}}{2}\right), \quad \Delta BIC_{10} = N \ln\!\dfrac{SS_{\text{within}}}{SS_{\text{total}}} + (k - 1)\ln N =\exp\!\left(\dfrac{90\ln\frac{3690}{3750} + 2\ln 90}{2}\right) = \exp\!\left(\dfrac{-1.45 + 9.00}{2}\right)

    \Delta BIC_{10} = 7.55

    BF_{01} = 43.55

    N is our 90 students and k - 1 = 2 is the extra means the difference model estimates. Its better fit lowers its score by 1.45; its two extra parameters add 2 × ln 90 = 9.00. The net 7.55, halved and exponentiated, is about 43.6 in favor of equal means. That is an approximation, far from the 5.74 the default prior gives below.

  7. 7. Weigh the Evidence with the Default Prior

    The default Bayes factor asks how well each hypothesis predicted the data: equal means, or differences with a Cauchy prior of scale 0.5 on their standardized size. There's no closed-form answer: the formula has to be averaged over the prior numerically, which software does. The value below is that software result, which we reproduced with our own integration.

    BF_{10} = \int_0^\infty (1 + gn)^{-\frac{k - 1}{2}} \left(1 - \dfrac{gn}{1 + gn}\,\eta^2\right)^{-\frac{N - 1}{2}} \pi(g)\, dg =\int_0^\infty (1 + 30g)^{-1} \left(1 - \dfrac{30g}{1 + 30g}\,(.016)\right)^{-44.5} \pi(g)\, dg

    BF_{10} = 0.17

    n is our 30 students per version, k - 1 = 2 halves to the exponent 1, \eta^2 is our .016, and (N - 1)/2 = 44.5. Averaged over the prior \pi(g) with r = 0.5, the difference model predicted the data 0.17 times as well as equal means did.

  8. 8. Turn It Around to Read as Evidence for Equal Means

    A Bayes factor below 1 is easier to read upside down. BF₀₁ puts the null on top.

    BF_{01} = \dfrac{1}{BF_{10}} =\dfrac{1}{0.1741}

    = 5.74

    BF_{10} is our 0.1741. 1 ÷ 0.1741 = 5.74, so equal population means predicted the data 5.74 times better than means differing by the amounts the default prior expects.

  9. 9. Turn the Evidence into How Probable Equal Means Are

    If, before the study, equal and unequal means seemed equally likely, the Bayes factor converts straight into a probability.

    P(H_0 \mid \text{data}) = \dfrac{BF_{01}}{1 + BF_{01}} =\dfrac{5.74}{1 + 5.74}

    = .85

    BF_{01} is our 5.74, and with even prior odds the posterior odds are the same 5.74 to 1. 5.74 ÷ 6.74 = .85.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)
  1. Analyze → Bayesian Statistics → One-way ANOVA. SPSS needs a numeric factor, so first recode app_version into a numeric variable (Transform → Recode into Different Variables) with 1 = audio, 2 = text, 3 = game.
  2. Move stress into Dependent and the recoded version variable into Factor.
  3. Under Bayesian Analysis, choose Use Both Methods (Bayes factor and posterior distribution).
  4. Click Bayes Factor and check the method. SPSS's default, labeled JZS, is the Zellner–Siow approach, which scales the prior differently from the Rouder et al. (2012) ANOVA prior that R's BayesFactor and JASP use.
  5. Click OK.

Reading the Output

Read the group means first, then the Bayes factor, then the classical F and p SPSS prints alongside it.

Because SPSS's default prior isn't the one this page uses, its Bayes factor for these data won't be exactly 5.74 (or its reciprocal, 0.17). Check which way SPSS puts the ratio: the direction is labeled in its output. SPSS also offers Rouder's method in the Bayes Factor dialog, with an inverse-gamma scale parameter b₀ (default 1). That default doesn't correspond to BayesFactor's r = 0.5, so report the method and setting you used. We couldn't run SPSS for this page, so no output is shown; for a result that matches this page exactly, use JASP or R.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R 
library(BayesFactor)

study <- read.csv("data.csv")
study$app_version <- factor(study$app_version, levels = c("audio", "text", "game"))

bf <- anovaBF(stress ~ app_version, data = study)   # rscaleFixed = "medium" (r = 0.5) by default
bf                          # BF10 for app_version against the intercept-only model
1 / extractBF(bf)$bf        # BF01

# Robustness: wider priors
anovaBF(stress ~ app_version, data = study, rscaleFixed = "wide")       # r = sqrt(2)/2
anovaBF(stress ~ app_version, data = study, rscaleFixed = "ultrawide")  # r = 1

# The classical ANOVA, for comparison
summary(aov(stress ~ app_version, data = study))

Bayes factor analysis
--------------
[1] app_version : 0.1741037 ±0.01%

Against denominator:
  Intercept only 
---
Bayes factor type: BFlinearModel, JZS

[1] 5.743705
Bayes factor analysis
--------------
[1] app_version : 0.1046855 ±0.01%

Against denominator:
  Intercept only 
---
Bayes factor type: BFlinearModel, JZS

Bayes factor analysis
--------------
[1] app_version : 0.05887926 ±0.01%

Against denominator:
  Intercept only 
---
Bayes factor type: BFlinearModel, JZS

            Df Sum Sq Mean Sq F value Pr(>F)
app_version  2     60   30.00   0.707  0.496
Residuals   87   3690   42.41

The first block is BF₁₀ = 0.174 for app_version against the intercept-only model, and the line after it is BF₀₁ = 1 / 0.174 = 5.74. The wider priors give BF₁₀ = 0.105 (BF₀₁ = 9.55) and 0.059 (BF₀₁ = 16.98). The ± percentage is the numerical error of the integral, negligible here. The classical ANOVA gives F(2, 87) = 0.707, p = .496. app_version must be stored as a factor, or anovaBF will treat it as a number.

Output from a real run: R 4.6.1 (BayesFactor 0.9.12-4.8), run 2026-09-29.

Python 
import numpy as np
import pandas as pd
import pingouin as pg
from scipy import integrate

study = pd.read_csv("data.csv")

# Classical ANOVA table, with eta squared (np2)
aov = pg.anova(data=study, dv="stress", between="app_version", detailed=True)
print(aov.round(4))

# Default JZS Bayes factor for a balanced one-way design (Rouder et al., 2012, Eq. 12)
def bf10_anova(eta2, N, k, r=0.5):
    n = N / k
    def f(g):
        prior = r / np.sqrt(2 * np.pi) * g**-1.5 * np.exp(-r**2 / (2 * g))
        return (1 + g * n)**(-(k - 1) / 2) * (1 - eta2 * g * n / (1 + g * n))**(-(N - 1) / 2) * prior
    return integrate.quad(f, 0, np.inf)[0]

eta2 = aov.loc[0, "np2"]
bf10 = bf10_anova(eta2, N=len(study), k=3)
print(bf10, 1 / bf10)

        Source      SS  DF       MS       F   p-unc    np2
0  app_version    60.0   2  30.0000  0.7073  0.4958  0.016
1       Within  3690.0  87  42.4138     NaN     NaN    NaN
0.1741036518298427 5.7437049107811555

pingouin has a Bayes factor for t-tests (pingouin.bayesfactor_ttest) but not for ANOVA, so the function above does the integral directly. It only works for equal group sizes; for unequal groups or more complex designs, use R's BayesFactor or JASP. The last line is BF₁₀ = 0.174 and BF₀₁ = 5.74. In a one-way design pingouin's np2 (partial η²) equals η².

Output from a real run: Python 3.9.6 (pingouin 0.5.5, scipy 1.13.1, pandas 2.3.3), run 2026-09-29.

JASP
  1. Open data.csv. Check that app_version is marked nominal and stress continuous (scale).
  2. Open the ANOVA menu and choose ANOVA under the Bayesian heading.
  3. Move stress into Dependent Variable and app_version into Fixed Factors.
  4. Under Additional Options, check that "r scale fixed effects" is 0.5, the default. Change it to 0.707 or 1 for the robustness check.
  5. Optionally, under Order, choose Compare to null model, and under Bayes Factor choose BF₀₁ to print the evidence for equal means directly.

Reading the Output

The main output is the Model Comparison table, one row per model: the Null model (equal means) and the model with app_version. Read it column by column:

  • P(M): each model's prior probability, .50 each by default.
  • P(M|data): its probability after seeing the data. Here that should be about .85 for the null and .15 for app_version, the same as Napkin Math's posterior.
  • BFM: how much the data changed that model's odds against all the others: about 5.74 for the null and 0.17 for app_version.
  • BF₁₀: each model against the one at the top of the table, which is the best model by default. With the null on top, app_version's BF₁₀ is about 0.17, so BF₀₁ = 5.74.
  • error %: the numerical error of the integral; small values can be ignored.

JASP's Bayesian ANOVA is built on R's BayesFactor (van den Bergh et al., 2020) with the same default prior, so it should give the R values above. We couldn't run JASP for this page, so no output is shown.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

Impact

Effect Size

Eta squared (η²) expresses the differences as the share of all the variation in the outcome that lines up with the groups, so it has no units and can be compared across studies that used different scales. It still depends on the design: which groups were chosen, how varied the people are, and how reliably the outcome was measured. It's the same number a classical ANOVA gives, and the Bayes factor is built from it.

\eta^2 = \dfrac{SS_{\text{between}}}{SS_{\text{between}} + SS_{\text{within}}} = \dfrac{60}{60 + 3690} = .016

Of all the differences in students' stress scores, 1.6% line up with which version of the app they used. The other 98% is everything else: exam season, sleep, personality, luck.

The effect size and the Bayes factor answer different questions. η² says how big the differences in this sample are; the Bayes factor says how strongly the data favor some difference over none. A small η² in a small study is weak evidence either way, while here, with 30 per group, the evidence moderately favors none. For a Bayesian estimate of the differences themselves, JASP also summarizes the posterior for each group with a 95% credible interval (van den Bergh et al., 2020). Put together: the observed differences are small, and under the specified prior the data favor exactly equal population means over the alternative; neither result establishes that all differences are too small to matter in practice.

Benchmarks are conventions. Whether a 2-point gap between the most and least stressful versions matters on a 0–40 scale is a question about stress, not statistics.

Benchmarks Eta Squared at a Glance
Below Small
η² < .01
Small
.01 ≤ η² < .06
Medium
.06 ≤ η² < .14
Large
η² ≥ .14

Full benchmarks and caveats ↑

Strength of Evidence

A Bayesian ANOVA doesn't test for significance. It replaces p with the Bayes factor: how many times better one hypothesis predicted the data than the other. BF₁₀ above 1 favors differences, BF₀₁ above 1 favors equal means, and values near 1 mean the data can't tell.

BF₀₁ = 5.74: the stress scores were about 5.74 times better predicted by the model with equal population mean stress than by the alternative, where the means differ by the amounts the default prior expects. The classical ANOVA on the same data gave p = .496, which only says the result isn't surprising under the null.

That's the difference that matters. A large p means "not enough evidence against the null," whether the null is true or the study was too small. A Bayes factor separates the two: it can show evidence for no difference (BF₀₁ well above 1) or show that the data are inconclusive (BF near 1) (van den Bergh et al., 2020). That evidence is relative: it says equal means predicted the data better than this particular alternative with its prior, not that the means are equal outright. A different alternative can give a different answer, which is what the prior-sensitivity table shows. The evidence labels in the guidelines above, such as "moderate" for a Bayes factor between 3 and 10, are rough conventions (Schönbrodt & Wagenmakers, 2018; Kass & Raftery, 1995, drew coarser bands), not thresholds like α = .05. Read BF₀₁ on the same scale as BF₁₀, flipped.

A Bayes factor is not the probability that a hypothesis is true. Posterior odds are the Bayes factor times the prior odds; only with even prior odds does BF₀₁ = 5.74 become an 85% chance that the population means are exactly equal. That's not the same as the app versions being practically equivalent: that claim needs a margin set in advance (say, differences under 2 stress points) and an analysis of that margin, such as the posterior probability that every difference falls inside it.

Benchmarks Bayes Factor at a Glance
Extreme for H₀
BF₁₀ < 1/100
Very Strong for H₀
1/100 ≤ BF₁₀ < 1/30
Strong for H₀
1/30 ≤ BF₁₀ < 1/10
Moderate for H₀
1/10 ≤ BF₁₀ < 1/3
Anecdotal for H₀
1/3 ≤ BF₁₀ < 1
Anecdotal for H₁
1 ≤ BF₁₀ < 3
Moderate for H₁
3 ≤ BF₁₀ < 10
Strong for H₁
10 ≤ BF₁₀ < 30
Very Strong for H₁
30 ≤ BF₁₀ < 100
Extreme for H₁
BF₁₀ ≥ 100

How many times better one hypothesis predicted the data than the other.

Full benchmarks and caveats ↑

Power and Sample Size

Power is the chance that a study detects an effect that's really there. Bayes factors don't have power in that sense, because there's no significance cutoff. Their version is a Bayes factor design analysis: before collecting data, work out how often a planned study would reach convincing evidence, say a Bayes factor of 3 or more, in the right direction, both when the groups truly differ and when they truly don't (Schönbrodt & Wagenmakers, 2018).

With 30 students per version, if the three versions truly worked the same, the study would reach BF₀₁ > 3 about 78% of the time. If the versions truly differed by a medium amount (Cohen's f = 0.25, where f is the standard deviation of the true group means in within-group standard deviations; Cohen, 1992), it would reach BF₁₀ > 3 only about 36% of the time, and about 19% of the time it would point the wrong way, with BF₀₁ > 3. To get BF₁₀ > 3 above 80% of the time, plan for 77 students per version.

Those figures are exact rather than simulated. For a one-way design with equal groups, the default Bayes factor depends on the data only through how much of the variation the groups explain (Rouder et al., 2012), and so only through F. Each chance is then the probability that F lands beyond the point where the Bayes factor crosses 3: under the ordinary F distribution if there's no effect, and under a shifted (noncentral) F distribution, as in a classical power analysis, if there is. For more complex designs, design analysis uses simulation.

Aim for a design that gives compelling evidence whichever hypothesis is true, and plan it before collecting data. The calculator shows both chances; the dashed line is the no-effect case. Try it:

Other Info

Follow-Up and Post Hoc Tests

Post Hoc Tests

  • Bayesian Pairwise Comparisons: Use when the overall Bayes factor favors differences, to see which pairs of groups differ. Here it favors equal means, so there is nothing to follow up; comparing pairs anyway would mostly show how weak each difference is.

References

  • Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56(5), 356–374. https://doi.org/10.1016/j.jmp.2012.08.001
    “These tests are based on multivariate generalizations of Cauchy priors on standardized effects” (Abstract)
    “where R2 is the unadjusted proportion of variance accounted for by the model” (p. 363, Eq. 12)
  • Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. https://doi.org/10.3758/BF03194105
    “Because ANOVA is a special case of regression, the same equation also holds for ANOVA.” (p. 798)
  • Morey, R. D., & Rouder, J. N. (2026). BayesFactor: Computation of Bayes factors for common designs (R package version 0.9.12-4.8). https://doi.org/10.32614/CRAN.package.BayesFactor
  • van den Bergh, D., van Doorn, J., Marsman, M., Draws, T., van Kesteren, E.-J., Derks, K., Dablander, F., Gronau, Q. F., Kucharský, Š., Komarlu Narendra Gupta, A. R., Sarafoglou, A., Voelkel, J. G., Stefan, A., Ly, A., Hinne, M., Matzke, D., & Wagenmakers, E.-J. (2020). A tutorial on conducting and interpreting a Bayesian ANOVA in JASP. L'Année psychologique, 120(1), 73–96. https://doi.org/10.3917/anpsy1.201.0073
    “the current implementation of Bayesian ANOVA in JASP is based on the R package BayesFactor (Morey & Rouder, 2015) which is itself based on the statistical work by Rouder et al. (2012)” (pp. 81–82)
    “For a Bayesian ANOVA, the main component is the Bayesian t-test.” (p. 89)
  • Schönbrodt, F. D., & Wagenmakers, E.-J. (2018). Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bulletin & Review, 25(1), 128–142. https://doi.org/10.3758/s13423-017-1230-y
    “Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness.” (Abstract)
    “The evidential categories serve as a rough heuristic whose main goal is to prevent researchers from overinterpreting the evidence in the data.” (Table 1 and the paragraph before it)
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
  • Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155
  • Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. https://doi.org/10.1080/01621459.1995.10476572
    “these categories are not a calibration of the Bayes factor, but rather a rough descriptive statement about standards of evidence in scientific investigation” (Section 3.2, p. 777)
  • Lee, M. D., & Wagenmakers, E.-J. (2014). Bayesian cognitive modeling: A practical course. Cambridge University Press. https://doi.org/10.1017/CBO9781139087759
  • Jeffreys, H. (1961). Theory of probability (3rd ed.). Clarendon Press.
  • Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863
    “Cohen (1988) has provided benchmarks to define small (η2 = 0.01), medium (η2 = 0.06), and large (η2 = 0.14) effects.” (Eta-squared in between and within-subjects comparisons)

Spot an error? Something still confusing? Tell us at support@statsforpsychology.com.