Differential Item Functioning (DIF)

• You think the gap grows or reverses across scores (nonuniform DIF) Logistic Regression →
• Your items are rated on a scale, not scored right or wrong Ordinal Regression →
• You want to know whether the groups differ overall Independent-Samples t-Test →
• You're asking whether the whole scale measures the same construct in both groups Confirmatory Factor Analysis (CFA) →
• You want each item's difficulty and discrimination, not a group comparison Item Analysis (Classical Test Theory) →

• Don'tCall an item biased because one group gets it right less often.

  InsteadMatch on total score first. A raw gap is impact, which can come entirely from the groups' overall difference.

• Don'tTreat a flagged item as proof that it's unfair.

  InsteadSend it to content experts, who judge whether the extra difficulty is irrelevant to the trait (an idiom in a reasoning item) or part of it.

• Don'tFlag items on p alone.

  InsteadEither adjust p across items (Holm or Bonferroni) or use a rule that also demands a meaningful size, such as ETS's categories. ETS applies its categories at an unadjusted 5% level on purpose, because missing a DIF item costs more than reviewing a clean one (Zwick, 2012). Say which approach you used.

• Don'tLeave the studied item out of the matching total.

  InsteadInclude it, as Holland and Thayer (1988) showed the procedure needs, and consider purifying the total of other flagged items.

• Don'tRead a non-significant MH test as proof that an item works the same for both groups.

  InsteadCheck the interval for Δ and the score-level gaps. Small samples and nonuniform DIF both hide real differences.

• Don'tClassify an item as C from the size of Δ alone.

  InsteadAlso check that |Δ| is significantly larger than 1, as ETS's rule requires. Some software (difR's printout, for one) labels A, B and C by size alone.

• Don'tRead a bigger χ² as a stronger effect.

  InsteadReport an effect size suited to the design, such as Cramér's V, an odds ratio or a model's coefficients. χ² grows with sample size even when the effect doesn't.

• Don'tRead a p-value as the chance that the null hypothesis is true.

  InsteadRead it as: if the null were true, how often data at least this extreme would turn up.

Equation Form

Table 1. Mantel-Haenszel DIF Statistics for the 10 Reasoning Items

Full Text (APA Style)

The equation form above, piece by piece, in this order: how big, how certain, then how surprising.

What Your Data Should Look Like

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the reasoning-test study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

COMPUTE total = SUM(item_01 TO item_10).
EXECUTE.

CROSSTABS
  /TABLES=language BY item_07 BY total
  /FORMAT=AVALUE TABLES
  /STATISTICS=CMH(1)
  /CELLS=COUNT.

library(difR)

reasoning <- read.csv("data.csv")
items <- reasoning[, paste0("item_", sprintf("%02d", 1:10))]
res <- difMH(Data = items, group = reasoning$language, focal.name = "non_native",
             p.adjust.method = "holm")
res

sqrt(res$varLambda)          # SE of ln(alpha_MH), Phillips-Holland
2.35 * sqrt(res$varLambda)   # SE of Delta_MH

Detection of Differential Item Functioning using Mantel-Haenszel method
with continuity correction and without item purification

Results based on asymptotic inference

Matching variable: test score

No set of anchor items was provided

Multiple comparisons made with Holm adjustement of p-values

Mantel-Haenszel Chi-square statistic:

        Stat.   P-value Adj. P
item_01  0.0002  0.9902  1.0000
item_02  2.2238  0.1359  1.0000
item_03  0.0053  0.9417  1.0000
item_04  4.3701  0.0366  0.3292
item_05  1.1163  0.2907  1.0000
item_06  0.1319  0.7165  1.0000
item_07 17.6790  0.0000  0.0003 ***
item_08  0.4734  0.4914  1.0000
item_09  1.0356  0.3088  1.0000
item_10  0.0001  0.9942  1.0000

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Detection threshold: 3.8415 (significance level: 0.05)

Items detected as DIF items:

 item_07

Effect size (ETS Delta scale):

Effect size code:
 'A': negligible effect
 'B': moderate effect
 'C': large effect

        alphaMH deltaMH
item_01  0.9819  0.0429 A
item_02  0.7583  0.6501 A
item_03  1.0034 -0.0080 A
item_04  0.6873  0.8811 A
item_05  1.2381 -0.5019 A
item_06  1.0813 -0.1838 A
item_07  2.1497 -1.7985 C
item_08  0.8752  0.3134 A
item_09  0.8180  0.4721 A
item_10  0.9729  0.0646 A

Effect size codes: 0 'A' 1.0 'B' 1.5 'C'
 (for absolute values of 'deltaMH')

import numpy as np
import pandas as pd
from statsmodels.stats.contingency_tables import StratifiedTable

reasoning = pd.read_csv("data.csv")
items = [f"item_{i:02d}" for i in range(1, 11)]
reasoning["total"] = reasoning[items].sum(axis=1)

# One 2 x 2 table per total score: rows native, non_native; columns right (1), wrong (0)
tables = []
for score, level in reasoning.groupby("total"):
    t = pd.crosstab(level["language"], level["item_07"]).reindex(
        index=["native", "non_native"], columns=[1, 0], fill_value=0)
    if (t.sum(axis=1) > 0).all() and (t.sum(axis=0) > 0).all():
        # keep scores where both groups and both answers appear
        tables.append(t.to_numpy())

st = StratifiedTable(tables)
print(st.oddsratio_pooled)                  # alpha_MH
print(-2.35 * np.log(st.oddsratio_pooled))  # MH D-DIF
print(st.logodds_pooled_se)                 # SE of ln(alpha_MH)
print(st.oddsratio_pooled_confint())
print(st.test_null_odds(correction=True))

2.149696071794304
-1.7985172049485931
0.1799286921724913
(1.5108504644200977, 3.0586701397094185)
pvalue      2.6149218134041696e-05
statistic   17.67904012538838

Post Hoc Tests

• Holm Correction: Use when you test every item on the test, as DIF analyses usually do. With 10 items, the smallest p is multiplied by 10, the next by 9, and so on (step 9 of the Napkin Math). difR: p.adjust.method = "holm".
• Bonferroni Correction: A simpler, more conservative option: judge each item's p against α ÷ the number of items (.05 ÷ 10 = .005 here).

• Related DesignLogistic Regression: Logistic regression DIF (Swaminathan & Rogers, 1990) predicts each item from total score and group. Its group effect asks the same question as the Mantel-Haenszel test, uniform DIF; adding a score-by-group interaction also tests nonuniform DIF, which the MH procedure can miss.
• Related DesignItem Response Theory (IRT): IRT-based DIF compares an item's characteristic curve (its difficulty and discrimination) between groups on the latent trait scale instead of the raw total, and can test nonuniform DIF through the discrimination. Holland and Thayer (1988) showed that under the Rasch model, if the other items have no DIF, matching on the total score (studied item included) gives exactly the constant odds ratio the MH procedure assumes.
• Related DesignPearson Chi-Square Test: The same 2 × 2 table of group by right-or-wrong, but a Pearson chi-square on the whole sample measures impact. The Mantel-Haenszel chi-square computes one table per score level and combines them, so only the gap left after matching counts.
• Related DesignItem Analysis (Classical Test Theory): Classical item analysis describes how hard each item is and how well it separates high from low scorers in one sample. DIF asks whether that behavior is the same in two groups.
• AlternativeMultilevel Logistic Regression: An alternative if “Independent Observations” doesn't hold. Model item responses with random effects for the clusters, and group as a predictor.

• Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129–145). Lawrence Erlbaum Associates.
• Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22(4), 719–748. https://doi.org/10.1093/jnci/22.4.719
• Holland, P. W., & Thayer, D. T. (1986, April). Differential item performance and the Mantel-Haenszel procedure [Paper presentation]. Annual Meeting of the American Educational Research Association, San Francisco, CA, United States. ERIC ED272577. https://eric.ed.gov/?id=ED272577

  “Values of Δ̂MH that are negative correspond to items that the reference group found easier on average than did comparable focal group members.” (p. 9)

  “We prefer the more neutral terms, differential item performance or differential item functioning, (i.e., dif), to item bias since in many examples of items that exhibit dif the term "bias" does not accurately describe the situation.” (p. 1)

  “We will always be comparing two groups of examinees, of which the performance of one, the focal group, F, is of primary interest.” (p. 1)

  “the criterion for matching, X+, includes the studied item,” (p. 16)

• Zwick, R. (2012). A review of ETS differential item functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement (Research Report No. RR-12-08). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2012.tb02290.x

  “MH-D-DIF is negative when the focal group odds of correct response are less than the reference group odds, conditional on the matching variable.” (p. 2)

  “An A item is one in which either the Mantel-Haenszel (MH) chi-square statistic is not significant at the 5% level or MH D-DIF is smaller than 1 in absolute value.” (p. 3)

  “In order to qualify as a C item, the MH D-DIF statistic must be significantly greater than 1 in absolute value at the 5% level and must have an absolute value of 1.5 or more.” (p. 4)

  “the C rule that is typically used for purposes of identifying DIF items for review by committees or possible deletion” (p. 11)

  “The Mantel-Haenszel (1959) approach to DIF analysis, developed by Holland and Thayer (1988), involves the creation of K two-by-two tables, where K is the number of score categories on the matching criterion.” (p. 2)

  “at least 200 members in the smaller group and at least 500 in total are needed for DIF analyses performed at the test assembly phase.” (p. 11)

• Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x

  “the logistic regression procedure is more powerful than the Mantel‐Haenszel procedure for detecting nonuniform DIF and as powerful in detecting uniform DIF.” (Abstract)

• Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. https://www.jstor.org/stable/4615733
• Muff, S., Nilsen, E. B., O'Hara, R. B., & Nater, C. R. (2022). Rewriting results sections in the language of evidence. Trends in Ecology & Evolution, 37(3), 203–210. https://doi.org/10.1016/j.tree.2021.10.009

  “we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing” (Abstract)