Item Analysis (Classical Test Theory)

The Basics

When to Use It

Use it when one group of people took a test made of several items scored right or wrong, and you want to know which items are doing their job before you trust, reuse or revise the test. It works item by item: how many people got each one right, whether the people who got it right also did well on the rest of the test, and, for multiple-choice items, which wrong options people chose. For example:

Which questions on a ten-item research methods quiz are working, and which need fixing?
Do the items of a new digit-span memory screen get harder in the order they're given?
Which items on a 20-item emotion-recognition test fail to separate people who score high on the rest of the test from people who score low?

Vocabulary You Need

Definition Item Difficulty (

p

)

The share of people who got an item right, from 0 to 1. Despite the name, a higher value means an easier item: p = .90 means 90% answered correctly. It describes the item in this group of test-takers, not the item alone; the same item looks harder in a weaker group. Items near .50 leave the most room to tell people apart (a little higher on multiple-choice items, where some right answers are guesses), while items almost everyone passes, or almost everyone fails, say little about who knows more. On multiple-choice items, blind guessing alone would give about .25 with four options, so a p near or below that is a warning sign: an attractive wrong option, or a wrong key. It's sometimes called the item's p-value, which has nothing to do with a significance test's p. Glossary →

Definition Item Discrimination (

D

)

How much better the high scorers on a whole test do on one item than the low scorers. The classic index, D, is the proportion correct in a top-scoring group minus the proportion correct in a bottom-scoring group, so it runs from −1 to 1. The groups are usually the top and bottom 27% (Kelley, 1939). The item-total (point-biserial) correlation measures the same idea using everyone. A positive value means the item agrees with the rest of the test; a value near zero means it doesn't tell strong from weak test-takers apart; a negative value usually means a wrong key, two defensible answers, or an item that measures something else. Very easy and very hard items can't discriminate much, however well they're written, because nearly everyone lands on the same side. Glossary →

Definition Item-Total Correlation (

r_{it}

)

The correlation between one item and the scale total. The corrected version, which is what software usually reports, leaves the item out of the total (correlating it with the remaining items) so it isn't correlated with itself. A high value means the item rises and falls with the rest of the scale; a value near zero means it measures something else, is mostly noise, or barely varies (as when nearly everyone gives the same answer), and a negative value usually means a reverse-keyed item wasn't recoded. It is read alongside “alpha if item deleted,” which shows how the scale's alpha would change without that item. Glossary →

Definition Cronbach's Alpha (

\alpha

)

An estimate of the reliability of a total score made by adding up several items, from a single sitting: k/(k − 1) × (1 − the sum of the item variances ÷ the total score's variance). When the items go together, the total varies much more than its parts added up separately, and alpha approaches 1. It equals the reliability only if every item measures the one construct equally strongly (tau-equivalence) and the items' errors are uncorrelated. With uncorrelated errors but unequal loadings it is a lower bound; with correlated errors, such as shared wording or a response style, it can overestimate. It rises with the number of items, it isn't a fixed property of a scale, and it says nothing about whether the items measure only one thing. It can even be negative, which usually means an item wasn't recoded. For items scored 0 or 1, alpha is the same number as KR-20 (Kuder & Richardson, 1937). Glossary →

Definition Distractor

One of the incorrect options on a multiple-choice item, written to look plausible to someone who doesn't know the answer. A working distractor draws some test-takers, mostly low scorers. One that almost nobody chooses (a common working cutoff is fewer than 5%) does no work and just shrinks the number of real choices. One that draws high scorers more than low scorers is often arguably correct, or the key is wrong. Distractor analysis tabulates who chose each option, overall and in the high- and low-scoring groups. Glossary →

Assumptions

1. Correct Scoring Key Each item has one right answer, and your key records it.

What It Means: Every item has exactly one defensible correct option, and the answer key names that option. Item statistics judge each item against the rest of the test, so a wrong key makes a good item look backwards, and an item with two defensible answers splits the strongest test-takers between them. Either error also drags down the test's reliability.
Applies To: Items
How to Check: Have a second content expert answer the items without seeing the key. Then read the item statistics: a negative item-total correlation or D, or a distractor chosen more often by high scorers than by low scorers, is the classic sign of a miskeyed or ambiguous item.
How Much It Matters: A lot, and it's the first thing item analysis catches. In the methods quiz study, one key error turned an item that works well (r_it = .42 once rescored) into one that discriminates backwards (r_it = −.23), and pulled the whole test's KR-20 down from .71 to .60.

If It's Violated

Correction Correct the key for the miskeyed item, rescore everyone, and rerun the analysis.
Correction If an item has two defensible answers, credit both for this sitting, or drop the item from the total.
Design Change Have a second expert answer every item blind to the key before the test is given.

2. One Underlying Trait Your items all measure one and the same trait.

What It Means: A single common factor is enough to reproduce how the items go together (with several constructs, each construct's items measure one trait). Coefficients built on a one-factor model, such as omega and alpha, and item-level models such as IRT, describe how well the items measure one trait only if there really is one. A scale made of distinct parts, such as worry and bodily symptoms, breaks it; score and analyze each part separately, or use a model with several factors.
Applies To: Items
How to Check: Start from the scale's content and theory. Then fit a one-factor model (or, with several constructs, one factor per construct) and look at its fit (the chi-square test, RMSEA, SRMR) and at the residual correlations it leaves between pairs of items. For right/wrong or yes/no items, fit that model to tetrachoric correlations (in lavaan, declare the items ordered, which uses WLSMV) or as a one-factor IRT model with its own fit statistics (M2, RMSEA, SRMSR); fit indices from ordinary Pearson correlations are misleading for 0/1 items. An exploratory factor analysis or parallel analysis can suggest how many factors there are. A high omega or alpha is not evidence of unidimensionality: several correlated factors can produce one.
How Much It Matters: A lot. Every item is judged against the total, so the total has to stand for one thing. If a quiz mixes, say, statistics and ethics, an item can look weak simply because it belongs to the smaller topic, and KR-20 understates how consistent each topic is on its own.

If It's Violated

Alternative Test Exploratory Factor Analysis (EFA). Check how many dimensions the items measure before judging them against one total.
Design Change If the test has distinct sections, run the item analysis within each section, against that section's total.

3. Independent Observations The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: Moderate. The item statistics still describe the group, but students who worked together or copied answers make items look more consistent than they are, and Feldt's interval for KR-20 comes out too narrow.

If It's Violated

Design Change Give the test under conditions where each student answers alone.
Proceed with Caveat If collaboration was possible (an online, untimed quiz), say so when reporting the statistics.

Limitations

This Is Not Your Test If…

You want item properties that don't depend on this group's ability Item Response Theory (IRT) →
You want to know whether an item works differently for two groups Differential Item Functioning (DIF) →
You want to know how many things your items measure Exploratory Factor Analysis (EFA) →
Your items are rating scales and you only need reliability Cronbach's Alpha →
You want to know how well a cutoff score classifies people Sensitivity and Specificity →

What It Doesn't Tell You

How hard the items are in general. A p of .54 describes these 250 students. In a class that had covered sampling more thoroughly, item 3 would look easier, and in a class whose knowledge varied less, every item would discriminate less.
Whether the quiz measures research methods knowledge. Items can hang together and still measure reading speed or test-taking savvy; that's a question of validity, which needs evidence from outside the quiz.
Why an item fails. The numbers flagged items 7 and 8, but it took reading the options to see that one was miskeyed and the other had two defensible answers.
Whether small differences between items are real. With 67 students in each group, D for an item near the middle can shift by about .10 to .15 either way in another sample, so items a few points apart shouldn't be ranked against each other.

Common Mistakes

Don'tRead item difficulty as “how hard the item is” in general.

InsteadRead it as the share of this group who got it right. The same item gives a higher p in a stronger class.
Don'tDrop a very easy item just because its discrimination is low.

InsteadAn item almost everyone gets right can't discriminate much. Keep it if it checks an essential idea or covers content the test needs.
Don'tCorrelate each item with a total that still includes it.

InsteadUse the corrected item-total correlation, with the item left out of the total, especially on short tests.
Don'tJudge an item from one class of 30 students.

InsteadTreat item statistics from small groups as rough. D from 27% groups of 8 students can swing by .30 or more.
Don'tDrop an item for a low or negative D before checking its key and wording.

InsteadLook at which option the high scorers chose. A miskeyed or ambiguous item can often be fixed rather than thrown out.
Don'tDrop every item with a lower item-total correlation or a higher “alpha if item deleted.”

InsteadJudge the item's content too, and expect small gains in alpha to shrink in new data.
Don'tRead alpha as the share of items that measure the construct, or as proof the scale is unidimensional.

InsteadRead it as an estimate of the total score's reliability, and check dimensionality separately, with a factor analysis.

Result Decoder

Interpretation Guidelines

Benchmarks Discrimination Index (

D

)

D < 0 Negative
Low scorers do better on the item than high scorers. Check the answer key first, then the wording.
0 ≤ D < .20 Poor
The item barely separates strong from weak test-takers. Revise it or drop it.
.20 ≤ D < .30 Marginal
Some separation, but the item usually needs revising.
.30 ≤ D < .40 Reasonably Good
The item works, with room to improve.
D ≥ .40 Very Good
The item clearly separates strong from weak test-takers.

These bands are long-standing classroom-testing conventions for D, not tested standards, and they assume a test meant to spread people out. On a short test D runs high, because each item is part of the total it's judged against; read it alongside the corrected item-total correlation and the item's content. For that correlation, a common rule of thumb is to check the key of any item below 0, look hard at any item below about .20, and treat .30 or more as working well. That too is a convention, not a test, and it shifts with test length and with how spread out the group is.

More about effect size ↓

How It's Reported

Equation Form

Example

p = .22 to .94, r_it = −.23 to .48, D = −.06 to .82, KR-20 = .60, 95% CI [.52, .67]

Table 1. Item Statistics for the Ten-Item Research Methods Quiz

Example

	$p$	$r_{it}$	$D$	KR-20 if Item Deleted
Dependent variable in a caffeine study (too easy)	.94	.20	.13	.59
What random assignment protects against	.73	.40	.61	.54
Sampling with equal chances	.54	.40	.75	.54
Reading a negative correlation	.64	.39	.70	.54
Why a trial has a placebo group	.48	.40	.82	.54
Within-subjects design	.83	.44	.52	.54
What a correlation supports (flawed distractor)	.56	.04	.34	.64
Test-retest reliability (miskeyed)	.22	−.23	−.06	.68
Operational definition	.36	.32	.64	.56
Who is blind in a double-blind study	.63	.48	.78	.52

Note. N = 250. p = proportion answering correctly. r_it = corrected item-total (point-biserial) correlation, each item with the total of the other nine. D = proportion correct in the upper group (total ≥ 8, n = 67) minus the lower group (total ≤ 4, n = 67), each 26.8% of students. For working items D runs higher than r_it: it compares only the extremes, and on a ten-item quiz the total that forms the groups includes the item. KR-20 for all ten items = .60, 95% CI [.52, .67] (Feldt). Item 8 is shown as scored with the original key (B); rescored with C, its r_it = .42 and KR-20 = .71.

Table 2. Distractor Analysis for Items 7 and 8

Example

	All	Upper 27%	Lower 27%	$U - L$
Item 7, A: Screen time causes anxiety	.09	.00	.24	−.24
Item 7, B: Either could be affecting the other	.25	.25	.13	.12
Item 7, C: There is no relationship	.10	.01	.24	−.22
Item 7, D: They tend to rise together (key)	.56	.73	.39	.34
Item 8, A: Compare two groups' means	.09	.01	.21	−.19
Item 8, B: Correlate odd and even items (key)	.22	.16	.22	−.06
Item 8, C: Correlate two sittings of the same test	.62	.81	.45	.36
Item 8, D: Ask experts to rate the items	.08	.01	.12	−.10

Note. Proportion of all students (N = 250), the upper group (n = 67) and the lower group (n = 67) choosing each option. U − L is each option's discrimination: it should be positive for the key and negative for every distractor. Key = the option the answer key credits. Item 7 reads “A study finds that students who spend more time on screens report more anxiety. What does this correlation support?” Its option B (“Either could be affecting the other”) is defensible as written, and draws more of the upper group than the lower. Item 8's key (B, correlating odd and even items) describes split-half reliability; the correct answer is C.

Full Text (APA Style)

Example

Scores on the ten-item quiz averaged 5.93 (SD = 2.09), and the share of students answering each item correctly ranged from .22 to .94. Seven items discriminated well, with corrected item-total correlations of .32 to .48 and discrimination indices of .52 to .82. Three did not. Item 1 was answered correctly by 94% of students (r_it = .20, D = .13). On item 7, a quarter of the upper group chose a distractor that is defensible as written, and the keyed answer was almost unrelated to the rest of the quiz (r_it = .04); crediting both answers gave r_it = .34. Item 8 correlated negatively with the rest of the quiz (r_it = −.23, D = −.06) because it was miskeyed. Reliability of the total was modest, KR-20 = .60, 95% CI [.52, .67] (Feldt). Rescored with the correct answer, item 8 had r_it = .42, and KR-20 rose to .71.

Give each item's p, corrected r_it and D in a table, and say how the upper and lower groups were formed (here, total ≥ 8 and ≤ 4, 67 students each).
Say whether item-total correlations are corrected (item left out of the total), since uncorrected ones run higher on a short test.
Report KR-20 with its interval and the number of items; reliability rises with test length.
Say what you changed after the analysis (a corrected key, a dropped item) and report the statistics both before and after.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how certain.

p = .22 to .94	The items run from very hard (item 8, answered “correctly” by 22%) to very easy (item 1, 94%). Item 1 is too easy to tell students apart: 94% of them land on the same side of it. Item 8's .22 is below the .25 that blind guessing would give on four options: fewer students chose the key than chance alone would predict, the first hint that something is wrong with the item or its key.
r_it = −.23 to .48	Seven items correlate .32 to .48 with the rest of the quiz: students who get them right tend to do better overall. Item 7 (.04) barely tracks the rest, and item 8 (−.23) runs backwards: students who got it “right” did worse on the other nine items.
D = −.06 to .82	For seven items the top group beat the bottom group by 52 to 82 percentage points, very good by the usual conventions. Item 1 manages only .13 (almost everyone gets it), and item 8 is negative. Item 7's .34 looks fine, but it's flattered: on a ten-item quiz the item itself helps decide who lands in the top group, and its corrected r_it of .04 tells the truer story. For the working items D also runs higher than r_it because it compares only the extremes, so judge each on its own yardstick. Benchmarks ↑
KR-20 = .60	About 60% of the spread in quiz totals is estimated to be consistent (true-score) variance, and 40% error from the particular items asked. These items differ in how strongly they separate students, so .60 likely understates the reliability somewhat. Even so, it's modest, typical of a ten-item quiz, and far short of the .90 Nunnally (1978) asked for before making decisions about one student. Whether that consistency is research methods knowledge, rather than, say, reading speed, is a validity question.
95% CI [.52, .67]	The quiz's reliability in the population these students represent is plausibly anywhere from .52 to .67: clearly above zero, clearly below the .70 often asked of a research measure.
KR-20 without item 8 = .68	Taking item 8 out raises reliability from .60 to .68, the clearest sign that it measures something opposite to the rest of the quiz as keyed. Before dropping it, the distractor analysis asks why.
KR-20 after rescoring = .71	With the key fixed, item 8 becomes one of the better items and the quiz's reliability rises to .71. Item 7 still needs rewriting before the quiz is used again.
62% chose C on item 8	Most of the class, and 81% of the top group, chose C on item 8. When the option the strongest students favor isn't the key, check the key before blaming the students. On item 7 a quarter of the top group chose distractor B, which is defensible as written; that item needs rewording, not rekeying.

Calculations

The Running Example

Example

The methods quiz study: 250 undergraduates, ten four-option multiple-choice items scored right or wrong, with every chosen option kept so the wrong answers can be examined too.

There's no null hypothesis to test here. Item analysis describes each item (how many got it right, how well it separates strong from weak students) and estimates the reliability of the total, with a confidence interval for that reliability. The decisions it feeds, such as fixing a key or rewriting an option, are judgments about the items, not significance tests.

What Your Data Should Look Like

One row per person and one column per item, scored 1 (right) or 0 (wrong). For multiple-choice items, keep the option each person chose as well, along with the answer key, so the wrong answers can be analyzed and the items rescored if the key turns out to be wrong. Blank answers are usually scored as wrong; say so if you do.

Example

student	q1	q2	q3	q4	q5	q6	q7	q8	q9	q10	q1_choice	q2_choice	q3_choice	q4_choice	q5_choice	q6_choice	q7_choice	q8_choice	q9_choice	q10_choice
1	1	1	0	0	1	1	0	0	0	0	A	C	A	B	A	B	B	C	A	D
2	1	1	0	1	0	1	1	0	0	0	A	C	D	D	C	B	D	D	B	B
3	1	0	0	1	0	1	1	0	1	0	A	A	A	D	B	B	D	D	C	B
4	1	1	1	1	1	1	0	0	0	1	A	C	B	D	A	B	B	C	D	A
5	1	1	0	0	0	0	1	0	0	1	A	C	A	A	B	D	D	C	B	A
6	1	1	0	0	0	1	0	1	0	0	A	C	D	B	B	B	A	B	A	C
… 244 more rows

Item	p	Item–Total r
Item 1	.94	.20
Item 2	.73	.40
Item 3	.54	.40
Item 4	.64	.39
Item 5	.48	.40
Item 6	.83	.44
Item 7	.56	.04
Item 8	.22	−.23
Item 9	.36	.32
Item 10	.63	.48
Total M (SD)	5.93 (2.09)

N = 250. The total adds the 10 items; each item's r is with the total of the other 9 (the corrected item–total correlation).

p is the proportion scoring 1 on the item (a right answer, or a yes).

Simulated data.

Download all 250 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the methods quiz study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Count How Many Students Got Each Item Right

The simplest question first. The share of students who answer an item correctly tells us whether it's pitched at the class or so easy that everyone gets it.

$p_i = \dfrac{\text{number who answered item } i \text{ correctly}}{N}$ = $\dfrac{135}{250}$

= .54

$p_1$ = .94
$p_8$ = .22

Item 3 asks which sampling method gives everyone an equal chance of being chosen. 135 of our 250 students picked the keyed answer, so $p_3$ = 135 ÷ 250 = .54, near the middle. Item 1 is the opposite: 236 got it right, $p_1$ = .94. Item 8, as keyed, only .22.
2. Check Whether Students Who Got It Right Did Better on the Rest of the Quiz

A good item is answered correctly more often by students who know the material. Compare the rest score (the total on the other items) for students who got the item right and wrong, and turn the gap into a correlation. Because the item is left out of the total, this point-biserial correlation is the corrected item-total correlation, $r_{it}$ .

$r_{pb} = \dfrac{M_1 - M_0}{s_n}\sqrt{p\,q}$ = $\dfrac{6.07 - 4.60}{1.84}\sqrt{.54 \times .46}$

= .40

$M_1$ = 6.07
$M_0$ = 4.60
$s_n$ = 1.84
$r_{it,7}$ = .04
$r_{it,8}$ = −.23

On the other nine items, the 135 students who got item 3 right averaged $M_1$ = 6.07 and the 115 who missed it $M_0$ = 4.60. The gap, 1.47 points, is 0.80 of the rest scores' standard deviation ( $s_n$ = 1.84). Multiply by √(.54 × .46) = .50 and you get .40, item 3's $r_{it}$ .
3. Compare the Top and Bottom Scorers Directly

The same idea with plain counting. Take the students with the highest and lowest totals and ask how much more often the top group got the item right. The usual groups are the top and bottom 27%. Kelley (1939) showed that this split makes the two groups most clearly different. Smaller groups are more extreme but give shakier proportions; larger ones give steadier proportions but sit closer together.

$D = p_U - p_L$ = $\dfrac{62}{67} - \dfrac{12}{67}$

= .75

$p_U$ = .93
$p_L$ = .18
$D_8$ = −.06

Our upper group is the 67 students with 8 or more right, and the lower group the 67 with 4 or fewer, each 26.8% of the class, as close to 27% as the whole-number totals allow. 62 of the upper group got item 3 right ( $p_U$ = .93) and 12 of the lower group ( $p_L$ = .18), so $D$ = .93 − .18 = .75.
4. Estimate How Consistent the Total Score Is

The items are meant to be added into one score. KR-20 asks how much of the spread in totals comes from students really differing rather than from the luck of which items they happened to know. For items scored 0 or 1 it is Cronbach's alpha, and like alpha it is a lower bound to the reliability unless every item measures the trait equally strongly.

$KR_{20} = \dfrac{k}{k - 1}\left(1 - \dfrac{\sum p_i q_i}{\sigma_X^2}\right)$ = $\dfrac{10}{9}\left(1 - \dfrac{2.00}{4.36}\right)$

= .60

$\sum p_i q_i$ = 2.00
$\sigma_X^2$ = 4.36

$k$ = 10 items. Each item's variance is $p_i q_i$ (for item 3, .54 × .46 = .25), and the ten add to 2.00. The totals have variance $\sigma_X^2$ = 4.36, dividing by N to match the $p_i q_i$ (the usual SD of 2.09 divides by n − 1). 2.00 ÷ 4.36 = .46, so 1 − .46 = .54, and 10/9 × .54 = .60.
5. Give a Range of Plausible Values for the Quiz's Reliability

Another 250 students would give a somewhat different KR-20. Feldt's (1965) interval, the same one used for Cronbach's alpha, shows how far the population value could plausibly be from ours.

$\left[\,1 - (1 - \hat{\alpha})\,F_{.975},\;\; 1 - (1 - \hat{\alpha})\,F_{.025}\,\right]$ = $1 - .399 \times 1.195,\;\; 1 - .399 \times 0.825$

= [.52, .67]

$F_{.975}$ = 1.195
$F_{.025}$ = 0.825

Here $\hat{\alpha}$ is our KR-20, .601 (for 0/1 items KR-20 is Cronbach's alpha), so $1 - \hat{\alpha}$ is .399. $F_{.975}$ = 1.195 and $F_{.025}$ = 0.825 cut off the top and bottom 2.5% of an F distribution with 249 and 2,241 df (n − 1 and (n − 1)(k − 1)). That gives 1 − .477 = .52 and 1 − .329 = .67.
6. See Which Items Pull Reliability Down

Recomputing KR-20 with each item left out shows which items help the total and which hurt it. An item whose removal raises reliability is working against the others.

$KR_{20}^{(-i)} = KR_{20} \text{ of the other } k - 1 \text{ items}$ = $KR_{20}^{(-8)},\; KR_{20}^{(-7)}$

$KR_{20}^{(-8)}$ = .68
$KR_{20}^{(-7)}$ = .64
$KR_{20}^{(-10)}$ = .52

Dropping any of the seven good items lowers KR-20 below .60 (to between .52 and .56). Dropping item 1 barely matters (.59). Dropping item 7 raises it to .64, and dropping item 8 to .68: with nine items, the quiz would be more reliable than with ten.
7. Look at Which Wrong Answers the Strong Students Chose

A negative discrimination has a few possible causes. Tabulating every option for the top and bottom groups shows which one it is (a wrong key, two defensible answers, or a confusing item).

$D_{\text{option}} = p_U(\text{option}) - p_L(\text{option})$ = $.81 - .45$

= .36

$D_{B}$ = −.06
$D_{7B}$ = .12
$p_{C}$ = .62

On item 8, option C was chosen by 54 of the upper group ( $p_U$ = .81) and 30 of the lower group ( $p_L$ = .45): $D_C$ = .81 − .45 = +.36, the pattern a correct answer shows. The keyed option B went the other way, .16 against .22. C, “correlate scores from two sittings of the same test,” is the textbook definition of test-retest reliability; the key was wrong.
8. Fix the Key and Check Again

Item analysis is a loop. Once a problem has a clear cause, fix it, rescore everyone and rerun the numbers to confirm the fix worked.

$x_{8j}' = \begin{cases} 1 & \text{student } j \text{ chose C} \\ 0 & \text{otherwise} \end{cases}$

$p_8'$ = .62
$r_{it,8}'$ = .42
$KR_{20}'$ = .71

Rescoring item 8 with C as the key, 154 students (.62) get it right. Its corrected correlation with the rest of the quiz jumps from −.23 to .42, in line with the good items, and the quiz's KR-20 rises from .60 to .71.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

Analyze → Scale → Reliability Analysis. Move q1 to q10 (the scored 0/1 items) into Items and leave Model on Alpha; for 0/1 items, alpha is KR-20.
Under Statistics, tick Item and Scale if item deleted (under Descriptives for). For Feldt's interval for KR-20, also tick Intraclass correlation coefficient, with Model Two-Way Mixed and Type Consistency. Then Continue and OK. The item means are each item's p.
For D and the distractor analysis, compute each student's total, then mark the upper and lower groups: Transform → Compute Variable (total = SUM(q1 TO q10)), then Transform → Recode into Different Variables (8 thru 10 → 2 “upper”, 0 thru 4 → 1 “lower”, everything else missing). Choose the cutoffs from the frequency table of total so each group holds about 27% of students.
Analyze → Descriptive Statistics → Crosstabs, with an item's _choice column in Row(s) and group in Column(s); under Cells, tick Column percentages.

Or paste this syntax:

RELIABILITY
  /VARIABLES=q1 q2 q3 q4 q5 q6 q7 q8 q9 q10
  /SCALE('Methods quiz') ALL
  /MODEL=ALPHA
  /STATISTICS=DESCRIPTIVE SCALE
  /SUMMARY=TOTAL
  /ICC=MODEL(MIXED) TYPE(CONSISTENCY) CIN=95.

COMPUTE total = SUM(q1 TO q10).
RECODE total (8 THRU 10=2) (0 THRU 4=1) (ELSE=SYSMIS) INTO group.
VALUE LABELS group 1 'lower' 2 'upper'.
EXECUTE.

MEANS TABLES=q1 TO q10 BY group /CELLS=MEAN COUNT.
CROSSTABS /TABLES=q7_choice q8_choice BY group /CELLS=COUNT COLUMN.

Reading the Output

Read each item's Mean (its p), then its Corrected Item-Total Correlation, then Cronbach's Alpha if Item Deleted. D is the upper group's mean minus the lower group's, from the MEANS table; the crosstabs show which options each group chose.

Reliability Statistics

Cronbach's Alpha	N of Items
.601	10

Item-Total Statistics

	Scale Mean if Item Deleted	Scale Variance if Item Deleted	Corrected Item-Total Correlation	Cronbach's Alpha if Item Deleted
q1	4.99	4.132	.204	.592
q2	5.20	3.512	.398	.545
q3	5.39	3.396	.397	.542
q4	5.29	3.445	.393	.544
q5	5.45	3.397	.395	.542
q6	5.10	3.616	.435	.543
q7	5.38	4.051	.039	.636
q8	5.71	4.616	-.230	.677
q9	5.57	3.563	.321	.563
q10	5.30	3.297	.481	.519

q8_choice * group Crosstabulation (% within group)

	lower	upper
A	20.9%	1.5%
B	22.4%	16.4%
C	44.8%	80.6%
D	11.9%	1.5%

The Item Statistics table (not shown) gives each item's mean, which is p: .944 for q1 down to .220 for q8. The negative corrected correlation for q8 is the flag, and the crosstab shows why: the upper group chose C, not the keyed B. SPSS has no built-in upper-lower D, which is why the groups are made by hand.

The Intraclass Correlation Coefficient table's Average Measures row is alpha again (.601), and its 95% confidence interval, [.523, .671], is Feldt's interval for KR-20.

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

library(psych)

quiz <- read.csv("data.csv")
items <- quiz[, paste0("q", 1:10)]
res <- alpha(items, warnings = FALSE)
round(res$total[1:4], 3)                             # raw_alpha = KR-20
round(res$item.stats[, c("mean", "r.drop")], 3)      # p and corrected r_it
round(res$alpha.drop[, "raw_alpha", drop = FALSE], 3) # KR-20 if item deleted

# D from upper and lower groups
total <- rowSums(items)
group <- ifelse(total >= 8, "upper", ifelse(total <= 4, "lower", NA))
round(colMeans(items[group %in% "upper", ]) - colMeans(items[group %in% "lower", ]), 3)

# Option proportions, from the chosen letters
key <- c(1, 3, 2, 4, 1, 2, 4, 2, 3, 1)   # A = 1, B = 2, C = 3, D = 4
choices <- sapply(quiz[, paste0("q", 1:10, "_choice")], match, table = LETTERS[1:4])
mc <- score.multiple.choice(key, choices, score = TRUE, short = FALSE)
round(mc$item.stats[7:8, ], 3)
round(prop.table(table(quiz$q8_choice, group), 2), 3)

 raw_alpha std.alpha G6(smc) average_r
     0.601     0.596    0.62     0.129

     mean r.drop
q1  0.944  0.204
q2  0.728  0.398
q3  0.540  0.397
q4  0.640  0.393
q5  0.484  0.395
q6  0.832  0.435
q7  0.556  0.039
q8  0.220 -0.230
q9  0.360  0.321
q10 0.628  0.481

    raw_alpha
q1      0.592
q2      0.545
q3      0.542
q4      0.544
q5      0.542
q6      0.543
q7      0.636
q8      0.677
q9      0.563
q10     0.519

    q1     q2     q3     q4     q5     q6     q7     q8     q9    q10
 0.134  0.612  0.746  0.701  0.821  0.522  0.343 -0.060  0.642  0.776

          key     1     2     3     4 miss      r   n  mean    sd
q7_choice   4 0.092 0.248 0.104 0.556    0  0.276 250 0.556 0.498
q8_choice   2 0.088 0.220 0.616 0.076    0 -0.038 250 0.220 0.415

   group
    lower upper
  A 0.209 0.015
  B 0.224 0.164
  C 0.448 0.806
  D 0.119 0.015

raw_alpha is KR-20. The mean and sd that res$total also holds are for the average item, not the total (0.5932 × 10 = 5.93). r.drop is the corrected item-total correlation. In score.multiple.choice, the columns 1 to 4 give the share choosing each option, and r is the item's correlation with the total including the item, which is why item 7 shows .276 here against its corrected .039.

Left to its default, alpha() warns that q8 correlates negatively with the first principal component and suggests check.keys = TRUE. Don't take the suggestion for a miskeyed test item: it would reverse-score q8, treating a wrong answer as right. Fix the key instead.

Output from a real run: R 4.6.1 (psych 2.6.5), run 2026-10-03.

Python

import pandas as pd
import pingouin as pg

quiz = pd.read_csv("data.csv")
items = quiz[[f"q{j}" for j in range(1, 11)]]
total = items.sum(axis=1)

p = items.mean()
r_it = items.apply(lambda x: x.corr(total - x))          # corrected point-biserial
upper, lower = total >= 8, total <= 4
D = items[upper].mean() - items[lower].mean()
kr20_if_deleted = {c: pg.cronbach_alpha(data=items.drop(columns=c))[0] for c in items}
pd.DataFrame({"p": p, "r_it": r_it, "D": D, "KR20_if_deleted": kr20_if_deleted}).round(3)

pg.cronbach_alpha(data=items)       # KR-20 and Feldt's 95% interval

group = pd.Series(pd.NA, index=quiz.index, dtype="object", name="group")
group[upper], group[lower] = "upper", "lower"
pd.crosstab(quiz["q8_choice"], group, normalize="columns").round(3)

         p   r_it      D  KR20_if_deleted
q1   0.944  0.204  0.134            0.592
q2   0.728  0.398  0.612            0.545
q3   0.540  0.397  0.746            0.542
q4   0.640  0.393  0.701            0.544
q5   0.484  0.395  0.821            0.542
q6   0.832  0.435  0.522            0.543
q7   0.556  0.039  0.343            0.636
q8   0.220 -0.230 -0.060            0.677
q9   0.360  0.321  0.642            0.563
q10  0.628  0.481  0.776            0.519

(0.6009259633081027, array([0.523, 0.671]))

group      lower  upper
q8_choice
A          0.209  0.015
B          0.224  0.164
C          0.448  0.806
D          0.119  0.015

For 0/1 items, pg.cronbach_alpha is KR-20, and its interval is Feldt's. Choose the cutoffs for upper and lower from total.value_counts() so each group holds about 27% of students.

Output from a real run: Python 3.9.6 (pingouin 0.5.5, pandas 2.3.3), run 2026-10-03.

Impact

Effect Size

There's no single effect size in item analysis. Each item gets its own size measures, and the one with conventional benchmarks is the discrimination index, D: the share of the top-scoring group who got the item right minus the share of the bottom-scoring group.

D = p_U - p_L = .93 - .18 = .75

On item 3, 93% of the students with 8 or more right answered correctly, against 18% of the students with 4 or fewer. That 75-point gap is what a working item looks like.

D is easy to compute by hand and easy to explain, but it has two quirks. It can't be large for an item almost everyone gets right or wrong: item 1, answered correctly by 94%, couldn't have reached much above .20 however well it was written. And on a short test it runs high, because the item helps decide who's in the top group; item 7's D of .34 hides a corrected item-total correlation of .04 (Henrysson, 1963, makes the same point about item-total correlations). On a short test, you can form the upper and lower groups on the rest score (the total without the item) instead: done that way, item 7's D drops to about .05, and miskeyed item 8's falls from −.06 to about −.26, a much plainer warning. Even on rest-score groups, D runs higher than r_it for most items, because it compares only the extremes. Read D next to r_it, and judge an item by what it's for: a very easy item that checks an essential idea can earn its place. The benchmarks are classroom-testing conventions, not laws.

Benchmarks Discrimination Index (

D

) at a Glance

Negative

D < 0

Poor

0 ≤ D < .20

Marginal

.20 ≤ D < .30

Reasonably Good

.30 ≤ D < .40

Very Good

D ≥ .40

Full benchmarks and caveats ↑

Uncertainty

Item analysis is usually reported without p-values. You could test whether each item's correlation with the rest of the test is above zero, but a working item should clear that bar easily, and with ten items you'd be running ten tests at once. The questions that matter are how big each item's statistics are and how much they would shift in another sample.

Item 3's corrected item-total correlation of .40, from 250 students, has a 95% interval of roughly .29 to .50, comfortably positive. Item 7's .04 has an interval of about −.08 to .16: we can't tell it apart from an item that doesn't discriminate at all.

For the total score, the confidence interval for KR-20 (.52 to .67 here) does the work a test would. None of these numbers says whether an item is good enough to keep; that's a judgment about the test's purpose and the item's content.

Precision and Sample Size

Power is the chance that a study detects an effect that's really there. Item analysis doesn't test an effect, so the planning question is precision: how many people you need before each item's statistics are stable enough to act on. Item statistics wobble more than people expect, and the discrimination index, which uses only the top and bottom 27%, wobbles most.

An item-total correlation near .30 has a 95% interval of about .11 to .47 with 100 students, .18 to .41 with our 250, and .22 to .38 with 500. D is noisier: with 27% groups of 67 students (250 in all), its standard error is about .08, so a D of .30 could plausibly be anywhere from about .14 to .46.

Plan for a few hundred test-takers before dropping or rewriting items on the numbers alone. With a class of 30, treat item statistics as prompts to reread the items, not verdicts. The planner below shows the interval for a correlation, which is what each item's corrected item-total correlation is. Try it:

Other Info

Follow-Up and Post Hoc Tests

None. This page estimates rather than tests, so there are no follow-up comparisons to run.

Related Methods

Related DesignCronbach's Alpha: KR-20, the reliability this page reports, is Cronbach's alpha for items scored 0 or 1, and the corrected item-total correlation and alpha if item deleted are the same item statistics the alpha page reports for rating scales.
General CasePearson Correlation: the point-biserial correlation is Pearson's r with one variable coded 0 and 1, and the corrected item-total correlation is that r between an item and the total of the other items.
Alternative MeasureItem Response Theory (IRT): IRT describes the same two item properties, difficulty and discrimination, as parameters of a model linking each item to the underlying trait. If the model fits, item parameters from different groups differ only by a shift and stretch of the trait scale, which linking removes, so they don't hinge on how able one group happened to be. It needs a larger sample than item analysis.
Related DesignDifferential Item Functioning (DIF): DIF analysis asks a follow-up question about each item: do two groups of equal overall ability (say, two language backgrounds) have different chances of getting it right?
AlternativeExploratory Factor Analysis (EFA): An alternative if “One Underlying Trait” doesn't hold. Check how many dimensions the items measure before judging them against one total.

References

Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17–24. https://doi.org/10.1037/h0057123
“The writer has stated that twenty-seven per cent should be selected at each extreme to yield upper and lower groups which are most indubitably different with respect to the trait in question.” (p. 17)
Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151–160. https://doi.org/10.1007/BF02288391
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
“is shown to be the mean of all split-half coefficients resulting from different splittings of a test” (Abstract, p. 297)
Henrysson, S. (1963). Correction of item-total correlations in item analysis. Psychometrika, 28(2), 211–218. https://doi.org/10.1007/BF02289618
“The biserial correlation between an item and the total test of which the item is a part tends to be misleadingly high when used in item analysis, since the item is included in the total test.” (Abstract, p. 211)
Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 9, 40. https://doi.org/10.1186/1472-6920-9-40
“Discriminating power is an index that measures the difference in the proportion of responses between the upper and lower 27% of examinees” (Methods)

“Non-functioning options were defined as ones that were chosen by fewer than 5% of examinees and those with a positive option discrimination statistic.” (Abstract, Methods)

“Items are considered discriminating if the index for the correct response is positive and the same statistic for the distractors is negative” (Methods)
Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529
“the incorrect options, which are also called the distractors” (Abstract)
Feldt, L. S. (1965). The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty. Psychometrika, 30(3), 357–370. https://doi.org/10.1007/BF02289499
Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93–103. https://doi.org/10.1177/014662168701100107
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74(1), 107–120. https://doi.org/10.1007/s11336-008-9101-0
“Alpha is a lower bound to the reliability, in many cases, even a gross underestimate” (Section 1, Introduction)
Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103. https://doi.org/10.1207/S15327752JPA8001_18
“(a) that it is a fixed property of the scale, (b) that it measures only the internal consistency of the scale, (c) that higher values are always preferred over lower ones, and (d) that it is restricted to the range of 0 to 1” (Abstract)
Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412. https://doi.org/10.1111/bjop.12046
“Unidimensionality is a requisite for all the models considered here.” (A practical alternative to alpha)