ROC Curve Analysis: Interpretation and APA Reporting

The Basics

When to Use It

Use it when you have a score, such as a questionnaire total, and a trusted yes/no diagnosis (the reference standard) for the same people, and you want to know how well the score separates the two groups or where its cutoff should go. For example:

How well does the full PHQ-9 score pick out primary-care patients with major depression?
How well does a 5-minute memory test separate older adults with dementia on a full neuropsychological assessment from those without, and at what score should it refer people on?
Does a teacher-rated ADHD scale distinguish children diagnosed with ADHD in a clinical evaluation from those who aren't, and does it do better than a parent-rated scale given to the same children?

If the cutoff is already fixed and you want its accuracy, with intervals and predictive values, the Sensitivity and Specificity page covers one point of this curve in detail.

Vocabulary You Need

Definition ROC Curve

For a test that gives a score, every cutoff turns the score into a yes/no result with its own sensitivity and specificity. The ROC curve plots all of them at once: the false-positive rate, 1 − specificity, along the bottom and sensitivity up the side, one point per cutoff. A strict cutoff sits near the bottom left (few people flagged, few false alarms); a lenient one near the top right (almost everyone flagged). A score that separates the groups well bows toward the top left corner, where sensitivity is high and false alarms are rare; a score that knows nothing follows the diagonal, where it flags cases and non-cases at the same rate. The curve doesn't depend on the base rate, and it doesn't show which cutoff each point came from, so tables of cutoffs go with it. ROC stands for receiver operating characteristic, a name from radar signal detection. Glossary →

Definition Area under the ROC Curve (

\text{AUC}

)

The area under the ROC curve, from 0 to 1. It has a plain reading (Hanley & McNeil, 1982): pick one person with the condition and one without, at random, and it's the chance the person with the condition has the higher score, with tied scores counting half. .50 means the score does no better than a coin toss; 1 means every person with the condition outscores every person without it. It is the Mann-Whitney U for the group with the condition (the pairs it wins, ties counting half) divided by the number of such pairs, so it equals the common-language effect size, and 2 × AUC − 1 is Somers' d. If your software reports the other group's U, subtract the result from 1. It summarizes the whole curve, including cutoffs nobody would use, so it says how well the score ranks people, not how accurate any one cutoff is, and not what a positive result means at a given base rate. Also called the c statistic or concordance statistic. Glossary →

Definition Diagnostic Cutoff

The threshold that turns a continuous score, such as a questionnaire total, into a yes/no result: at or above the cutoff is positive, below it negative. Every cutoff has its own sensitivity and specificity, and moving it trades one for the other: a lower cutoff catches more true cases but flags more people who don't have the condition. The right cutoff depends on what a missed case and a false alarm each cost. A cutoff chosen because it looked best in one sample tends to look worse in the next, so it should be fixed in advance or checked in new data. Also called a cut-point or threshold. Glossary →

Definition Youden's Index (

J

)

J = sensitivity + specificity − 1. It is 0 for a test that does no better than flipping a coin, however biased the coin, and 1 for a test with no false positives and no false negatives. It falls below 0 only if the test does worse than chance, which usually means the positive and negative codes are swapped. It weights a missed case and a false alarm equally, which is rarely how their costs compare, and it hides the trade-off between the two, so it's reported alongside them rather than instead of them. Across the cutoffs of a scored test, the cutoff with the largest J is a common but data-driven choice. Also called Youden's J statistic. Glossary →

Assumptions

1. A Trustworthy Reference Standard, Applied to Everyone Everyone gets the same accurate reference standard, judged without seeing the test result.

What It Means: Accuracy is measured against the reference standard, so the estimates are only as good as it is. It should classify people correctly, be applied to every participant (or to a random subset), be the same for everyone, and be judged without knowing the index test's result. Errors in the reference standard are counted as errors of the test, which usually makes the test look worse if the two err independently, and better if they tend to err together. Checking mainly the people who test positive (partial verification) inflates sensitivity and deflates specificity. Letting the assessor see the test score inflates both. Using a different standard for positives and negatives can push either way.
Applies To: Design
How to Check: By design, not by a test. Did every participant get the same reference standard, close in time to the index test? Was its assessor blind to the test result, and the test's scorer blind to the diagnosis? Draw the flow of participants, as STARD asks, and count who got which standard.
How Much It Matters: A lot, and it can't be fixed afterward. The whole curve is drawn against the interview. If only people with high PHQ-9 scores are interviewed, depressed patients with low scores never appear, so sensitivity looks too high and specificity too low, and the area and the best cutoff shift in ways that are hard to predict; if the interviewer sees the score, the diagnosis leans toward it. A weaker reference standard shifts every point: Levis et al. (2019) found the PHQ-9's sensitivity 5 to 22 percentage points higher, depending on the cutoff (5 to 15), against semistructured interviews than against fully structured ones.

If It's Violated

Design Change Give every participant the reference standard, or a random sample of those with low scores, and weight the results back up.
Design Change Have the interviewer work without the questionnaire score, and score the questionnaire without the diagnosis.
Proceed with Caveat Say who got which reference standard and when, and which way any shortfall would push the area and the cutoff-by-cutoff estimates.

2. The Right People in the Sample Your sample looks like the people the test will actually be used on.

What It Means: Sensitivity and specificity describe how the test separates the people in the sample. If the sample holds only clear-cut cases and obviously healthy volunteers, the test separates them easily and looks better than it will in practice, where the cases are milder and the non-cases include people with look-alike problems. The fix is to study a consecutive or random series of people from the setting where the test will be used, such as everyone attending a primary-care clinic.
Applies To: Design
How to Check: By design. How were participants found: consecutively, at random, or as known cases and controls? Compare the severity of the cases, and the other problems the non-cases have, with the intended setting. STARD asks for both.
How Much It Matters: A lot. The area measures how much the two groups' scores overlap, so it depends on who is in them. Severe cases from a psychiatric clinic against healthy volunteers push the groups apart and the area toward 1; mild cases against people with look-alike problems such as grief, anxiety or physical illness pull them together. The best cutoff moves too.

If It's Violated

Design Change Recruit everyone (or a random sample) attending the setting where the score will be used, such as consecutive primary-care visits.
Proceed with Caveat Report the severity of the cases and the other problems among the non-cases, so readers can judge how far the curve travels.

3. Independent Observations The people (or other units) in your study don't influence each other.

What It Means: Each person's data comes from a separate, unconnected source, so knowing one person's scores tells you nothing about anyone else's. In designs that measure people more than once, their own scores are meant to be related; it's different people who must be independent. Methods built for clustered data, such as multilevel models and GEE, also allow members of the same group to be related, as long as the analysis names the grouping; then it's the groups that must be independent.
Applies To: Observations
How to Check: By design, not by a test. Did each person contribute their own data, with no clusters such as roommates, couples, therapy groups or classrooms whose members influence each other? If there are clusters, does the analysis model every one of them?
How Much It Matters: Moderate. The area still describes the sample, but DeLong's standard error treats every patient as a separate piece of evidence. Patients counted twice, or clustered in clinics with their own practices, make the interval too narrow.

If It's Violated

Design Change Count each patient once, at a single screening visit.
Robust Estimation With several clinics, resample clinics rather than patients to build the interval.

Limitations

This Is Not Your Test If…

Your test gives only yes or no, at a fixed cutoff Sensitivity and Specificity →
You want to combine several measures into one prediction Logistic Regression →
You're comparing two groups' scores, not judging a test Mann-Whitney U Test →
You're checking two raters against each other, neither the standard Cohen's Kappa →
There's no diagnosis to check your score against Convergent Validity →

What It Doesn't Tell You

Which cutoff to use. The curve shows every trade-off between catches and false alarms; choosing among them depends on what a missed case and an unnecessary interview each cost, which the data can't say.
What a positive result means. An area of .93 says nothing about how many positive screens are confirmed. At a cutoff of 10 and a base rate of 15%, fewer than half are, and that share moves with the base rate (Trevethan, 2017).
Who else. The area describes patients like these 300 in primary care, against this interview. With milder cases, or non-cases with grief or physical illness, the groups' scores overlap more and the curve sinks toward the diagonal.
Whether the best cutoff holds up. A cutoff of 11 won in this sample because of 10 patients who scored exactly 10. Another sample could easily crown 9, 10, 12 or 13: all are within .04 of it.

Common Mistakes

Don'tReport the cutoff with the largest Youden's J from your sample as if its accuracy were settled.

InsteadReport a cutoff fixed in advance too, and check a data-chosen one in new data. Data-driven cutoffs flatter accuracy, especially with under 50 cases (Ewald, 2006).
Don'tChoose the cutoff by Youden's J when a missed case and a false alarm cost different amounts.

InsteadTurn each cutoff's rates into people at your base rate, decide how many false alarms one missed case is worth, and pick the cutoff with the lowest total (see Effect Size).
Don'tLet the software decide which direction of the score counts as positive.

InsteadSet it before the analysis, such as higher PHQ-9 means depressed (pROC's direction = "<"). Automatic direction keeps every area at .50 or above, even for a score that runs backward.
Don'tCompare two scores' areas from the same patients as if they came from separate samples.

InsteadUse a paired comparison, such as DeLong's test, which accounts for the correlation between the two areas.
Don'tReport the area without its interval or the numbers in each group.

InsteadGive the 95% interval, the method, and how many people did and didn't have the condition.
Don'tRead an AUC of .90 as “the test classifies 90% of people correctly.”

InsteadRead it as the chance a random case outscores a random non-case. Accuracy belongs to a cutoff: report sensitivity and specificity there.
Don'tRead sensitivity as the chance that a positive result is correct.

InsteadRead it as the share of people with the condition who test positive. The chance a positive result is correct is the positive predictive value.

Result Decoder

Interpretation Guidelines

Benchmarks Area under the ROC Curve

.50 ≤ AUC < .60 Fail
Barely better than chance at telling the two groups apart.
.60 ≤ AUC < .70 Poor
Someone with the condition outscores someone without it only a little more often than not.
.70 ≤ AUC < .80 Fair
Useful ranking, with a lot of overlap between the groups' scores.
.80 ≤ AUC < .90 Good
At or above the level Nahm (2022) calls generally acceptable.
AUC ≥ .90 Excellent
The groups' scores overlap little. Still check sensitivity, specificity and predictive values at the cutoff you'd use.

Nahm's (2022) labels, which are conventions. Other widely used labels call .80 to .90 excellent and .90 or more outstanding, so check which scheme a paper uses. Our .93 is excellent by them, and close to the .95 Kroenke et al. (2001) reported for the real PHQ-9, yet at the cutoff of 10 fewer than half of positive screens in a clinic like ours are confirmed. The area describes ranking, not what a positive result means. An area well below .50, with an interval that excludes .50, means the score runs the wrong way: check the coding of the diagnosis and the direction of the score. One just under .50 is a score that knows nothing.

More about effect size ↓

How It's Reported

Equation Form

Example

AUC = .93, 95% CI [.89, .97]; at a cutoff of 10, sensitivity = .80 and specificity = .85; Youden's J largest at 11, J = .69

Table 1. Sensitivity and Specificity of the PHQ-9 at Selected Cutoffs

Example

	Depressed Flagged	Not Depressed Flagged	Sensitivity	Specificity	$J$
≥ 5	44 / 45	121 / 255	.98	.53	.50
≥ 8	40 / 45	60 / 255	.89	.76	.65
≥ 9	39 / 45	50 / 255	.87	.80	.67
≥ 10 (standard)	36 / 45	38 / 255	.80	.85	.65
≥ 11 (largest J)	36 / 45	28 / 255	.80	.89	.69
≥ 12	34 / 45	22 / 255	.76	.91	.67
≥ 13	32 / 45	10 / 255	.71	.96	.67
≥ 15	25 / 45	2 / 255	.56	.99	.55

Note. N = 300 primary-care patients (simulated): 45 with major depression on a structured interview, 255 without. A score at or above the cutoff counts as positive. Youden's J = sensitivity + specificity − 1. The cutoff of 10 was fixed in advance; the cutoff of 11 has the largest J in this sample and would need checking in new data.

Table 2. Area under the ROC Curve for the PHQ-9

Example

	Depressed	Not Depressed	AUC	$SE$	95% CI
PHQ-9 total score	45	255	.93	.022	[.89, .97]

Note. The area is the empirical (trapezoid) area, equal to the chance that a randomly chosen depressed patient scores higher than a randomly chosen non-depressed patient, ties counting half. SE and the 95% CI are DeLong's nonparametric estimates.

Full Text (APA Style)

Example

The 45 patients with major depression on the structured interview scored higher on the PHQ-9 (Mdn = 15) than the 255 without (Mdn = 4). The area under the ROC curve was .93, 95% CI [.89, .97] (DeLong): a randomly chosen depressed patient outscored a randomly chosen non-depressed patient in 93% of pairs, counting ties as half (Table 2). At the prespecified cutoff of 10, the PHQ-9 caught 36 of the 45 depressed patients (sensitivity .80) and cleared 217 of the 255 others (specificity .85; Table 1). Youden's J was largest at a cutoff of 11 (sensitivity .80, specificity .89, J = .69); because that cutoff was chosen in this sample, its accuracy is likely to be optimistic.

Give the area with its interval and name the method (DeLong, or a bootstrap), and give the numbers with and without the condition, since the interval rests on both.
Report sensitivity and specificity at the cutoffs readers will use, in a table, and say which cutoff was fixed in advance (STARD item 12a).
If you pick a cutoff from the data, say how (for example, the largest Youden's J) and treat its accuracy as optimistic until it's checked in new data.
Show the ROC curve itself, with the diagonal for a score that knows nothing, and mark the cutoffs you discuss.

Reading Each Piece

The equation form above, piece by piece, in this order: how big, then how certain.

At ≥ 10: sensitivity = .80	The standard cutoff of 10 catches 36 of the 45 depressed patients, 4 in 5, the same result as on the Sensitivity and Specificity page.
Specificity = .85	It correctly clears 217 of the 255 patients without depression; the other 38 get a false alarm.
AUC = .93	Pick one depressed and one non-depressed patient at random: 93 times in 100 the depressed patient has the higher PHQ-9 score (ties counting half). That's excellent by the labels above, but it describes how well the score ranks patients, not how often any cutoff is right. Benchmarks ↑
95% CI [.89, .97]	In patients like these, the chance a depressed patient outscores a non-depressed one is plausibly anywhere from about 89% to 97%. Even the bottom of the range is well above a coin toss.
Youden's J = .69 at ≥ 11	Among all cutoffs, 11 sits highest above the diagonal in this sample. It wins by 10 patients who happened to score exactly 10, which is exactly the kind of luck that won't repeat: treat it as a cutoff to test, not a new standard.

Calculations

The Running Example

Example

The depression-screening study: 300 primary-care patients, each with a PHQ-9 score from 0 to 27 and a structured interview diagnosis of major depression (45 patients) or not (255), done without seeing the score.

There's no hypothesis test here. The question is how well the score separates the two groups and what each cutoff would do, so the page estimates the area under the curve with a 95% confidence interval and lays out sensitivity and specificity cutoff by cutoff. Testing whether the area beats .50, a coin toss, would ask far too little of a screen.

What Your Data Should Look Like

One row per person, with the score in one column and the reference-standard diagnosis in another, coded 1 for the condition and 0 for not. Software sorts people by score and works out sensitivity and specificity at every cutoff; you tell it which diagnosis code is the condition and which direction of the score points toward it (here, higher PHQ-9 means more likely depressed). Keep the raw score rather than a yes/no screen result, since the curve needs every cutoff.

Example

patient	phq9	screen_positive	depressed
1	9	0	0
2	22	1	1
3	10	1	0
4	2	0	0
5	2	0	0
6	3	0	0
… 294 more rows

	n	Mdn	Q₁–Q₃	Mean Rank
Depressed	45	15	12–17	259.94
Not Depressed	255	4	3–7	131.19

Q₁–Q₃ is where the middle half of the scores lies, from the 25th to the 75th percentile (R's default quantile definition).

Simulated data.

Download all 300 rows (data.csv)

Napkin Math

The whole test, one idea at a time. Under each heading, a line says why the step matters, and the green note reads the equation in plain English with the depression-screening study's numbers. Collapse all to read just the headings: they tell the whole story on their own.

1. Turn One Cutoff into One Point on the Graph

Pick a cutoff and you're back on the Sensitivity and Specificity page: some depressed patients are caught and some others get a false alarm. Those two shares are one point on the ROC graph, false alarms along the bottom and catches up the side.

$\text{Se}_{10} = \dfrac{\text{depressed flagged}}{n_1}$ = $\dfrac{36}{45}$

= .80

$1 - \text{Sp}_{10}$ = .149
$\text{Sp}_{10}$ = .85

At $c$ = 10, the PHQ-9 catches 36 of the 45 depressed patients, $\text{Se}_c$ = .80, and flags 38 of the 255 others, $1 - \text{Sp}_c$ = 38 ÷ 255 = .149. So the cutoff of 10 is the point $(1 - \text{Sp}_c,\; \text{Se}_c)$ = (.149, .80): about 15% along and 80% of the way up.
2. Slide the Cutoff through Every Possible Score

Nothing forces the cutoff to be 10. Every score from 0 to 27 could be the line, and each gives its own point. Joined up, the points make the ROC curve, running from (0, 0), where no one is flagged, to (1, 1), where everyone is.

$\text{ROC} = \{(1 - \text{Sp}_c,\; \text{Se}_c) : \text{every cutoff } c\}$ = $c = 8: (.235,\; .89), \quad c = 12: (.086,\; .76)$

$\text{Se}_{8}$ = .89
$1 - \text{Sp}_{8}$ = .235
$\text{Se}_{12}$ = .76
$1 - \text{Sp}_{12}$ = .086

Lower the cutoff $c$ to 8 and the point moves up and to the right: $\text{Se}_c$ = 40 ÷ 45 = .89, but $1 - \text{Sp}_c$ = 60 ÷ 255 = .235. Raise it to 12 and it moves down and to the left: .76 caught, .086 flagged. Our patients' scores run from 0 to 22, which gives 22 points between the two corners.
3. Count How Often a Depressed Patient Outscores a Non-Depressed One

The area under the curve sums up every cutoff at once, and it has a plain meaning: pair each depressed patient with each non-depressed patient and ask how often the depressed one has the higher score. It's the same count as the Mann-Whitney test's U. Why an area? Lower the cutoff from the top score down. Each depressed patient you pass lifts the curve 1/45 of the way up, and the strip under the curve that this lift adds runs from there to the right edge, so its width is the share of non-depressed patients who scored lower than that patient. Add up the strips and you've counted the pairs each depressed patient wins. When a depressed and a non-depressed patient share a score, the curve steps diagonally and takes only half that bit of strip: that's why ties count half.

$\text{AUC} = \dfrac{W + \tfrac{1}{2}T}{n_1 n_0}$ = $\dfrac{10{,}542 + \tfrac{1}{2} \times 241}{45 \times 255} = \dfrac{10{,}662.5}{11{,}475}$

= .93

$n_1 n_0$ = 11475 pairs
$W$ = 10542 pairs
$T$ = 241 pairs

$n_1 n_0$ = 45 × 255 = 11,475 pairs. In $W$ = 10,542 of them the depressed patient scored higher, and $T$ = 241 were tied, which count half. 10,662.5 ÷ 11,475 = .929. The data box's mean ranks give the same area: ranked together, the depressed patients average 259.9 and the others 131.2, and .5 + (259.9 − 131.2) ÷ 300 = .929.
4. Figure Out How Precisely the Sample Pins Down the Area

Another 300 patients would give a somewhat different area. DeLong's method asks how much each patient's own record of wins varies, within each group, and combines the two. With only 45 depressed patients, most of the uncertainty comes from them.

$SE = \sqrt{\dfrac{S_{10}}{n_1} + \dfrac{S_{01}}{n_0}}, \quad S_{10} = \dfrac{\sum_i (V_{10}(x_i) - \text{AUC})^2}{n_1 - 1}, \quad S_{01} = \dfrac{\sum_j (V_{01}(y_j) - \text{AUC})^2}{n_0 - 1}$ = $\sqrt{\dfrac{0.01927}{45} + \dfrac{0.01018}{255}}$

= 0.022

$S_{10}$ = 0.0193
$S_{01}$ = 0.0102

Each depressed patient outscores some share of the 255 others, $V_{10}$ ; those shares vary with $S_{10}$ = 0.0193. Each non-depressed patient is outscored by some share of the 45, $V_{01}$ , varying with $S_{01}$ = 0.0102. 0.0193 ÷ 45 + 0.0102 ÷ 255 = 0.00047, and its square root is 0.022.
5. Give a Range of Plausible Values for the True Area

Instead of a single best guess, give the range of areas that fit the data.

$\text{AUC} \pm z^{*} \times SE$ = $.929 \pm 1.96 \times 0.0216$

= [.89, .97]

$z^{*}$ = 1.96 holds the middle 95% of the normal curve, and $SE$ is 0.0216. 1.96 × 0.0216 = 0.042, so the range is .929 ± .042, from .89 to .97.
6. Put the Area on the Scale of a Rank Correlation

Subtract the pairs ordered the wrong way from the pairs ordered the right way and you get Somers' d, the ordinal measure of association, with the diagnosis as the predictor. It's the same information as the area, on a −1 to 1 scale.

$d_{YX} = 2 \times \text{AUC} - 1$ = $2 \times .929 - 1$

= .86

The area, .929, doubled is 1.858; minus 1 is .86. Of every 100 depressed/non-depressed pairs, about 86 more are ordered the right way than the wrong way.
7. Find the Cutoff That Sits Highest above the Diagonal

Youden's J, sensitivity plus specificity minus 1, is the height of each point above the diagonal. The cutoff with the largest J is a common data-driven choice, but it counts a missed case and a false alarm as equally bad, and it was picked because it looked best in this sample.

$J = \text{Se} + \text{Sp} - 1$ = $.80 + .890 - 1$

= .69

$c^{*}$ = 11
$\text{Sp}_{11}$ = .890
$J_{10}$ = .65

At a cutoff of 11 the PHQ-9 still catches 36 of 45, $\text{Se}$ = .80, and clears 227 of 255, $\text{Sp}$ = .890, so $J$ = .69. At 10 it was .80 + .851 − 1 = .65. The only difference: 10 of our non-depressed patients, and none of the depressed ones, scored exactly 10.

Run It in Software

The code below reads the running example's data from data.csv; save it in the same folder you run the code from.

SPSS (version 30, as documented by IBM)

Analyze → Classify → ROC Curve.
Move phq9 into Test Variable and depressed into State Variable, and type 1 as the Value of State Variable.
Check ROC Curve, With diagonal reference line, Standard error and confidence interval, and Coordinate points of the ROC Curve.
Click Options and check that Larger test result indicates more positive test is selected, then click Continue and OK.

Or paste this syntax:

ROC phq9 BY depressed (1)
  /PLOT=CURVE(REFERENCE)
  /PRINT=SE COORDINATES
  /CRITERIA=CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95)
  /MISSING=EXCLUDE.

Reading the Output

Look at the curve first, then the Area Under the Curve table: the Area (.929), its Std. Error (.022) and the Asymptotic 95% Confidence Interval (.887 to .971). The Asymptotic Sig. (< .001) tests the area against .50, which a screen should clear easily; it isn't the result to lead with.

SPSS's standard error plugs the data into Hanley and McNeil's formula rather than using DeLong's, so its interval can differ from R's in the third decimal (R gives .887 to .972 here).

The Coordinates of the Curve table lists every cutoff with its sensitivity and 1 − specificity. SPSS puts its cutoffs halfway between observed scores, so the row for 9.50 means "positive at 10 or more": sensitivity .800, 1 − specificity .149. Youden's J isn't printed; for each row it's sensitivity − (1 − specificity).

Output laid out from this page's computed results, not captured from a live run; labels and layout may differ slightly in your version.

R

library(pROC)
screening <- read.csv("data.csv")

rocobj <- roc(screening$depressed, screening$phq9,
              levels = c(0, 1), direction = "<")
rocobj                      # the AUC
ci.auc(rocobj)              # DeLong 95% CI

# Sensitivity and specificity at chosen cutoffs (pROC's thresholds sit halfway between scores)
coords(rocobj, x = c(7.5, 9.5, 10.5, 11.5),
       ret = c("threshold", "sensitivity", "specificity", "youden"))

# The cutoff with the largest Youden's J
coords(rocobj, "best", best.method = "youden")

Call:
roc.default(response = screening$depressed, predictor = screening$phq9,     levels = c(0, 1), direction = "<")

Data: screening$phq9 in 255 controls (screening$depressed 0) < 45 cases (screening$depressed 1).
Area under the curve: 0.9292

95% CI: 0.8868-0.9716 (DeLong)

  threshold sensitivity specificity   youden
1       7.5   0.8888889   0.7647059 1.653595
2       9.5   0.8000000   0.8509804 1.650980
3      10.5   0.8000000   0.8901961 1.690196
4      11.5   0.7555556   0.9137255 1.669281

  threshold specificity sensitivity
1      10.5   0.8901961         0.8

Set levels and direction yourself. By default pROC guesses which group is the cases and which way the score points, and it will always pick the direction that puts the area at .50 or above. A threshold of 9.5 means "positive at 10 or more". pROC's youden column is sensitivity + specificity, so subtract 1 for Youden's J: .651 at 10, .690 at 11. For a paired comparison of two scores in the same patients, roc.test(roc1, roc2, method = "delong").

Output from a real run: R 4.6.1 (pROC 1.19.1), run 2026-09-29.

Python

import pandas as pd
from sklearn.metrics import roc_auc_score, roc_curve

screening = pd.read_csv("data.csv")
y, score = screening["depressed"], screening["phq9"]
print(round(roc_auc_score(y, score), 4))

fpr, tpr, thresholds = roc_curve(y, score, drop_intermediate=False)   # keep every cutoff; positive when score >= threshold
roc = pd.DataFrame({"cutoff": thresholds, "sensitivity": tpr,
                    "specificity": 1 - fpr, "J": tpr - fpr})
print(roc[roc["cutoff"].between(8, 12)].round(3).to_string(index=False))
print(roc.loc[roc["J"].idxmax()].round(3).to_dict())

0.9292
 cutoff  sensitivity  specificity     J
   12.0        0.756        0.914 0.669
   11.0        0.800        0.890 0.690
   10.0        0.800        0.851 0.651
    9.0        0.867        0.804 0.671
    8.0        0.889        0.765 0.654
{'cutoff': 11.0, 'sensitivity': 0.8, 'specificity': 0.89, 'J': 0.69}

scikit-learn gives the area and the curve but no standard error or interval. For DeLong's interval, use R's pROC; a stratified bootstrap (resampling depressed and non-depressed patients separately) is the usual Python route.

Output from a real run: Python 3.9.6 (scikit-learn 1.6.1, pandas 2.3.3), run 2026-09-29.

Impact

Effect Size

The area under the curve is the page's effect size. It describes how much the two groups' scores overlap, as a probability: the chance that a randomly chosen person with the condition outscores a randomly chosen person without it, ties counting half (Hanley & McNeil, 1982). It doesn't depend on the base rate or the scale of the score, so it can be compared across tests and studies, provided the samples are alike.

\text{AUC} = \dfrac{W + \tfrac{1}{2}T}{n_1 n_0} = \dfrac{10{,}542 + 120.5}{45 \times 255} = .93

Pair every depressed patient with every non-depressed patient: in about 92 pairs out of 100 the depressed patient has the higher PHQ-9 score (10,542 of 11,475), and about 2 are tied (241); counting ties as half gives .93. That's the same number as the Mann-Whitney test's common-language effect size, and 2 × .93 − 1 = .86 is Somers' d with the diagnosis as the predictor.

The labels above are conventions (Nahm, 2022), and an "excellent" area can still go with a weak screen in practice. The area averages over every cutoff, including ones nobody would use, and it says nothing about what a positive result means at a given base rate. What a clinic would weigh is the cutoff table: at 10, 36 of 45 depressed patients caught and 38 of 255 others sent for an interview they didn't need.

Choosing a cutoff is a separate decision. Youden's J is largest at 11 here (.69, against .65 at 10), but only because 10 non-depressed patients and no depressed ones happened to score exactly 10. Ewald (2006) found that cutoffs picked this way make tests look better than they are, especially with fewer than 50 cases; we have 45. And J counts a missed depression and a false alarm as equally bad. If a positive screen only leads to a 20-minute interview, a lower cutoff that catches more cases may be the better choice.

To weigh the two mistakes, turn the rates into people. In 1,000 primary-care patients, 15% of them depressed, a cutoff of 8 misses about 17 depressed patients and sends 200 others to an interview they don't need; 10 misses 30 and sends 127; 13 misses 43 and sends 33. If one missed depression is as bad as five unneeded interviews, 8 costs about 17 × 5 + 200 = 285, 10 about 277 and 13 about 248, so 13 is the cheapest of the three. If a miss is as bad as twenty, 8 wins. Formally, the cheapest cutoff is where the curve's slope equals (cost of a false alarm ÷ cost of a miss) × (people without the condition ÷ people with it). The rates come from 45 cases, so the same caution about sampling luck applies.

Comparing two scores. To ask whether one questionnaire separates the groups better than another given to the same patients, compare their areas with a paired test such as DeLong's, which allows for the two areas being correlated because they come from the same people (DeLong, DeLong & Clarke-Pearson, 1988). Two separate intervals that overlap don't settle it either way.

Benchmarks Area under the ROC Curve at a Glance

Fail

.50 ≤ AUC < .60

Poor

.60 ≤ AUC < .70

Fair

.70 ≤ AUC < .80

Good

.80 ≤ AUC < .90

Excellent

AUC ≥ .90

Full benchmarks and caveats ↑

Uncertainty

This page doesn't test a null hypothesis, so there's no p in the report. Software will test whether the area is above .50, and that test is the Mann-Whitney test in another form: both ask whether one group's scores tend to be higher. For a screening questionnaire it asks far too little. A score barely better than a coin toss passes it in a large enough sample; the useful questions are how large the area is and how precisely you know it.

Our area of .93 is far above .50 (Mann-Whitney z = 9.22, p < .001), which tells us nothing we couldn't see from the interval, .89 to .97. Its lower end already rules out anything close to a coin toss.

The interval here is DeLong's (DeLong, DeLong & Clarke-Pearson, 1988), which is what R's pROC reports (Robin et al., 2011). SPSS's nonparametric standard error plugs the data into Hanley and McNeil's formula instead; with these data the two agree to three decimals (0.022). Both lean on the normal curve, so with few cases, or an area close to 1, a bootstrap interval is a sensible check. Say which you used.

The same goes for comparing two scores: a significant difference between two areas says they differ, not by how much or whether it matters. Report the difference with its interval.

Precision and Sample Size

Power is the chance that a study detects an effect that's really there. For a screen, detecting that the area beats .50 is too low a bar, so the planning question is precision: how many people you need for the interval around the area to be narrow enough to be useful. STARD asks studies to report their intended sample size and how it was set (Bossuyt et al., 2015).

The catch, as on the Sensitivity and Specificity page, is that the interval rests on both groups, and the smaller one, usually the people with the condition, limits it. Hanley and McNeil (1982) showed how to run the area's standard error forward for planning, from the area you expect and the numbers in each group. For planning, when there are no data yet, their closed-form (negative exponential) approximation fills in the formula's two variance terms from the area you expect; that's what's used below. Here it's more conservative than the data-based standard error SPSS reports: with our data it would give 0.027 rather than 0.022. With other score distributions it can go either way, so treat it as a rough guide.

Suppose we expected an area of about .90 and wanted the 95% interval to be no wider than ±.05. Hanley and McNeil's formula asks for about 68 depressed patients, and at a base rate of 15% that means about 453 patients in all (385 without depression). For ±.05 around an area of .80, it's about 117 depressed patients, 780 in all. Our 45 depressed patients gave ±.04 with DeLong's method; the planning formula is more cautious and would have predicted ±.05.

Plan the number of people with the condition first, then divide by the base rate you expect to get the total. If you'll also report sensitivity and specificity at a fixed cutoff, plan for those too, and recruit the larger total. There's no calculator for the area yet; the Sensitivity and Specificity page's planner covers one cutoff's sensitivity.

Other Info

Follow-Up and Post Hoc Tests

None. This page estimates rather than tests, so there are no follow-up comparisons to run.

Related Methods

General CaseSensitivity and Specificity: one point on this page's curve: a score with a single fixed cutoff gives one sensitivity and one specificity. ROC analysis repeats that calculation at every cutoff, and the Sensitivity and Specificity page is where each point's intervals and predictive values are worked out.
Its TestMann-Whitney U Test: the same pair counting, used as a test. Split people by diagnosis and compare their scores: U divided by n₁ × n₂ is the area under the curve (Hanley & McNeil, 1982), the same quantity as the probability of superiority, often called the common-language effect size when ties count half. Here U = 10,662.5 of 11,475 pairs, .929. Its test of no difference is the test of AUC = .50.
Alternative MeasureSomers' D: the same comparison on a −1 to 1 scale. With the diagnosis as the yes/no predictor and the score as the outcome, d = 2 × AUC − 1 (Newson, 2002): .86 here.
Related DesignLogistic Regression: the way to combine several measures, such as a questionnaire score, age and past episodes, into one predicted probability. The ROC curve and its area then judge how well those predictions separate the groups.

References

Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. https://doi.org/10.1148/radiology.143.1.7063747
“the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject” (Abstract)

“this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic” (Abstract)
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845. https://doi.org/10.2307/2531595
“When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data.” (Abstract)

“The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates.” (Abstract)
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 77. https://doi.org/10.1186/1471-2105-12-77
“CIs are computed with Delong's method” (Confidence intervals)
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3
Ewald, B. (2006). Post hoc choice of cut points introduced bias to diagnostic research. Journal of Clinical Epidemiology, 59(8), 798–801. https://doi.org/10.1016/j.jclinepi.2005.11.025
“The use of data-driven cut points exaggerates test performance in many simulated data sets, and this bias probably affects many published diagnostic validity studies.” (Abstract, Conclusion)
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x
“ROC analysis showed that the area under the curve for the PHQ-9 in diagnosing major depression was 0.95” (Results)
Nahm, F. S. (2022). Receiver operating characteristic curve: Overview and practical use for clinicians. Korean Journal of Anesthesiology, 75(1), 25–36. https://doi.org/10.4097/kja.21209
“Generally, an AUC ≥ 0.8 is considered acceptable.” (Table 4 and the text below it)
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., Kressel, H. Y., Rifai, N., Golub, R. M., Altman, D. G., Hooft, L., Korevaar, D. A., Cohen, J. F., & the STARD Group. (2015). STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies. BMJ, 351, h5527. https://doi.org/10.1136/bmj.h5527
“Definition of and rationale for test positivity cut-offs or result categories of the index test, distinguishing pre-specified from exploratory” (Table 1, item 12a)

“Whether clinical information and index test results were available to the assessors of the reference standard” (Table 1, item 13b)

“Whether participants formed a consecutive, random, or convenience series” (Table 1, item 9)
Newson, R. (2002). Parameters behind “nonparametric” statistics: Kendall's tau, Somers' D and median differences. The Stata Journal, 2(1), 45–64. https://doi.org/10.1177/1536867X0200200103
Levis, B., Benedetti, A., Thombs, B. D., & the DEPRESsion Screening Data (DEPRESSD) Collaboration. (2019). Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: Individual participant data meta-analysis. BMJ, 365, l1476. https://doi.org/10.1136/bmj.l1476
Trevethan, R. (2017). Sensitivity, specificity, and predictive values: Foundations, pliabilities, and pitfalls in research and practice. Frontiers in Public Health, 5, 307. https://doi.org/10.3389/fpubh.2017.00307
“the four metrics should not be regarded as unquestionably valid and fixed attributes of a screening test: the values that are entered into the cells of Figure 1 depend on how stringent the screening test is and the prevalence of the target condition in the sample of people used in the analysis.” (Determining Sensitivity, Specificity, and Predictive Values)