Unstatistical

Base-rate neglect

A test that is right 95% of the time can still be wrong about most of the people it flags.

Try it

Set how rare a condition is and see how many positives are real.

Hypothetical numbers. Defaults: 1 in 1,000 have it, with a test that is wrong 5% of the time for healthy people.

■ has it, tests positive   ■ healthy, tests positive   ■ has it, missed   ■ healthy, tests negative

What is going on

How likely something is before you test is the base rate. If few people have the condition, even a small false-positive rate produces more false alarms than the true cases it finds.

The cure is to count people instead of juggling percentages. Imagine 1,000 people, work out how many have it, how many of those test positive, and how many of the healthy test positive too. Then compare the two groups of positives.

How to spot it

  • A claim of accuracy with no mention of how common the thing is.
  • Screening results quoted as “95% accurate”.
  • Ask: out of 1,000 people like me, how many would test positive, and how many of them actually have it?

Doctors and a 5% false-positive test

In a much-repeated 1978 survey, Casscells, Schoenberger and Graboys asked staff and students at Harvard teaching hospitals about a test with a 5% false-positive rate for a disease affecting 1 in 1,000 people. Only a small minority gave the right answer of about 2%; the commonest answer was 95%.

Gigerenzer and Hoffrage later showed that presenting the same problem as natural frequencies (“out of 1,000 people…”) greatly improves people’s reasoning. That is the method the demo uses. The numbers you set in the demo are hypothetical.

Sources