p-hacking
Twenty tests of nothing at all. About two in three times, one of them comes out “significant”.
Try it
Run 20 experiments on pure noise and see what “works”.
Each tile is a new experiment: 30 people per group, both groups drawn from identical data.
Simulated. Two-sided z-test on two groups of 30 drawn from the same normal distribution. A tile is highlighted when p is below 0.05.
What is going on
A p-value below 0.05 means a result at least this extreme would turn up by chance less than 1 time in 20 if nothing were going on. Run 20 independent tests of nothing and the chance that at least one passes is 1 minus 0.95 to the 20th power, about 64%.
The problem is not testing; it is telling the story only of the test that worked. Trying several outcomes, subgroups or stopping points and reporting the winner has the same effect, even when nobody sets out to cheat.
How to spot it
- One surprising finding picked from many measured outcomes or subgroups.
- No mention of how many comparisons were tried.
- Ask: how many things were tested, and was the analysis planned in advance?
Jelly beans cause acne
The webcomic xkcd 882 makes the point with 20 colours of jelly bean tested against acne: one colour is “significant” and the headline writes itself. It is a cartoon, and the real-world version is documented by Simmons, Nelson and Simonsohn, who showed that ordinary flexibility in how data is analysed can make almost any effect appear at p < .05.
Every experiment in the demo is simulated: both groups are drawn from exactly the same distribution, so any “difference” is luck.