The law of small numbers
In a league of identical schools, the winners and the losers are mostly the small ones.
Try it
Rank 100 identical schools and see who ends up on top and bottom.
Simulated. 100 schools of different sizes (10 to 400 pupils). In every school each pupil independently has a 50% chance of passing. There is no difference in quality at all.
What is going on
The average of a few observations bounces around much more than the average of many. So the highest and lowest rates in any ranking tend to belong to the smallest groups, whether or not those groups are really different.
People expect small samples to look like the population they come from. They do not. Tversky and Kahneman called that expectation a belief in the “law of small numbers”.
How to spot it
- League tables and “best and worst” lists where the groups differ in size.
- A dramatic rate based on a handful of cases.
- Ask: how many observations is this based on, and does the extreme result come from the smallest groups?
Small schools and kidney cancer
Howard Wainer documents two classic cases. A study of US school performance found that small schools were over-represented among the best, which helped motivate large grants to break up big schools; they were also over-represented among the worst. Likewise the counties with the lowest and highest kidney cancer rates were both small, rural counties.
The schools in the demo are simulated and identical by construction.