Law of Small Numbers

The Law of Small Numbers names a powerful but mistaken intuition: that the Law of Large Numbers applies to small samples as well. Instead of expecting substantial sampling variation, people assume that a handful of observations will reproduce the essential properties of the larger population. This belief makes random outcomes appear explainable, encourages overconfidence in fragile evidence, and leads both ordinary observers and trained researchers to draw conclusions that the data cannot support.

Definition and Core Mechanism

The rule is presented as a belief rather than a genuine statistical law. In the book’s formulation, people behave as if even a small sample must be highly representative of its source population. A short sequence of coin tosses is expected to contain the right local balance of heads and tails; a few research observations are expected to confirm a valid hypothesis; and an unusually high or low performance is treated as evidence of a stable underlying quality. Randomness is therefore imagined as orderly at every scale, even though chance produces extreme results more often in small samples than in large ones.

This intuition reflects a broader tendency of System 1 to construct coherent explanations quickly. System 1 is adept at identifying apparent causal connections, but it is inept when confronted with merely statistical facts that alter probabilities without causing events. A random collection of outcomes does not offer the satisfying causal structure supplied by an event such as a hammer striking an egg. Yet the mind instinctively searches for a reason why a small group differs from the larger population, turning fluctuation into a story about lifestyle, ability, character, or intervention.

The kidney-cancer example makes the error visible. Among the 3,141 counties of the United States, the counties with the lowest incidence of kidney cancer were mostly rural, sparsely populated, and located in traditionally Republican regions of the Midwest, South, and West. The same description also applied to the counties with the highest incidence. Rural cleanliness, fresh food, and low pollution could be made to explain the low rates; poverty, poor medical care, alcohol, and tobacco could be made to explain the high rates. The contradiction disappears once population size is considered: rural counties have small populations, and small populations produce more extreme rates by chance.

Sampling, Research, and Misleading Extremes

The belief became personally consequential in the collaboration between Daniel Kahneman and Amos Tversky. Their first joint study showed that even sophisticated researchers had poor intuitions about sampling effects. Daniel Kahneman had routinely selected samples that were too small, obtained results that made little sense, and then felt embarrassed to discover that the odd findings were artifacts of his research method. Although he taught statistics and knew how to calculate a sample size capable of reducing the risk of failure, he had relied on tradition and intuition rather than computation when planning experiments.

Kahneman and Tversky then tested whether this mistake was widespread. They designed a questionnaire using realistic research situations, including replications of successful experiments. Participants had to choose sample sizes, estimate the risks of failure, and advise hypothetical graduate students. The subjects included mathematically sophisticated researchers, among them authors of statistical textbooks. Their joint article, “Belief in the Law of Small Numbers,” described the resulting error with deliberate irony: intuitive beliefs about random sampling appeared to assert that the Law of Large Numbers applied to small numbers as well.

The practical consequence is not merely that a small study may fail to confirm a true hypothesis. In a population, girls may indeed have a larger average vocabulary than boys of the same age, while a small sample may show an inconclusive difference or even a higher score for boys. That reversal does not disprove the population relationship; it demonstrates sampling variation. Researchers who choose samples that are too small increase the chance of obtaining misleading results, waste time and effort when a true hypothesis is not confirmed, and overinterpret findings that are unlikely to replicate. The recommended corrective is correspondingly direct: statistical intuitions should be treated with suspicion, and impression formation should be replaced by computation whenever possible.

The same bias distorts judgments about performance. In the flight-training example, instructors observed that cadets praised for an excellent aerobatic maneuver often performed worse on their next attempt, while cadets reprimanded for a poor maneuver often improved. The instructors concluded that punishment worked better than reward. The apparent effect was instead explained by regression to the mean: unusually good or bad performances contain a temporary component of luck or noise, which is unlikely to repeat at the same extreme. A golfer who scores 66 when the tournament average is 72 may be both talented and lucky; a golfer who scores 77 may be both weaker and unlucky. The best prediction for the second day must allow luck to return toward average rather than simply repeat the first day’s extreme result.

Confidence, Prediction, and the Neglect of Uncertainty

The Law of Small Numbers is also a bias toward certainty over doubt. When told that 60 percent of a telephone poll of 300 seniors support the president, people tend to remember the political message rather than the sample size. Unless the sample is absurdly small or enormous, many readers respond similarly to a poll of 150 and one of 3,000. System 1 treats the story as the important information and the reliability of its source as background. It constructs a coherent interpretation from what is immediately available and does not naturally preserve the uncertainty introduced by sample size.

This tendency extends to professional prediction. Philip Tetlock’s twenty-year study collected more than 80,000 forecasts from 284 people who earned their living commenting on political and economic trends. The experts were asked to assign probabilities to alternative outcomes, including the continuation of the status quo, an increase, or a decrease. They performed worse than a simple equal-probability baseline, and even in regions they knew best they were not significantly better than nonspecialists. Greater knowledge produced only a slight improvement, while the most knowledgeable forecasters were often less reliable because knowledge strengthened the illusion of predictive skill. Famous experts, especially those in demand, tended to make more flamboyant and more overconfident forecasts.

A similar pattern appeared in the survey of chief financial officers conducted by professors at Duke University. Across 11,600 forecasts of the following year’s Standard & Poor’s index returns, the correlation between the officers’ estimates and the actual market value was slightly less than zero. When the officers predicted that the market would fall, it was slightly more likely than not to rise. Their confidence intervals revealed an even sharper failure of calibration. They expected about 20 percent of outcomes to fall outside an 80 percent confidence interval, but 67 percent did so. The forecasts were not merely inaccurate; they were much narrower and more certain than the evidence justified.

Subjective probability distributions show the same overprecision. A properly calibrated judge should see the true values fall within the stated 98 percent interval on about 98 percent of problems. In many studies, however, the true values fell outside the relevant extreme bounds in about 30 percent of cases. This bias affected both naive and sophisticated judges and persisted even when proper scoring rules supplied incentives for calibration. When people begin with a best estimate and adjust upward or downward to set extreme percentiles, the adjustment is usually insufficient, leaving the interval too tight. The sample of remembered evidence is treated as more complete and representative than it is.

Relation to Other Heuristics and Biases

The Law of Small Numbers belongs to the broader research program developed by Daniel Kahneman and Amos Tversky on judgment under uncertainty. Their “Judgment Under Uncertainty: Heuristics and Biases” described simplifying shortcuts of intuitive thinking and connected roughly twenty systematic biases to those shortcuts. The Law of Small Numbers is especially closely related to representativeness: people expect an event or sample to resemble the process that generated it, even when the sample is too small for such resemblance to be reliable.

The belief also interacts with availability and question substitution. When a complex statistical question is difficult, System 1 may answer an easier related question instead. Rather than asking how much confidence a sample size warrants, a person may ask whether the result forms a vivid, coherent, and familiar story. Rather than considering the base rate and the diagnostic strength of evidence separately, the person may judge whether a description seems representative of a category. Bayesian reasoning requires base rates to remain relevant even when individual evidence is compelling, but associative coherence often makes the evidence seem more diagnostic than it is.

The resulting picture is a mind that runs ahead of the facts. A few observations can generate a rich impression of a person, a research effect, a market, or a historical development. In this sense, the Law of Small Numbers is related to the halo effect and to the general tendency to exaggerate consistency and coherence. System 1 suppresses ambiguity and rarely pauses to maintain incompatible possibilities. System 2 can sustain doubt and calculate the implications of sample size, but doing so requires effort; sliding into a confident story is easier.

Significance and Corrective Discipline

The significance of the Law of Small Numbers lies in the breadth of its reach. It affects epidemiological interpretation, experimental design, judgments of skill, political polling, financial forecasting, and the evaluation of experts. It explains why a striking result may appear persuasive precisely because it is extreme, why a failed replication may be misread as evidence against a sound hypothesis, and why predictions based on vivid descriptions often ignore the reliability of the information.

The corrective is not to distrust every observation or to deny that small samples can contain information. It is to recognize that small samples are intrinsically volatile and that extreme results require more caution, not less. Large samples are more precise; correlations that are imperfect produce regression toward the mean; prior probabilities remain relevant; and predictions should be recorded and evaluated against outcomes rather than protected by retrospective explanation. The book’s final synthesis places the principle among three economical but error-prone heuristics: representativeness, availability of instances or scenarios, and adjustment from an anchor. These shortcuts are often effective, but they generate systematic and predictable mistakes. Understanding the Law of Small Numbers therefore means learning to distinguish a pattern that the data genuinely support from a pattern that the mind has supplied.