Law of Large Numbers

The Law of Large Numbers is the statistical principle that repeated observations from a stable population tend to approach the population’s underlying proportions as the number of observations increases. Its decisive significance in the book is psychological as much as mathematical: people readily notice striking differences in small samples, but often fail to recognize that sample size determines how much trust those differences deserve.

Definition and Statistical Logic

The principle is illustrated through an urn containing equal numbers of red and white marbles. If a patient counter repeatedly draws four marbles, records the composition, replaces them, and repeats the process, the result of two red and two white marbles appears roughly six times as often as the extreme results of four red or four white. The exact outcomes remain random, but repeated sampling displays a regular mathematical pattern. Unlike an egg struck by a hammer, the sampling process offers no satisfying causal image; its predictability lies in aggregate frequencies rather than in a visible mechanism.

Sample size also governs the frequency of extreme results. When one counter draws four marbles at a time and another draws seven, the smaller samples produce homogeneous results, all red or all white, far more often. Over many trials, the four-marble counter observes such extreme samples about eight times as frequently as the seven-marble counter: the expected percentages are 12.5 percent and 1.56 percent. Nothing causes one counter to obtain more extreme compositions. The difference follows from the mathematics of sampling.

Kidney Cancer Counties

The principle clarifies the apparently puzzling distribution of kidney cancer across United States counties. Counties with the highest incidence tend to be rural, sparsely populated, and concentrated in traditionally Republican states in the Midwest, the South, and the West. That pattern invites a causal story involving poverty, limited medical care, diet, alcohol, or tobacco. Yet the same rural way of life cannot plausibly explain both the counties with exceptionally high incidence and those with exceptionally low incidence.

The relevant feature is population size. Rural counties provide smaller samples of the national population, just as four-marble draws are smaller than seven-marble draws. Small samples are more likely to depart sharply from the average in either direction, so they are especially likely to contain both very high and very low recorded cancer rates. The county’s small population neither causes nor prevents cancer; it merely makes an extreme observed rate more probable.

The apparent difference is therefore a sampling artifact rather than a stable epidemiological fact. If the analysis is repeated the following year, the general pattern should persist—extreme values will still be more common in small counties—but the particular counties with high incidence need not remain the same. A result that looks like a characteristic of a place may thus be a temporary accident of sample size.

Sample Size and Human Judgment

The principle exposes a recurring weakness in intuitive reasoning. Readers presented with the kidney-cancer example immediately search for a cause and do not initially treat “sparsely populated” as the central clue. Even people who know that large samples are generally more reliable can be surprised by how much more often samples of four produce extreme outcomes than samples of seven. Statistical knowledge may be available in memory without being spontaneously applied to the case at hand.

The same insensitivity appears in a hospital comparison involving the proportion of boys born on different days. Most people judge the probability of obtaining more than 60 percent boys to be similar in a small hospital and a large one because the same percentage seems equally representative of the general population. Sampling theory gives a different prediction: the small hospital should experience more days above that threshold, because a smaller sample is more likely to stray from the underlying 50 percent than a large sample is.

A related urn problem compares a five-ball sample containing four red and one white ball with a twenty-ball sample containing twelve red and eight white. Intuition is often drawn to the first sample because its red proportion is higher, but, assuming equal prior probabilities, the correct posterior odds that the urn contains two-thirds red balls rather than one-third red balls are 8 to 1 for the 4:1 sample and 16 to 1 for the 12:8 sample. The larger sample provides stronger evidence even though its observed proportion is less extreme. The mind privileges vivid proportion; statistical reasoning also counts the amount of evidence.

Consequences for Inference and Prediction

The Law of Large Numbers supports a broader discipline of inference. Bayesian reasoning requires base rates to remain in view even when a specific case supplies vivid evidence, and it requires the diagnostic value of that evidence to be assessed rather than simply felt. In the cab problem, 85 percent of the city’s cabs are Green and 15 percent are Blue, while a witness identifies the cab as Blue with 80 percent reliability. These statistical base rates and case-specific testimony must be combined rather than replaced by an intuitive story. Causal base rates are different because they describe how the individual case came about; statistical base rates describe the population to which it belongs and are often underweighted when vivid details are present.

The principle also underlies regression to the mean. A golfer who scores 66 when the tournament average is 72 has probably displayed both above-average talent and unusually good luck; a golfer who scores 77 has probably displayed both weaker performance and bad luck. On the second day, talent may persist, but luck is not predictable, so the best forecast moves each player’s performance toward the average rather than simply repeating the first day’s result. The same logic prevents observers from mistaking a fluctuation followed by a more ordinary result for proof that a reward or punishment caused the change.

Reference-class forecasting applies the same corrective to large projects and public predictions. Instead of building a forecast entirely from the promising details of one case, it begins with the outcomes of comparable cases. In the curriculum project described in the book, an informal survey suggested a 40 percent failure rate and a completion time of seven to ten years. That outside-view baseline provided a more realistic anchor than the team’s confident inside-view estimate, which extrapolated from early progress and ignored unknown disruptions. Large collections of comparable cases reveal the risks that a single attractive narrative leaves out.

The principle also sets a standard for evaluating forecasters. In a twenty-year study, Philip Tetlock collected more than 80,000 predictions from 284 people who earned their living commenting on political and economic trends. The experts performed worse than a simple strategy that assigned equal probabilities to three possible outcomes, and specialists were not significantly better than nonspecialists even in regions they knew best. Greater knowledge produced only a slight improvement and could also strengthen overconfidence. The lesson is not that expertise is worthless, but that prediction must be judged across many cases, not by the persuasive coherence of isolated successes.

Calibration and Significance

Large samples make it possible to distinguish genuine predictive skill from luck. A judge who assigns subjective probability intervals is properly calibrated when the stated proportions correspond to the frequencies of outcomes: for example, approximately 98 percent of true values should fall between the one-percent and ninety-nine-percent limits. Studies instead found that actual values fell outside such extreme limits in about 30 percent of cases, showing that both naive and sophisticated judges commonly offered intervals that were too narrow. Repeated observations expose this systematic overconfidence more reliably than a memorable individual forecast can.

The same pattern appeared in a survey of 11,600 forecasts by chief financial officers of large corporations concerning the following year’s Standard and Poor’s index. The correlation between their estimates and the actual market value was slightly below zero, yet the participants did not seem to recognize that their forecasts were nearly useless. Their 80 percent confidence intervals contained too many surprises: 67 percent of outcomes fell outside them, more than three times the expected 20 percent. Aggregated evidence therefore reveals not merely occasional error but a stable mismatch between confidence and accuracy.

The Law of Large Numbers does not make every individual outcome predictable, nor does it supply a cause for every pattern. Its force is more restrained and more profound: random observations can display dependable regularities in the aggregate, while small samples can manufacture the appearance of exceptional places, exceptional people, or exceptional effects. It teaches that trustworthy inference requires attention to sample size, repeated cases, base rates, regression, and calibration before a compelling story is allowed to become an explanation.