The kidney cancer counties study is a classic demonstration of the Law of Small Numbers: when observations are drawn from small populations, unusually high and unusually low results occur more often than they do in large populations. Its decisive significance is methodological rather than epidemiological. The study exposes how readily System 1 turns a statistical artifact into a causal story, while showing why statistical reasoning must attend to sample size, regression toward the mean, and the difference between apparent variation and genuine underlying differences.

The Pattern in the Counties

A study of kidney-cancer incidence across all 3,141 counties of the United States first presents a striking geographical pattern. The counties with the lowest recorded incidence are mostly rural, sparsely populated, and located in traditionally Republican states across the Midwest, the South, and the West. The pattern seems to demand an explanation, and the rural setting makes an attractive one available: clean living, little air or water pollution, fresh food, and fewer additives appear to offer a plausible account of unusually low cancer rates.

The same geographical description, however, applies to the counties with the highest recorded incidence. These counties are also predominantly rural, sparsely populated, and concentrated in the same broad regions and political settings. A second explanation can be supplied just as easily: rural poverty, limited access to medical care, a high-fat diet, and excessive alcohol and tobacco use might seem to account for unusually high rates. The two stories cannot both be true as explanations of the rural lifestyle. Rural life cannot simultaneously produce the lowest and the highest incidence of kidney cancer through the contrasting causes supplied by intuition.

The Statistical Mechanism

The relevant feature is not rural culture or Republican politics but population size. Rural counties have small populations, and small samples are more likely to produce extreme proportions by chance. The United States can therefore be imagined as a giant urn of individuals, some marked as having kidney cancer. When the individuals are divided into county-sized samples, the rural counties receive smaller samples. Their observed rates will fluctuate more widely, producing both unusually high and unusually low values without requiring any corresponding difference in the true cancer risk of rural residents.

The marble example makes the mechanism visible. If an urn contains equal numbers of red and white marbles, repeated samples of four marbles produce the homogeneous outcomes of four red or four white much more often than samples of seven marbles do. A sample of four yields an extreme result in an expected 12.5 percent of trials, compared with 1.56 percent for a sample of seven, a difference by a factor of eight. Nothing causes the extreme draw: the pattern follows mathematically from sample size. The kidney-cancer counties reproduce the same structure. Sparse counties are the statistical equivalent of the smaller draws, so they are more likely to occupy both ends of the ranking.

The explanation is therefore noncausal. A county's small population neither causes kidney cancer nor protects its residents from it; it merely permits the observed incidence to stand farther from the population average in a particular year. If the analysis is repeated the following year, the general pattern should persist: small counties will again be overrepresented among extreme results. But the counties that recorded high incidence in the first year will not necessarily record high incidence in the next. Their earlier positions were artifacts of sampling rather than stable facts about the places themselves.

Why Intuition Misreads the Evidence

The episode begins with a fact that appears to call for a cause: kidney-cancer incidence varies widely across counties, and the differences seem systematic. System 1 is exceptionally quick to identify causal connections, even when the connection is spurious. Faced with the high-incidence counties, it assumes that those counties must differ from others for a reason and searches for a story that explains the difference. Faced with the low-incidence counties, it constructs a different but equally coherent story. The coherence of each narrative makes the underlying statistical structure harder to notice.

The example also shows why “knowing” the Law of Large Numbers does not automatically produce disciplined judgment. The fact that sparsely populated counties have smaller samples does not immediately stand out as relevant when the reader is presented with an epidemiological pattern. A causal interpretation arrives first because it is vivid, concrete, and psychologically satisfying. Sampling variation, by contrast, changes the probability of an outcome without causing the outcome, and System 1 is poorly equipped to find meaning in that kind of merely statistical fact.

The kidney-cancer example belongs to a wider distinction between statistical and causal base rates. Statistical base rates describe the population to which a case belongs and are often underweighted when specific information about the case is available. Causal base rates are different: they alter the story of how the individual case came about and are therefore readily combined with other case-specific information. The county study demonstrates the danger of treating a statistical feature, such as small population, as though it were a causal property of the people living there.

Consequences for Inference and Prediction

The study warns against interpreting extreme observations as reliable evidence about the underlying population. Large samples deserve more trust because their averages are less vulnerable to accidental fluctuation; small samples deserve caution precisely because they can generate spectacular departures from the mean. The proper response to an extreme county rate is not to invent a vivid explanation immediately but to ask whether the method of measurement, especially unequal sample sizes, could have produced the pattern.

This principle is closely related to Bayesian reasoning. Bayes's rule requires prior beliefs, or base rates, to be combined with the diagnosticity of the evidence rather than displaced by an appealing description. If the evidence is weakly diagnostic and the sample is small, the appropriate posterior belief should remain close to the relevant baseline. The same logic underlies reference-class forecasting: a case-specific impression should be adjusted from a baseline established by comparable cases, not allowed to float free on the strength of a persuasive inside story.

The lesson extends to regression toward the mean. Extreme outcomes often contain both stable factors and temporary or random factors. A county with an unusually high rate in one year may possess some genuine underlying risk, but its recorded extremity also reflects sampling variation. On a later measurement, the random component is unlikely to repeat in the same direction, so the observed rate will tend to move closer to the average. Treating that natural movement as proof that a newly discovered cause or intervention worked would repeat the same error exposed by the kidney-cancer rankings.

Significance of the Study

The kidney cancer counties study is memorable because it stages a conflict between two kinds of explanation. One explanation is causal, populated by pollution, diet, poverty, medical access, and political geography. The other is statistical, built from urns, sample sizes, and the predictable frequency of extreme draws. The causal stories feel richer, but the statistical account explains why opposite stories can arise from exactly the same geographical pattern.

Its deepest implication is that some apparent differences have nothing substantial behind them beyond the conditions under which the observations were collected. A ranking can look systematic while being generated by random variation; a place can appear exceptionally healthy or exceptionally sick while remaining close to the broader population in its underlying risk. The study thus serves as a compact warning against the mind's demand that every pattern have a cause, and as a practical argument for checking sample size before believing the story that the pattern seems to tell.