Broken-Leg Rule

The Broken-Leg Rule is a principle for disciplined exceptions to statistical prediction. Associated with Paul Meehl’s criticism of expert judgment, it holds that a formula should ordinarily be trusted over an impressionistic judgment, but that a rare fact with overwhelming relevance can properly displace the formula. The rule’s significance lies in its narrowness: unusual information matters not because it makes a case feel unique, but because it has a demonstrably decisive connection to the outcome.

Definition and Origin

The rule appears in the discussion of why experts are often inferior to algorithms. Experts tend to complicate predictions by considering elaborate combinations of features and by overriding simple formulas with additional information about an individual case. Paul Meehl argued that this usually reduces predictive validity. Human decision makers frequently feel entitled to correct a formula, yet they are wrong more often than not; simple combinations of features generally perform better than clever, case-specific adjustments.

Meehl’s thought experiment supplied the memorable exception. A formula predicts whether a particular person will go to the movies that evening. If information arrives that the person broke a leg that day, it is proper to disregard the formula. A broken leg is both very rare and decisive: its occurrence directly changes the practical possibility of going to the movies, unlike the countless vivid but weakly relevant details that often tempt a forecaster to intervene. The name Broken-Leg Rule became attached to this principle.

Operation in Prediction

The rule establishes a hierarchy between ordinary evidence and exceptional evidence. In ordinary circumstances, a prediction formula aggregates features consistently and avoids the unstable weighting that characterizes expert intuition. The formula should not be discarded merely because the decision maker knows something additional, notices a striking detail, or believes that the present case is unlike the cases used to construct the model.

The exception requires two properties at once. The information must be rare, so that it is not already adequately represented by the ordinary statistical pattern, and decisive, so that it has a powerful causal or practical bearing on the predicted outcome. The broken leg satisfies both conditions. A marginal preference, a colorful biography, or an expert’s sense that a case is special does not automatically satisfy either one. The rule therefore protects statistical reasoning from casual exceptions while preserving room for facts that genuinely invalidate the baseline prediction.

Relation to Expert Error

The Broken-Leg Rule responds to a broader problem of unreliable human judgment. In repeated evaluations of the same information, professionals often contradict their own earlier judgments. Experienced radiologists classified the same chest X-ray differently on separate occasions about 20 percent of the time, and a study of 101 independent auditors found a comparable degree of inconsistency. A review of 41 studies suggested that such instability was typical across several professions. A judgment that cannot be reproduced reliably cannot be a dependable predictor.

The rule also clarifies why expertise and confidence are poor safeguards. In a twenty-year study, Philip Tetlock collected more than 80,000 predictions from 284 people who earned their living commenting on political and economic trends. These experts performed worse than a baseline that assigned equal probabilities to three possible outcomes, and specialists were not significantly better than nonspecialists even in regions they knew best. Knowledge improved forecasting only slightly, while the most knowledgeable and famous forecasters often became more overconfident. The Broken-Leg Rule does not reject expertise; it demands that experts identify an actual decisive exception rather than convert confidence into permission to override evidence.

Relation to Statistical Calibration and Outside Views

The principle belongs to the same family of safeguards as regression to the mean, the outside view, and reference class forecasting. Extreme results often contain both a stable component and luck or other temporary factors, so predictions should generally move back toward the average rather than simply repeat the initial extreme. A golfer who scores 66 on the first day of a tournament may possess above-average talent and have enjoyed unusually good luck; the second-day prediction should retain the talent component but treat future luck as average. The corresponding error is to mistake an exceptional observation for a complete and permanent description of the person.

The same discipline appears in the planning fallacy. When a team designing a curriculum for the Israeli Ministry of Education estimated how long its textbook would take to complete, it relied on the inside view and produced an unrealistically optimistic forecast. The remedy was to use the outside view: identify a reference class of comparable projects and draw on their actual completion times and overruns. Bent Flyvbjerg’s formulation emphasizes that planners should use distributional information rather than ignore it. Reference class forecasting applies this method through large databases containing the plans and outcomes of hundreds of projects, while allowing a departure from the baseline when a concrete, decisive difference justifies it.

Significance and Limits

The Broken-Leg Rule is best understood as an exception-management rule, not as an argument for intuition against statistics. Its memorable example gives statistical discipline a boundary: formulas are powerful precisely because they resist inconsistent human adjustments, but no model should be treated as invulnerable to information that directly changes the case. The rule’s force comes from the contrast between a genuinely decisive interruption and the ordinary flood of supposedly special details that merely make a story feel coherent.

This distinction is especially important because confidence intervals and subjective probability judgments are commonly too narrow. When judges estimate quantities such as the Dow Jones average, they often begin with a best estimate and adjust upward or downward to create percentile values. The adjustments are usually insufficient, producing confidence intervals that express more certainty than the evidence warrants; in many studies, actual values fell outside the stated extreme bounds in about 30 percent of cases rather than the expected 2 percent. A legitimate exception must therefore be grounded in the structure of the event, not in the forecaster’s certainty that an exception exists. The Broken-Leg Rule preserves that demanding standard: override the algorithm only when the evidence is both unusual and decisive.