positive test strategy

Positive test strategy is a method of inquiry in which a person looks for evidence likely to confirm an existing belief instead of searching for observations that could disprove it. The book presents it as both a deliberate reasoning habit of System 2 and an expression of the confirmatory tendencies of System 1: memory supplies material that fits the question, and the resulting evidence search strengthens a story that may never have been seriously tested.

Definition and Cognitive Operation

Positive test strategy appears in the example of asking whether Sam is friendly or unfriendly. The question determines which instances of Sam’s behavior come to mind: a request about friendliness retrieves different memories from a request about unfriendliness. The strategy therefore begins before evidence is consciously evaluated. By directing attention toward confirming cases, it changes the sample from which the judgment is built.

The same pattern operates when people test a hypothesis deliberately. Rather than following the philosophers’ recommendation to seek refuting evidence, people, including scientists quite often, tend to seek data compatible with their current beliefs. System 1 favors the uncritical acceptance of suggestions, while System 2 is supposed to examine and doubt them; yet even deliberate reasoning commonly adopts a confirmatory search procedure.

Confirmation, Belief, and Emotional Coherence

Positive test strategy is closely connected to the operations of associative memory. Once a proposition or suspicion is active, related memories and images become easier to retrieve, while disconfirming material receives less attention. This makes a belief feel supported even when the search has been biased from the outset. The resulting coherence is psychological rather than necessarily evidential: the story seems to fit because the mind has preferentially collected pieces that fit it.

The book connects this process with the tendency to believe extreme and improbable events after vivid images have been activated. Asked about the probability of a tsunami hitting California within thirty years, a person is likely to imagine tsunamis and consequently overestimate the disaster’s probability. The imagined event supplies its own apparent evidence. In the same way, nonsense assertions such as “whitefish eat candy” can become easier to accept when the mind has been guided toward believing rather than challenging them.

Positive test strategy also helps produce the halo effect, in which liking one aspect of a person encourages favorable judgments about other, unobserved aspects. If someone approves of a president’s politics, that person may also approve of the president’s voice and appearance. A single positive evaluation becomes a search-and-interpretation frame for the whole individual, making the representation generated by System 1 simpler and more coherent than the underlying reality.

Bayesian Error and Base Rates

The strategy conflicts with disciplined Bayesian reasoning because it encourages people to exaggerate the diagnosticity of case-specific evidence while neglecting prior probabilities. Bayes’s rule requires a prior belief, such as a base rate, to be combined with the degree to which new evidence favors one hypothesis over an alternative. If 3% of graduate students study computer science and a description is four times more likely for computer-science students than for students in other fields, the correct posterior probability is 11%, not the much higher probability suggested by the vividness of the description. With an 80% base rate, the same evidence produces a 94.1% posterior probability.

The cab problem illustrates the distinction. When 85% of the city’s cabs are Green and 15% are Blue, and a witness identifies a cab as Blue with 80% reliability, the statistical base rate is easily underweighted because the testimony feels like a direct clue about the individual case. Causal base rates are treated differently: if Green drivers cause more than five times as many accidents as Blue drivers, that fact becomes part of a causal story about the particular accident and is readily combined with the witness’s testimony. The two stories then pull in opposite directions, leaving the estimated probabilities for the two colors approximately equal, with Green at about 41%.

The contrast shows why positive test strategy is not simply a failure to know statistics. It is a failure to ask what evidence would count against the favored explanation. A vivid witness statement or causal narrative can dominate a population fact, even when the population fact remains logically relevant. The mind tests the story it has formed rather than the full set of alternatives.

Consequences for Prediction and Decision

The same confirmatory mechanism sustains the illusion of validity in forecasting. Philip Tetlock’s twenty-year study collected more than 80,000 predictions from 284 people who earned their living commenting on political and economic trends. The experts were asked to assign probabilities to alternative outcomes, including continuation of the status quo, an increase, or a decrease. Their performance was worse than that of an equal-probability baseline, and even specialists were not significantly better than nonspecialists in the regions they knew best.

Knowledge did not reliably correct the bias. Those who knew more forecast only slightly better, while the most knowledgeable often became less dependable because additional knowledge strengthened the illusion of skill and encouraged overconfidence. The more famous forecasters were more flamboyant and more overconfident. Positive test strategy helps explain this pattern: a forecaster can collect and interpret confirming developments while failing to seek the evidence that would expose the forecast’s weakness.

Financial forecasting provides a similarly concrete consequence. A Duke University survey gathered 11,600 predictions from chief financial officers of large corporations about the following year’s Standard and Poor’s index returns. The correlation between their estimates and actual returns was slightly less than zero, yet the forecasters did not appear to recognize that their predictions were worthless. Their 80% confidence intervals produced surprises in 67% of cases rather than the expected 20%, revealing excessive certainty. Because people construct estimates from information that comes to mind, they cannot readily allow for information they never knew or never sought.

Discipline and Limits

The corrective is not to suppress intuition but to discipline the evidence search. Bayesian reasoning requires attention to base rates and a realistic assessment of diagnosticity. Reference-class forecasting supplies a baseline before case-specific adjustments: in a curriculum project, a survey of comparable projects suggested a 40% failure rate and a seven-to-ten-year completion time, a far grimmer starting point than the project team’s inside-view forecast. The outside view does not guarantee a correct prediction, but it prevents a persuasive internal story from becoming the sole basis of judgment.

Decision theory supplies a parallel check through dominance and invariance. Rational choice requires preference for an option that is at least as good in every respect and better in one, and it requires equivalent prospects to receive the same preference despite changes in description. Yet framing experiments show that people may prefer certainty when outcomes are described as lives saved and gamble when the same outcomes are described as lives lost. Such framing effects persist among sophisticated respondents and remain even when the same people answer both versions within minutes.

For that reason, the book recommends deliberate attempts to frame a decision in more than one way and to test whether the preference survives the transformation. A canonical representation in terms of total assets could reduce some inconsistencies, but intuitive computation often cannot combine all outcomes in concurrent decisions, and no single objective metric works equally well for finance, health, safety, and quality of life. Positive test strategy thus belongs to a larger problem: confidence in a coherent answer does not show that the answer has survived a serious search for disconfirmation.