Judging Judgment: Ouch!

Tom Peters

Managers get paid to make judgments. Human judgment stinks. Add those two statements up and you’ve got trouble.

Judgment Under Uncertainty, edited by professors Daniel Kahneman, Paul Slovic, and Amos Tversky, is a revelation. In 555 well-documented pages, they present a persuasive case for the unreliability of judgment:

Anchoring. Take a simple case. Subjects were given a multiplication to perform: One set got 8x7x6x5x4x3x2xl; another, lx2x3x4x5x6x7x8. The two are clearly equivalent. The catch: Subjects had five seconds to make an estimate. The average answer from the 8x7x … group was 2,250; the lx2x … group, 512. (The correct response is 40,320.) Welcome to the world of “anchoring”: Where you finish depends on where you start. When eight is on the left, where we normally begin, we lock on to it and offer an answer four times higher than those “anchored” on the left by the numeral one.

Focus on the case, forget the context. Researchers label this the fundamental information-processing error. Subjects in one experiment were asked to read descriptions purportedly taken from a pool of 70 engineers and 30 lawyers (or 30 engineers, 70 lawyers). After reading a dossier made to sound somewhat “lawyer-like” or “engineer-like,” the subjects were asked what the odds were that the description was from a lawyer/an engineer. Judgments were overwhelmed by the “story” and insensitive to the “background information”—respondents completely ignored the statistical likelihood that weighed the odds 70-30 in one direction or the other.

The illusion of control. “(People) behave as though chance events are subject to control,” social psychologist Ellen Langer writes. In one experiment, members of adjacent offices were given an opportunity to buy tickets for a lottery. Participants in one group chose their ticket (a card emblazoned with a football player’s name, team, and picture) from a box. Participants in the other group were simply handed a card of the same type. Later, participants were asked how much they’d want for selling their ticket back, prior to the drawing. People who had selected a card wanted $8.67 on average; those handed a card asked $1.96. The effect of self-selection—in this purely chance event—upped perceived ticket value by a factor of 4.4!

Immutability of opinions. Confronting data that contradicts our opinion hardens rather than softens that opinion. Almost any study that supports our views is seen as more convincing and better done than studies which refute it. For example, a group supporting capital punishment reviewed “pro” and “con” studies: After perusing conclusions to the pro study, strength of conviction increased slightly; it remained up after examining the methodology. Upon subsequently reading the con study’s conclusion, a minuscule drop in strength of conviction was registered—but upon scrutinizing the con study’s methodology (equivalent to the pro study), the subjects’ pro conviction soared to new heights.

We ain’t as complex as we think. Studies using elementary mathematical models that incorporate experts’ past decisions readily predict their future decisions in complex matters. Experts get stuck in grooves, though they invariably insist they’re making subtle judgments.

Multi-stage events. Almost all of business life involves betting on complex, multi-step events. And while research shows that we’re conservative in assessing a single-step event, we are absurdly optimistic about compound doings.

Suppose a project consists of 10 key steps; further imagine that the odds of making it through any one step are 70 percent. What are the odds of surviving all 10 steps? The correct probability is .7x.7x.7x.7x.7x.7x.7x.7x.7x.7—or 3 percent. If success odds at any one stage are 50 percent, the overall chance of success (.5x.5 …) plummets to one-tenth of 1 percent. Now the research: In a typical experiment involving two to eight stages and 25 different conditions, subjects overestimated success odds by a factor of two (in a simple, two-stage state) to a factor of 900,000 in a complex, eight-stage situation).

In short, we are rotten judges. Moreover, the professional judgment of doctors, auto mechanics, and geophysicists proves only about as reliable as that of laymen.

The authors provide modest suggestions for inoculating ourselves against judgmental malfunctions. Self-awareness is chief among them. But some commentators point out the benefit of faulty reasoning: Since the odds of success are so low in most settings, it can be useful to be overly optimistic and not dissuaded by realists—elsewise, why get out of bed in the morning?

If outrageously poor judgment is pervasive, perhaps it’s one more reason for managers to empower lots of people to “get on with it.” Though the judgment of those empowered is not likely to be much good either, they are closer to the action, deal with slightly less complex events, and receive less distorted information. At a minimum, such a strategy increases the number of people who try stuff. That’s not a very scintillating solution, but it may be better than relying on the plans and commands of a small number of self-deluded “experts.”