NPR’s All Things Considered is talking this week in a special series Risk and Reason about how people interpret probability . Yesterday, they talked about weather – as in … What does a 20% chance of rain mean? Do you know?
Today, they looked at the CIA using words rather than discrete numbers to assess the probability of something happening. “Likely”, “not likely” are encouraged instead of “67% chance” or “25% chance”. The three comments I saw on the page linked (there may be more when you check) were inane, but one commenter said
It’s a little disconcerting to be reminded that many of our policy makers can’t handle numbers, or apply critical thinking to them. […]
Silly. Nobody can handle numbers. Not as far as probability is concerned. I happen to think that the word approach is more meaningful in the CIA analysis context than a numerical score. People latch onto numbers and attribute not necessarily warranted credibility to them. I say “not necessarily” because there are legitimate cases of credible probability (coin flips, dice rolls), but those are few. And the more precise seeming of the numbers, the more they believe.
Something called the fallacy of false precision comes into play in these cases. The fallacious part is that you can’t have more precision than your least precise component. The necessity for words instead of numbers is simple: any time you have an opportunity to interpret data, the least precise component is a human being. Five analysts in a room …five different answers. Ten analysts…well. you know.
There are many, many examples of people falling to the lure of numbers and not in a good way (math geek here…there is a good way, even if you don’t think so). A quick look at an unfiltered Facebook feed will surely reveal at least one “97% of the people won’t…” But, words require justification, reasoning. Numbers do, too, but pulling the thread on the data is challenging. And on the analyst’s side, how do you explain precision when you’re assessing a terrorist threat? That’s only slightly more possible than guessing the likelihood that you’ll get a booth the next time you eat out.
Mark Twain popularized and attributed to Disraeli (unconfirmed): “There are three kinds of lies: lies, damn lies, and statistics.” (I also add NRA gun safety propaganda, but to the point I’m trying to make here…) Assigning a precise numerical probability to a chaotic situation – and if it involves humans, it’s chaotic – is ludicrous. Using words like “likely” or “unlikely” begs further questions so that the decision maker can make an informed decision.
The decision makers can handle numbers fine, and do apply critical thinking. But it’s the right kind of critical thinking for the data being assessed. “Why do you think that?” “Why do you think [Bob] has a different recommendation?”
Probability can be consciously or unconsciously manipulated by interpretation. It can also be 100% accurate and totally meaningless. Think about it: If your parents didn’t have any children, there’s a very high probability you won’t either.
And in case you are wondering, the 20% chance of rain means that in weather situations like those predicted, measurable rain occurred 20% of the time.