1/ Superforecasting: The Art and Science of Prediction (Tetlock, Gardner)
"Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs." (p. 18)
amazon.com/Superforecasti…
"Foresight isn’t a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs." (p. 18)
amazon.com/Superforecasti…
2/ "The news delivers forecasts without asking how good the forecasters really are. Corporations and governments pay for forecasts that may be prescient, worthless, or something in between. All of us make critical decisions based on forecasts whose quality is unknown." (p. 3)
3/ "Pundits are not on the news because they possess proven skill at forecasting. Old forecasts are like old news—soon forgotten. Pundits are not asked to reconcile what they said with what happened later.
"Talking heads are skilled at telling a compelling story with conviction.
"Talking heads are skilled at telling a compelling story with conviction.
4/ "Many have become wealthy peddling forecasting of untested value to executives, government officials, and ordinary people who would swallow medicine of unknown efficacy but who routinely pay for forecasts that are as dubious as elixirs sold from the back of a wagon.
5/ "It was easiest for experts beat chance on questions that only required looking one year out. Accuracy fell as they tried to forecast further out—approaching the dart-throwing-chimp level for 3-5 years. That is suggestive of the limits of expertise in a complex world." (p. 5)
6/ "Clouds form when water vapor coalesces around dust particles. Exactly how a particular cloud develops—its shape—depends on complex interactions among droplets. Computer modelers need equations that are highly sensitive to tiny butterfly-effect errors in data collection.
7/ "Even if we know everything about cloud formation, we can't predict a particular cloud's shape.
"In one of history’s great ironies, scientists know more than a century ago and have more computing power, but they are much less confident about perfect predictability." (p. 9)
"In one of history’s great ironies, scientists know more than a century ago and have more computing power, but they are much less confident about perfect predictability." (p. 9)
8/ "Accuracy almost never determined with sufficient regularity and rigor that conclusions can be drawn.
"The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measuremen, which means no revision and no improvement.
"The consumers of forecasting—governments, business, and the public—don’t demand evidence of accuracy. So there is no measuremen, which means no revision and no improvement.
9/ "Forecasts are used as entertainment (CNBC), to advance political agendas (as activists hope to do when they warn of looming horrors), and to impress (what banks deliver when they pay a famous pundit to tell wealthy clients about the global economy in 2050).
10/ "They are also used to assure audience that their beliefs are 'correct' and the future will unfold as expected. Partisans are fond of these forecasts.
"This jumble of goals is seldom acknowledged, which makes it difficult to start working toward measurement and progress.
"This jumble of goals is seldom acknowledged, which makes it difficult to start working toward measurement and progress.
11/ "Thanks to the frankly quite amazing lack of rigor in so many forecasting domains, the opportunity for improvement is huge.
"To seize it, all we have to do is set a clear goal—accuracy!—and get serious about measuring." (p. 16)
"To seize it, all we have to do is set a clear goal—accuracy!—and get serious about measuring." (p. 16)
12/ "The difference between heavyweight and amateur poker players is that the heavyweights know the difference between a 60/40 bet and a 40/60 bet.
"Our psychology convinces us we know things that we really don't. For centuries, this hobbled progress in medicine.
"Our psychology convinces us we know things that we really don't. For centuries, this hobbled progress in medicine.
13/ "When physicians finally accepted that their experience and perceptions were not reliable means of determining whether a treatment works, they turned to scientific testing—and medicine finally to rapidly advance.
"The same revolution needs to happen in forecasting." (p. 19)
"The same revolution needs to happen in forecasting." (p. 19)
14/ "In 1954, Paul Meehl reviewed 20 studies showing that well-informed experts predicting outcomes—whether a student would succeed or a parolee be sent back to prison—were less accurate than simple algorithms using indicators like ability test scores and records of past conduct.
15/ "Meehl’s claim upset experts, but subsequent research (>200 studies) has shown that statistical algorithms beat subjective judgment. In the handful of studies where they don’t, they usually tie. Given that algorithms are quick and cheap, a tie supports algorithm." (p. 21)
16/ "Did the ostrich egg poultices applied by ancient Egyptian physicians actually heal head fractures? What about bloodletting? Everyone from the ancient Greeks to George Washington’s doctors swore that it was wonderfully restorative, but did it work?
17/ "It was not unusual for a sick person to be better off if there was no physician available: letting an illness take its natural course was less dangerous than what a physician would inflict. Treatments seldom got better, no matter how much time passed.
18/ "When George Washington fell ill in 1799, his esteemed physicians bled him relentlessly, dosed him with mercury to cause diarrhea, induced vomiting, and raised blood-filled blisters by applying hot cups to the old man’s skin.
19/ "There are too many factors involved, too many possible explanations, too many unknowns.
"If physicians are already inclined to think the treatments work, ambiguity is likely to be read as support for the treatments' efficacy. More rigorous experimentation was never done.
"If physicians are already inclined to think the treatments work, ambiguity is likely to be read as support for the treatments' efficacy. More rigorous experimentation was never done.
20/ "People do experiments when they aren’t sure what the truth is. But Galen was untroubled by doubt. “All who drink of this treatment recover in a short time, except those whom it does not help, who all die. It is obvious, therefore, that it fails only in incurable cases.”
21/ Feynman: "Because we have doubt, we propose looking in new directions for new ideas. The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test." (p. 30)
22/ "Only after WWII were the first serious randomized controlled trials were attempted. They delivered excellent results.
"Scientists who promoted the modernization of medicine routinely found that the medical establishment was uninterested or even hostile to their efforts.
"Scientists who promoted the modernization of medicine routinely found that the medical establishment was uninterested or even hostile to their efforts.
23/ "Physicians and their institutions didn’t want to let go of the idea that their judgment revealed truth. They were backed up by respected authorities [the National Health Service].
"But the alternative is an uncontrolled experiment that produces only the illusion of insight.
"But the alternative is an uncontrolled experiment that produces only the illusion of insight.
24/ "Those in power would say a policy worked; opponents would say it failed. Nobody truly knew.
"The government just assumed that policy would work without doing experiments. This was the same toxic brew of ignorance+confidence that had kept medicine in the dark for millennia.
"The government just assumed that policy would work without doing experiments. This was the same toxic brew of ignorance+confidence that had kept medicine in the dark for millennia.
25/ “A bat and ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?”
"Thinking the problem through requires sustained focus. “Ten cents” feels right, but it’s actually wrong if you give it a sober second thought.
"Thinking the problem through requires sustained focus. “Ten cents” feels right, but it’s actually wrong if you give it a sober second thought.
26/ "This is from the Cognitive Reflection Test, which has shown that most people—including very smart ones—aren’t very reflective. They scribble down “10¢” without thinking carefully (and never find the mistake).
"Human behavior follows hunches: if it feels 'true,' it is.
"Human behavior follows hunches: if it feels 'true,' it is.
27/ "Intuitive System 1 can only do its job of delivering strong conclusions at lightning speed if it never pauses to wonder whether evidence is inadequate or if there is better evidence elsewhere. Kahneman used an oddly memorable label: WYSIATI (What You See Is All There Is).
28/ "We are hard-wired to invent stories that impose coherence on the world.
"Sane people are expected to have sensible-sounding reasons. It is awkward to say, “I have no idea why.“
"This compulsion arises every time a journalist says, “The Dow rose today on news that…”
"Sane people are expected to have sensible-sounding reasons. It is awkward to say, “I have no idea why.“
"This compulsion arises every time a journalist says, “The Dow rose today on news that…”
29/ "It’s rare for a journalist to say, “The market rose today for any one of a hundred different reasons, or a mix, so no one knows.”
"The problem is that we move too fast from confusion and uncertainty to a clear and confident conclusion without spending any time in between.
"The problem is that we move too fast from confusion and uncertainty to a clear and confident conclusion without spending any time in between.
30/ "The best evidence that a hypothesis is true is an experiment designed to prove the hypothesis is false but fails to do so.
"Scientists must be able to answer the question, “What would convince me I am wrong?” If they can’t, they may be too attached to their beliefs.
"Scientists must be able to answer the question, “What would convince me I am wrong?” If they can’t, they may be too attached to their beliefs.
31/ "Doubt can be reduced by better evidence from better studies.
"But our natural inclination is to grab on to the first plausible explanation and happily gather supportive evidence without checking its reliability. That is what psychologists call confirmation bias.
"But our natural inclination is to grab on to the first plausible explanation and happily gather supportive evidence without checking its reliability. That is what psychologists call confirmation bias.
32/ "We rarely seek out contrary evidence. When it is shoved under our noses, we are motivated skeptics—finding reasons, however tenuous, to belittle it.
Kahneman: “High confidence mainly tells you that an individual has constructed a coherent story, not that the story is true.”
Kahneman: “High confidence mainly tells you that an individual has constructed a coherent story, not that the story is true.”
33/ " “Should I worry about the shadow in the long grass?” is hard. Without more data, it may be unanswerable. So we substitute an easier question: “Can I easily recall a lion attacking someone from the long grass?” That question becomes a proxy for the original question.
34/ "If the answer is yes to the second question, the answer to the first also becomes yes.
"So the availability heuristic is a bait-and-switch maneuver.
"We may substitute ‘Do most climatologists think climate change is real?’ for ‘Is climate change real?’ ” " (p. 39)
"So the availability heuristic is a bait-and-switch maneuver.
"We may substitute ‘Do most climatologists think climate change is real?’ for ‘Is climate change real?’ ” " (p. 39)
35/ "Whether intuition generates delusion or insight depends on whether you work in a field full of valid cues that you can learn from.
"If you have the time to think, do so. Be prepared to accept that what seems obviously true now may turn out to be false later." (p. 44)
"If you have the time to think, do so. Be prepared to accept that what seems obviously true now may turn out to be false later." (p. 44)
36/ "Vague thoughts are easily expressed with vague language. When forecasters are forced to translate terms like “serious possibility” into numbers, they have to think more carefully. Forecasters who practice get better at distinguishing finer degrees of uncertainty." (p. 57)
37/ "If an intelligence agency says there is a 65% chance that an event will happen, it risks being pilloried if it does not. Because the forecast itself says there is a 35% chance it will not happen, that’s a big risk. So what’s the safe thing to do? Stick with vague language.
38/ "A “fair chance” can retroactively be stretched to mean something considerably bigger than 50%—so the forecaster nailed it. It can also be shrunk to something much smaller—again, the forecaster nailed it. With such perverse incentives, it’s no wonder people prefer vagueness.
39/ "In some fields, precision has become standard. “Slight chance of showers” has given way to “30% chance of showers” in weather forecasts.
"But hopelessly vague language is still so common, particularly in the media, that we rarely notice how vacuous it is." (p. 58)
"But hopelessly vague language is still so common, particularly in the media, that we rarely notice how vacuous it is." (p. 58)
40/ "For events like presidential elections, it would take centuries—undisturbed by wars, plagues, and other shocks that perturb the true underlying causes—to pile up enough forecasts to make the statistics work.
41/ "The many forecasts required for calibration calculations make it impractical to judge forecasts about rare events, and even with common events it means we must be patient data collectors—and cautious data interpreters." (p. 62)
42/ "If I say a win is 95% likely, I’ve put the odds at 19 to 1. That’s extreme. If you agree to bet $100, I will owe you $1,900 if the Yankees lose. Our scoring system for forecasting should capture that pain." (p. 64)
43/ "After the 2012 presidential election, poll aggregators were hailed for correctly predicting all fifty state outcomes, but almost no one noted that a crude, across-the-board prediction of “no change” [voting the same in 2012 as in 2008]—would have scored 48 out of 50.
44/ "Improving predictions tends to be a game of inches.
"Another key benchmark is other forecasters. Who can beat the consensus?
"There are two statistically distinguishable groups of experts. The first fail to do better than random guessing.
"Another key benchmark is other forecasters. Who can beat the consensus?
"There are two statistically distinguishable groups of experts. The first fail to do better than random guessing.
45/ "The second group beats random guessing, though not by a wide margin. Indeed, they only barely beat simple algorithms like “always predict no change” or “predict the recent rate of change.” Still, however modest their foresight is, they have some.
46/ Hedgehogs: "The first group organizes thinking around Big Ideas (ideology). They squeeze complex problems into their preferred cause-effect templates and treat what doesn't fit as irrelevant distractions.
"They push their analyses to the limit (and then some)...
"They push their analyses to the limit (and then some)...
47/ "...piling up reasons why they were right and others wrong. They were unusually confident and likelier to declare things “impossible” or “certain.” They were reluctant to change their minds even when their predictions clearly failed. They would tell us, “Just wait.“
48/ Foxes: "The other, more pragmatic, group draws on many analytical tools, with the choice of tool hinging on each particular problem.
"They gather information from as many sources as possible. They often shift mental gears, sprinkling their speech with transition markers...
"They gather information from as many sources as possible. They often shift mental gears, sprinkling their speech with transition markers...
49/ "...such as “however,” “but,” “although,” and “on the other hand.” They talk about possibilities and probabilities, not certainties. And while no one likes to say “I was wrong,” these experts more readily admit it and change their minds.
50/ "Foxes beat hedgehogs. And the foxes didn’t just win by acting like chickens, playing it safe with 60%-70% forecasts where hedgehogs boldly went with 90% and 100%. They beat hedgehogs on both calibration and resolution. Foxes have real foresight. Hedgehogs don’t." (p. 68)
51/ "The hedgehog’s ideology doesn’t improve his foresight. It distorts it. More information doesn’t help because it’s all seen through the same tinted glasses. It may increase the hedgehog’s confidence, but not his accuracy. That’s a bad combination.
52/ "When hedgehogs made forecasts on the subjects they knew the most about—their own specialties—their accuracy declined.
"The more famous an expert was, the less accurate he was. Editors, producers, and the public hire hedgehogs, who also happen to be bad forecasters.
"The more famous an expert was, the less accurate he was. Editors, producers, and the public hire hedgehogs, who also happen to be bad forecasters.
53/ "Animated by ideology, hedgehogs tell tight, simple, clear stories that grab and hold audiences. As anyone who has done media training knows, the first rule is “keep it simple, stupid.”
"Better still, hedgehogs come across as confident.
"Better still, hedgehogs come across as confident.
54/ "Hedgehogs pile up reasons why they are right—“furthermore,” “moreover”—without considering other perspectives and the pesky doubts and caveats they raise.
"People tend to find uncertainty disturbing: “maybe” underscores uncertainty with a bright red crayon.
"People tend to find uncertainty disturbing: “maybe” underscores uncertainty with a bright red crayon.
55/ "The simplicity and confidence of the hedgehog impairs foresight, but it calms nerves—good for their careers.
"Foxes don’t fare so well in the media. They’re less confident, less likely to say something is “certain” or “impossible,” likelier to settle on shades of “maybe.”
"Foxes don’t fare so well in the media. They’re less confident, less likely to say something is “certain” or “impossible,” likelier to settle on shades of “maybe.”
56/ "Foxes' stories are complex, full of “howevers” and “on the other hands,” because they look at problems one way, then another, and another. This aggregation of many perspectives is bad TV. But it’s good forecasting." (p. 72)
57/ "Foxes deploy not one analytical idea, but many, and seek out information not from one source, but many. Then they synthesize it into a single conclusion. In a word, they aggregate. The 'wisdom of crowds' occurs within one skull." (p. 73)
Ensembles:
Ensembles:
https://twitter.com/GestaltU/status/1110908075333038081
58/ "Consider a game in which players must guess a number between 0 and 100. The person whose guess comes closest to 2/3 of the average guess of all contestants wins.
"(The Financial Times held this contest in 1997.)
"From game theory, the Nash equilibrium solution is zero.
"(The Financial Times held this contest in 1997.)
"From game theory, the Nash equilibrium solution is zero.
59/ "In the actual contest, many people did work all the way down to 0, but 0 was not the right answer. The average guess of all the contestants was 18.91, so the winning guess was 13.
"I failed because I only looked at the problem from one perspective—the perspective of logic.
"I failed because I only looked at the problem from one perspective—the perspective of logic.
60/ "Surely some people are cognitively lazy. Because I didn't factor in that perspective in my own judgment, I was wrong.
"Unfortunately, aggregation doesn’t come naturally. Intuition insists that it sees reality objectively, so we don’t consider alternative views." (p. 77)
"Unfortunately, aggregation doesn’t come naturally. Intuition insists that it sees reality objectively, so we don’t consider alternative views." (p. 77)
61/ "“Analysis should have been better. The result would have been to make intelligence assessments less certain rather than to reach a fundamentally different conclusion.” The IC still would have concluded that Saddam had WMDs; they just would have been much less confident.
62/ "A less-confident conclusion from the IC could have made a huge difference: If some in Congress had set the bar at “beyond a reasonable doubt” for supporting the invasion, then a 60%-70% probability estimate that Saddam was churning out WMDs would not have satisfied them.
63/ "Statements like “Iraq has…” “Baghdad has…” admit no possibility of surprise.
"Postmortems revealed that the IC had never seriously explored the idea that it *could* be wrong.
"Absent accuracy metrics, there is no way to hold intelligence analysts accountable." (p. 85)
"Postmortems revealed that the IC had never seriously explored the idea that it *could* be wrong.
"Absent accuracy metrics, there is no way to hold intelligence analysts accountable." (p. 85)
64/ "The list of organizations that produce or buy forecasts without bothering to check for accuracy is astonishing.
"We now know ordinary people and some simple math can not only compete with professionals supported by a multibillion-dollar apparatus but also beat them." (p.91)
"We now know ordinary people and some simple math can not only compete with professionals supported by a multibillion-dollar apparatus but also beat them." (p.91)
65/ "Slow regression to the mean is more often seen in activities dominated by skill, while faster regression is more associated with chance.
"Each year, 30% of the individual superforecasters fall from the ranks of the top 2% next year; 70% of superforecasters remain as such.
"Each year, 30% of the individual superforecasters fall from the ranks of the top 2% next year; 70% of superforecasters remain as such.
66/ "The chances of such consistency arising among coin-flip guessers (where the year-to-year correlation is 0) is less than 1 in 100,000,000, but the chances of it arising among forecasters (where year-to-year correlation is 0.65) is far higher, about 1 in 3." (p. 104)
67/ "Regular forecasters scored higher on intelligence and knowledge tests than 70% of the population. Superforecasters did better, placing higher than 80%.
"But history is replete with brilliant people who made forecasts that proved considerably less than prescient.
"But history is replete with brilliant people who made forecasts that proved considerably less than prescient.
68/ "McNamara—defense secretary under Kennedy and Johnson and famously one of “the best and brightest”—escalated Vietnam war in the firm belief that if South Vietnam were lost to the Communists, all Southeast Asia would follow.
"This certainty wasn’t based on serious analysis.
"This certainty wasn’t based on serious analysis.
69/ "No serious analysis was conducted until 1967—years later. “Our decision making was gravely flawed,” McNamara wrote in his autobiography. “We failed to analyze our assumptions critically.” Ultimately, it’s not the crunching power that counts. It’s how you use it." (p. 110)
70/ "When we estimate, we start with a number (the 'anchor') and adjust.
"We underadjust, so a bad anchor produces a bad estimate. Kahneman and Tversky showed you could influence people’s judgment by exposing them to any number, even one randomly selected by the spin of a wheel.
"We underadjust, so a bad anchor produces a bad estimate. Kahneman and Tversky showed you could influence people’s judgment by exposing them to any number, even one randomly selected by the spin of a wheel.
71/ "A better anchor [a base rate] is a distinct advantage.
"Asking people to assume their initial judgment is wrong (and why), then make another judgment, produces a second estimate which, combined with the first, improves accuracy almost as much as a second person's estimate.
"Asking people to assume their initial judgment is wrong (and why), then make another judgment, produces a second estimate which, combined with the first, improves accuracy almost as much as a second person's estimate.
72/ "The same effect was produced by letting several weeks pass before asking for the second estimate.
"There is an even simpler way of getting another perspective on a question: tweak its wording. “Will he get his visa?” tilts your mental playing field in one direction.
"There is an even simpler way of getting another perspective on a question: tweak its wording. “Will he get his visa?” tilts your mental playing field in one direction.
73/ "You may slide into confirmation bias: “Black government officials suffered under apartheid. Of course they will give a visa to Tibet’s own Nelson Mandela.”
"Turn the question on its head and ask, “Will the South African government *deny* the Dalai Lama for six months?”
"Turn the question on its head and ask, “Will the South African government *deny* the Dalai Lama for six months?”
74/ "That encourages you to lean in the opposite direction and look for reasons why it would deny the visa—for example, a desire not to anger its biggest trading partner.
"The commentary superforecasters post on forums is rife with “on the one hand/on the other” dialectics.
"The commentary superforecasters post on forums is rife with “on the one hand/on the other” dialectics.
75/ "Superforecasters pursue point-counterpoint discussions routinely, and they keep at them long past the point where most people would succumb to migraines.
"For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded." (p. 127)
"For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded." (p. 127)
76/ "A smart executive will not expect universal agreement and will treat its appearance as a warning flag that groupthink has taken hold.
"Distinctive, unique opinions are evidence that people are actually thinking for themselves and offering their own perspectives." (p. 130)
"Distinctive, unique opinions are evidence that people are actually thinking for themselves and offering their own perspectives." (p. 130)
77/ "Robert Rubin was often frustrated when briefing top policy makers in the White House and Congress. People treated an 80% probability as a certainty. “You almost had to pound the table, to say ‘yes there’s a high probability, but this also might not happen.’ “
78/ "Outside of a classroom, away from abstractions, when dealing with real issues, these educated, accomplished people reverted to the intuitive. Only when the probabilities were closer to even did they easily grasp that the outcome may or may not happen." (p. 140)
79/ "Scientists relish uncertainty, or at least accept it, because in scientific models of reality, certainty is illusory.
"One of twentieth-century science’s great accomplishments has been to show that uncertainty is an ineradicable element of reality.
"One of twentieth-century science’s great accomplishments has been to show that uncertainty is an ineradicable element of reality.
80/ "This is true both at the margin of scientific knowledge and at what currently appears to be its core. Scientific facts that look as solid as rock to one generation can be crushed to dust beneath the advances of the next. All scientific knowledge is tentative." (p. 140)
81/ "When superforecasters sense that a question is loaded with irreducible uncertainty—say, a currency-market question—they have learned to be cautious, keeping their initial estimates inside the shades-of-maybe zone between 35% and 65% and moving out tentatively." (p. 144)
82/ "Superforecasters try to be so precise that they sometimes debate differences most of us see as inconsequential—whether the correct probability is 5% or 1%, or whether a fraction of 1% is close enough to zero to justify rounding down.
83/ "Precision on that scale means moving from a world in which a threat or an opportunity is extremely improbable to one in which it is *categorically* impossible—which matters if the consequences are great enough. Imagine an Ebola outbreak. Or funding the next Google." (p. 144)
84/ "Superforecasters update much more frequently, on average, than regular forecasters. That obviously matters. An updated forecast is likely to be a better-informed forecast and therefore a more accurate forecast." (p. 154)
85/ “Belief perseverance:” "People are capable of rationalizing like crazy to avoid acknowledging new information that upsets their settled beliefs.
"For facts that are impossible to ignore, we budge, grudgingly, but the degree of change is smaller than it should be.
"For facts that are impossible to ignore, we budge, grudgingly, but the degree of change is smaller than it should be.
86/ "Once the brain has things neat and orderly, it tries to keep disturbances to a minimum.
"Our judgments about risks are driven less by a careful weighing of evidence than by our identities... people’s views on gun control often correlate with their views on climate change.
"Our judgments about risks are driven less by a careful weighing of evidence than by our identities... people’s views on gun control often correlate with their views on climate change.
87/ "Psycho-logic trumps logic. When Kahan asks people who feel strongly that gun control increases (or diminishes) risk to imagine conclusive evidence that shows they are wrong, and then asks if they would change their position in light of that evidence, they typically say no.
88/ "Superforecasters may have a surprising advantage: they’re not experts or professionals, so they have little ego invested in each forecast.
"They aren’t deeply committed to their judgments, which makes it easier to admit when a forecast is off-track and adjust.
"They aren’t deeply committed to their judgments, which makes it easier to admit when a forecast is off-track and adjust.
89/ "The self-esteem stakes are far less than those for career CIA analysts or acclaimed pundits with their reputations on the line. That helps them avoid underreaction when new evidence calls for updating beliefs." (p. 163)
90/ "Bayes’ core insight involves constantly updating in proportion to the weight of the evidence.
“I think it is likely that I have a better intuitive grasp of Bayes’ theorem than most people, even though if you asked me to write it down from memory, I’d probably fail.” (p.171)
“I think it is likely that I have a better intuitive grasp of Bayes’ theorem than most people, even though if you asked me to write it down from memory, I’d probably fail.” (p.171)
91/ "Police officers spend a lot of time figuring out who is telling the truth and who is lying, but research has found they aren’t nearly as good at it as they think they are and they tend not to get better with experience.
"Experience must be accompanied by clear feedback.
"Experience must be accompanied by clear feedback.
92/ "Charges may be laid, a trial held, and a verdict delivered, or there may be a plea bargain. But this can take months or years, and even when there is a resolution, a huge range of factors could have gone into it.
"Officers grow confident faster than they grow accurate.
"Officers grow confident faster than they grow accurate.
93/ "Gaps like that are far from unusual. Research on calibration—how closely confidence matches accuracy—routinely finds people are too confident.
"But neither meteorologists nor seasoned bridge players suffer from overconfidence; both get clear, prompt feedback." (p. 181)
"But neither meteorologists nor seasoned bridge players suffer from overconfidence; both get clear, prompt feedback." (p. 181)
94/ "Vague terms like “probably” and “likely” make it hard to judge forecasts.
"Bertram Forer asked students to complete a personality test, then gave them individual personality profiles based on the results and asked how well the test captured their individual personalities.
"Bertram Forer asked students to complete a personality test, then gave them individual personality profiles based on the results and asked how well the test captured their individual personalities.
95/ "People were impressed, giving the test an average rating of 4.2/5—remarkable, since he'd had taken vague statements like “you have a great need for people to like and admire you” from a book on astrology, assembled them into a profile, and given the same profile to everyone.
96/ "Vague language is elastic language. The students stretched it to fit their self-images, even though they thought they were judging the test objectively. The lesson for forecasters who would judge their own vague forecasts is: don’t kid yourself." (p. 182)
97/ "When forecasts span months or years, the long wait for results allows the flaws of memory to creep in.
"In 1992–93, I returned to the experts, reminded them about the question I'd asked 1988, and asked them to recall their probability forecasts.
"In 1992–93, I returned to the experts, reminded them about the question I'd asked 1988, and asked them to recall their probability forecasts.
98/ "On average, the experts recalled a number 31 percentage points higher than the correct figure. So an expert who thought there was only a 10% chance might remember herself thinking there was a 40% or 50% chance.
99/ "There was even a case in which an expert who pegged the probability at 20% recalled it as 70%—hindsight bias expressed as the “I knew it all along” effect.
"Forecasters who use ambiguous language and rely on flawed memories don’t get clear feedback to learn from." (p. 183)
"Forecasters who use ambiguous language and rely on flawed memories don’t get clear feedback to learn from." (p. 183)
100/ "People often assume that when a decision is followed by a good outcome, the decision was good, which isn’t always true. That assumption can be dangerous if it blinds us to the flaws in our thinking." (p. 186)
More on this:
https://twitter.com/ReformedTrader/status/1302337159303540737
101/ "People had their mind made up about Sarah Palin from the outset because of her partisan affiliation, because she was a woman, etc., and learning facts about her just gave them excuses for their prejudices. Feelings were masquerading as knowledge, as thoughts.”
102/ "That sort of critical, psychologically astute observation helps make a forecaster a superforecaster. So does careful, accurate research." (p. 189)
103/ "The strongest predictor of becoming a superforecaster is the degree of commitment to belief updating and self-improvement. It is 3x as powerful a predictor as its closest rival, intelligence.
"Superforecasting seems to be roughly 75% perspiration, 25% inspiration." (p.192)
"Superforecasting seems to be roughly 75% perspiration, 25% inspiration." (p.192)
104/ On groupthink: "Groups that get along too well don’t question assumptions or confront uncomfortable facts. So everyone agrees, which is pleasant, and the fact that everyone agrees is tacitly taken to be proof the group is on the right track." (p. 196)
105/ "Aggregation can only do its magic when people form judgments independently, like the fairgoers guessing the weight of the ox. The independence of judgments ensures that errors are more or less random, so they cancel each other out.
106/ "In a group, independence of thought can be lost. Maybe one person is a loudmouth who dominates the discussion, or a bully, or a superficially impressive talker, or someone with credentials that cow others into line.
"Mistakes can pile up rather than cancel out.
"Mistakes can pile up rather than cancel out.
107/ "This is the root of collective folly, whether it’s Dutch investors becoming collectively convinced that a tulip bulb was worth more than a laborer’s annual salary, or American home buyers talking themselves into believing that real estate could only go up." (p. 198)
108/ "We gave teams a primer based on research in group dynamics. Groupthink is a danger: Be cooperative but not deferential. Consensus is not always good; disagreement not always bad. Don’t take that agreement—in itself—as proof that you are right. Never stop doubting.
109/ "Pointed questions are essential to working on a team."
“What do you mean by ‘pastime’?”
“What evidence is there that soccer’s popularity is declining? Over what time frame?”
"Answers to precise questions reveal people's thinking so it can be probed and tested." (p. 200)
“What do you mean by ‘pastime’?”
“What evidence is there that soccer’s popularity is declining? Over what time frame?”
"Answers to precise questions reveal people's thinking so it can be probed and tested." (p. 200)
110/ “The team is much more effective at gathering information than one person could ever be, especially with members' different research styles.”
"On average, a superforecaster in year 1 (2) put on a superforecaster team in year 2 (3) became 50% more accurate.
"On average, a superforecaster in year 1 (2) put on a superforecaster team in year 2 (3) became 50% more accurate.
111/ "The results were clear-cut each year. Teams of ordinary forecasters beat all forecasters' unweighted average by 10%. Prediction markets beat ordinary teams by 20%. Superforecaster teams beat prediction markets by 15-30%." (p. 207)
112/ "The aggregation of different perspectives is a potent way to improve judgment, but the key word is different. Combining uniform perspectives only produces more of the same, while slight variation will produce slight improvement.
113/ "Think back to Obama asking each of his advisers how likely it was that the unusually tall man in the mystery house in Pakistan was bin Laden. The answers averaged out to 70%. That’s the wisdom of the crowd: a hard-to-beat number that should have been given more respect.
114/ "The more diverse his team, the greater the chance that some advisers will possess scraps of information that others don’t. Since these scraps mostly point toward “it’s bin Laden,” if all the advisers were given all the scraps, they would individually raise their estimates.
115/ "That would boost the “wisdom of the crowd” figure—maybe to 80% or 85%. That’s the thinking behind the extremizing algorithm from chapter 4.
"Superforecaster teams were good at sharing information with each other, and extremizing didn’t help them much.
"Superforecaster teams were good at sharing information with each other, and extremizing didn’t help them much.
116/ "But we got major gains when we extremized regular forecaster teams. Indeed, extremizing gave them a big enough boost to pass some superforecaster teams, and extremizing a large pool of regular forecasters produced, as we saw earlier, tournament-winning results." (p. 209)
117/ "The Prussian military had long appreciated uncertainty—they had invented board games with dice to introduce the element of chance missing from games like chess." (p. 213)
The Japanese literally used games as battle simulations:
The Japanese literally used games as battle simulations:
https://twitter.com/ReformedTrader/status/1283625896616718338
118/ "Scenarios were laid out, and students were invited to suggest solutions and discuss them collectively. Disagreement was not only permitted: it was expected. Even the views of instructors and generals were subject to scrutiny.
119/ "The fundamental message: think. Discuss your orders. Even criticize them. And if you absolutely must—and you better have a good reason—disobey them.
"Commanders were to tell subordinates their goals but not give step-by-step orders about how to reach those goals." (p. 216)
"Commanders were to tell subordinates their goals but not give step-by-step orders about how to reach those goals." (p. 216)
120/ "This is the opposite of the image most people have of Germany’s WWII military.
"But what is often forgotten is that the Nazis did not create the Wehrmacht. They inherited it. And it could not have been more different from the unthinking machine we imagine." (p. 218)
"But what is often forgotten is that the Nazis did not create the Wehrmacht. They inherited it. And it could not have been more different from the unthinking machine we imagine." (p. 218)
121/ "Hitler took direct control in violation of these principles, with disastrous effect for Normandy. The Allies feared that German tanks would drive them back to the beaches and into the sea, but Hitler had directed that the reserves could only move on his personal command.
122/ "Hitler slept late. For hours after the Allies landed on the beaches, the dictator’s aides refused to wake him to ask if he wanted to order the tanks into battle." (p. 219)
In this case, Eisenhower showed better decision making than the Germans did:
In this case, Eisenhower showed better decision making than the Germans did:
https://twitter.com/ReformedTrader/status/1305152868341538816
123/ "Shortly after WWI, Eisenhower published an article in the US Army’s Infantry Journal stating that “the clumsy, awkward and snail-like progress of the old tanks must be forgotten, and in their place we must picture this speedy, reliable and efficient engine of destruction.”
124/ "Eisenhower was dressed down. “I was told my ideas were not only wrong but dangerous, and that henceforth I was to keep them to myself. Particularly, I was not to publish anything incompatible with solid infantry doctrine. If I did, I would be hauled before a court-martial.”
125/ "In the U.S. army, subordinates were to salute and obey superiors without question. Orders were long and detailed—“the orders for the American forces to land in North Africa were the size of a Sears Roebuck catalogue”—and left little room for individual initiative." (p. 221)
126/ "Even a smart person like Annie Duke is always tempted by a simple cognitive shortcut: “I know the answer. I don’t need to think long and hard about it. I am a very successful person with good judgment. The fact that I believe my judgment is correct proves it is.”
127/ "The humility required for good judgment is not self-doubt—believing you are untalented, unintelligent, or unworthy. It is intellectual humility: recognizing that reality is profoundly complex, that seeing things clearly is a constant struggle, when it can be done at all.
128/ "Human judgment must therefore be riddled with mistakes (for fools and geniuses alike).
"It’s quite possible to think highly of yourself and be intellectually humble. Humility compels the careful reflection necessary for good judgment; confidence inspires action." (p. 228)
"It’s quite possible to think highly of yourself and be intellectually humble. Humility compels the careful reflection necessary for good judgment; confidence inspires action." (p. 228)
129/ "Understanding what worked in the Wehrmacht requires engaging in the toughest of all forms of perspective-taking: acknowledging something we despise possesses impressive qualities. Forecasters who can’t do so risk making the most serious error: underestimating your opponent.
130/ "We don’t want analysts to assume jihadist groups must be inept or that vicious regimes can’t be creatively vicious.
F. Scott Fitzgerald: “The test of a first-rate intelligence is the ability to hold two opposed ideas in mind and still retain the ability to function.”
F. Scott Fitzgerald: “The test of a first-rate intelligence is the ability to hold two opposed ideas in mind and still retain the ability to function.”
131/ "It requires teasing apart our feelings about the Nazi regime from our judgments about the Wehrmacht’s organizational resilience, to see it as both horrific and an organization we can learn from.
"There is no logical contradiction, just a psycho-logical tension." (p. 229)
"There is no logical contradiction, just a psycho-logical tension." (p. 229)
132/ "Flynn believes “societal conflict” is at a “pretty unprecedented” level.
"That can be checked via the reports that have quantified global violence since WWII. Interstate wars have been declining since the 1950s. Civil wars have been declining since the end of the Cold War.
"That can be checked via the reports that have quantified global violence since WWII. Interstate wars have been declining since the 1950s. Civil wars have been declining since the end of the Cold War.
133/ "This is reflected in the number of battle deaths per year, which, with a few blips, declined throughout the period. You don’t have to be the head of the DIA to find those reports. But Flynn saw no need to Google before drawing a sweeping conclusion and sharing it." (p. 232)
134/ "Flynn saw bad news on his desk every day, and his conclusion felt right—so That Was All There Was. As a lifelong intelligence officer, Flynn knew the importance of checking assumptions, but it didn’t feel like an assumption. It felt true.
135/ "I am not belittling Michael Flynn. The fact that a man so accomplished made so obvious an error is precisely what makes the error notable. We are all vulnerable.
"There’s no way to make ourselves bulletproof, as the famous Müller-Lyer optical illusion illustrates." (p.232)
"There’s no way to make ourselves bulletproof, as the famous Müller-Lyer optical illusion illustrates." (p.232)
136/ "You know the lines are the same. You want to see them that way. But you can’t. Not even knowing it’s an illusion can switch it off.
"Cognitive illusions are similarly impossible to stop. Superforecasters are just a System 2 slipup away from a blown forecast." (p. 233)
"Cognitive illusions are similarly impossible to stop. Superforecasters are just a System 2 slipup away from a blown forecast." (p. 233)
137/ Scope insensitivity: "On average, people said they would pay $10 to clean up the lakes in a small region. Kahneman asked another randomly selected group how much they would be willing to pay to clean up every one of the 250,000 lakes in Ontario. They too said about $10.
138/ "One study informed people that 2,000 birds/year drown in oil ponds. How much would you pay to stop that? Other subjects were told 20,000 birds died each year. A third group was told it was 200,000. Yet in each case, the average amount people said they would pay was $80.
139/ "People were responding to the prototypical image that came to them—a polluted lake or a drowning, oil-soaked duck. Kahneman: “The prototype automatically evokes an affective response, and the intensity of that emotion is then mapped onto the dollar scale.”
140/ "It’s classic bait and switch. Instead of answering the question asked—a difficult one that requires putting a money value on things we never monetize—people answered “How bad does this make me feel?”
"Scope recedes into the background: out of sight, out of mind." (p. 234)
"Scope recedes into the background: out of sight, out of mind." (p. 234)
141/ "The vast majority of forecasters were scope insensitive. Regular forecasters said there was a 40% chance Assad’s regime would fall over three months and a 41% chance it would fall over six months.
142/ "The superforecasters did much better: They put the probability of Assad’s fall at 15% over three months and 24% over six months. If we bear in mind that no one was asked both the three- and six-month version of the question, that’s quite an accomplishment." (p. 236)
143/ "My sense is that some superforecasters are so well practiced in System 2 corrections—such as stepping back to consider other points of view—that these techniques have become habitual. In effect, they are now part of their System 1.
144/ "Software tools may also help lighten the cognitive load.
"No matter how physically or cognitively demanding a task may be—cooking, sailing, surgery, operatic singing, flying fighter jets—deliberative practice can make it second nature." (p. 236)
"No matter how physically or cognitively demanding a task may be—cooking, sailing, surgery, operatic singing, flying fighter jets—deliberative practice can make it second nature." (p. 236)
145/ "At the start of each decade between 1900 and 2000, the strategic situation [from a national defense perspective] underwent a stunning change from ten years earlier.
"If you have to plan for the distant future, plan for *surprise.*
"If you have to plan for the distant future, plan for *surprise.*
146/ "Brushing off surprises makes the past look more predictable than it was—and this encourages the belief that the future is much more predictable than it is.
"In the 1980s and early 1990s, there was a prevalent belief that Japan would soon dominate the global economy.
"In the 1980s and early 1990s, there was a prevalent belief that Japan would soon dominate the global economy.
147/ "Its subsequent decline should at least give pause to those asserting China’s ascendancy. But it often doesn’t: when looking back, it seems strange that anyone ever thought Japan would take the lead." (p. 246)
More on the Japanese bubble:
More on the Japanese bubble:
https://twitter.com/ReformedTrader/status/1307413356442050560
148/ "Savoring how history could have generated an infinite array of alternative outcomes and could now generate a similar array of alternative futures instills profound humility.
"In the big scheme of things, human foresight is puny." (p. 248)
"In the big scheme of things, human foresight is puny." (p. 248)
149/ "In my daydreams, serious change will begin. Consumers of forecasting will stop being gulled by pundits with good stories and start asking pundits how their past predictions fared—and reject answers that consist of nothing but anecdotes and credentials.
150/ "Just as we now expect a pill to have been tested in peer-reviewed experiments before we swallow it, we will expect forecasters to establish the accuracy of their forecasting with rigorous testing before we heed their advice." (p. 251)
151/ "Republicans accused Silver of bias. Democrats defended his integrity and praised his forecasting acumen—showering him with praise when Obama won. But in 2014, when Silver's forecasts suggested Republicans would take control of the Senate, Democrats had a change of heart.
152/ "Officials even passed around old forecasts of Silver’s that had failed. Same forecaster, same track record—but when his forecasts ceased to align with partisan purposes, he was demoted.
"In reality, accuracy is often only one of many goals. Sometimes it is irrelevant.
"In reality, accuracy is often only one of many goals. Sometimes it is irrelevant.
153/ "An analyst with the Banco Santander Brasil SA warned wealthy clients that Brazil’s stock market and currency would likely decline if a left-wing candidate continued her ascent in the polls. The candidate and her party were livid and demanded the analyst be fired.
154/ "She was fired immediately. In Brazil, it’s not good for a bank to be on bad terms with future presidents. Whether the analyst’s forecast was accurate did not matter.
"The goal of forecasting is not accuracy. It is to advance the interests of the forecaster and his tribe.
"The goal of forecasting is not accuracy. It is to advance the interests of the forecaster and his tribe.
155/ "Accurate forecasts may do that sometimes, and when they do, accuracy is welcome, but it is pushed aside if that’s what the pursuit of power requires."
When it comes to rousing people to action, "Lenin would say forecasting does exactly what it's supposed to do." (p. 254)
When it comes to rousing people to action, "Lenin would say forecasting does exactly what it's supposed to do." (p. 254)
156/ More on analysts' forecasting ability:
"Analysts expect quality stocks to perform worse than junk stocks, but the opposite tends to be true going forward."
Analysts forecast in the wrong direction for factors in general:
"Analysts expect quality stocks to perform worse than junk stocks, but the opposite tends to be true going forward."
https://twitter.com/ReformedTrader/status/1182308282062393346
Analysts forecast in the wrong direction for factors in general:
https://twitter.com/ReformedTrader/status/1163209969577017344
157/ "Evidence-based policy is a movement modeled on evidence-based medicine, with the goal of subjecting government policies to rigorous analysis so that legislators will actually know—not merely think they know—whether policies do what they are supposed to do.
158/ "The same thinking is transforming charitable foundations, which are increasingly making funding contingent on stringent program evaluation. Bill Gates’s foundation insistents on having clear goals and measures and is renowned for the rigor of its evaluations.
159/ "Evidence-based forecasting is another manifestation of a broad shift away from decision making based on experience, intuition, and authority—“Do this because I think it will work and I’m an expert”—toward quantification and analysis made possible by technology." (p. 256)
160/ "Don’t suppose that because a scoring system needs tweaking that it isn’t a big improvement over intuitive judgment.
"Consider consumer credit ratings, whose flaws are so often criticized. Decades ago, before credit scores, loan officers made decisions almost whimsically.
"Consider consumer credit ratings, whose flaws are so often criticized. Decades ago, before credit scores, loan officers made decisions almost whimsically.
161/ "Your fate could hinge on whether you reminded them of someone, or they had slept poorly the night before, or they harbored stereotypes of “shiftless blacks” and “flighty women.” Credit scores may be miles from perfect, but they’re a huge advance from that.
162/ "Similarly, while I can’t claim my scoring system is perfect, it is a big improvement on judging a forecaster by the criteria used nowadays—titles, confidence, skill at spinning a story, number of books sold, and CNN appearances." (p. 260)
163/ "Will N. Korea conduct another nuclear test? Will it rebuff diplomatic talks on its nuclear program? Will it fire artillery at S. Korea? Will a N. Korean ship fire on a S. Korean ship? The more yeses or nos, the likelier we have an answer to the big question." (p. 264)
164/ "Intuition poses provocative questions that superforecasters should use to sharpen their foresight; superforecasters generate well-calibrated answers that intution-based superquestioners should use to fine-tune and occasionally overhaul their mental models of reality.
165/ "But there’s a much bigger collaboration I’d like to see. It would be the Holy Grail of my research program: using forecasting tournaments to depolarize unnecessarily polarized policy debates and make us collectively smarter.
166/ "Key figures could work together to identify which forecasts would meaningfully test their disagreements. The key is precision. How much inflation? Measured by what benchmark? Over what time period? The result must be a forecast question that reduces ambiguity to a minimum.
167/ "A major point of view rarely has zero merit, and if a forecasting contest produces a split decision we will have learned that the reality is more mixed than either side thought. If learning, not gloating, is the goal, that is progress." (p. 269)
168/ "Let’s not make the cynic’s mistake of thinking that those who dominate debates are the only debaters. We hear them most because they are the loudest and the media reward people who shout into bullhorns.
"But there are less voluble and more reasonable voices." (p. 269)
"But there are less voluble and more reasonable voices." (p. 269)
169/ Related reading:
Magazine cover predictions, shoeshine boy tips, and other questionable calls
Thoughts on causality
Thinking in Bets
The Undoing Project
Magazine cover predictions, shoeshine boy tips, and other questionable calls
https://twitter.com/ReformedTrader/status/1316380442765914115
Thoughts on causality
https://twitter.com/ReformedTrader/status/1200970345928261633
Thinking in Bets
https://twitter.com/ReformedTrader/status/1302337159303540737
The Undoing Project
https://twitter.com/ReformedTrader/status/1317876706414227457
170/ * Issues with expert advice
* Siloing of information
* Application of misdirected focus
Solution:
* Ask experts about their incentives
* Determine your own goals before talking to an expert
* Cross silos: get information from multiple perspectives
artofmanliness.com/articles/podca…
* Siloing of information
* Application of misdirected focus
Solution:
* Ask experts about their incentives
* Determine your own goals before talking to an expert
* Cross silos: get information from multiple perspectives
artofmanliness.com/articles/podca…
171/ "In the Forbes 400, there are only two people who have used macro-economic predictions to consistently make money: George Soros and Ray Dalio.
"It's important to note that they don’t bet everything on a single outcome, and they are frequently wrong."
"It's important to note that they don’t bet everything on a single outcome, and they are frequently wrong."
https://twitter.com/ValueStockGeek/status/1329409095171010570
• • •
Missing some Tweet in this thread? You can try to
force a refresh
