The art of superforecasting: how FT readers fared against the experts in 2023
Roula Khalaf, Editor of the FT, selects her favourite stories in this weekly newsletter.
Every day, Peter Stamp goes into his home office in northern Germany and clears all distractions for an hour. He opens his laptop and scans through a list of over 140 questions that ask about the future. These questions span global conflicts, currency movements and economic policy. Stamp looks at each one, thinking through whether his predictions have changed from the day before.
“I’m trying to cover topics from all angles imaginable, even weird ones from political sites I totally disagree with, but they might offer a perspective that’s valuable,” Stamp says. “Then in the end [I’m] weighing the arguments found and thinking about what else might influence my opinion and writing everything down.”
Stamp is a “superforecaster”, an analyst of sorts whose forecasts can be proven over time to be more accurate than those of the general public. Since 2020, he has worked for Good Judgment, a company that aggregates such forecasts for major corporations, investment managers and governments, and popularised the term.
After becoming one of the top ranked forecasters on Good Judgment Open, the company’s public platform, he was invited to join its elite ranks. It is now his part-time job alongside being a financial consultant and insurance broker. He’s made predictions on over 800 questions in the past three years with a goal of working across as many subjects as possible.
Stamp has always considered himself to be good with numbers and is a member of Mensa. But those aren’t the traits that he says make him a great superforecaster. Instead his flexibility and open-mindedness are the biggest contributors to his success, he says. “And practice. At least a few hundred hours of actual practice. In my case, it’s two or three thousand hours of actual practice.”
The art of superforecasting has its origins in the work of Philip Tetlock, one of the founders of Good Judgment. Tetlock began studying how experts make geopolitical and economic predictions when he was an associate psychology professor at UC Berkeley in 1984.
Over the next 19 years, he and his team gathered over 250 experts across academia, the government and the media to make 28,000 predictions. As a whole, he found that expert predictions did little better than chance.
His research eventually caught the attention of the US intelligence community. In 2011, the Intelligence Advanced Research Projects Activity (IARPA), a US government agency that funds research to make intelligence more effective, launched a series of forecasting tournaments to improve knowledge on how to make accurate predictions. Thousands of forecasters participated over four years, making over one million predictions across 500 questions relevant to US national security.
Tetlock and his research and life partner Barbara Mellers led one of five research groups chosen to participate. One of their primary goals was to identify what exactly makes a forecaster accurate. The existing literature on forecasting offered no “consistent” advice on how to increase accuracy, Mellers says. “So we did what made sense to us.”
By tweaking the make-up of the teams and their methods, they ended up outperforming all the other research teams and even professional intelligence analysts who had access to classified information. “We had lots of bets on what we all thought would work well, and we were all surprised about [nearly] everything that happened,” Mellers says.
One of their most surprising insights was that forecasting was a skill that could be honed and developed. They would go on to conclude that there are three ways to improve accuracy: having more information, decreasing bias and decreasing the impact of irrelevant variables.
Geopolitical forecasting has gained traction in the past decade as businesses have encountered shocks from Brexit and Trump’s presidency to the Ukraine war. Famke Krumbmüller, director of EY’s Geostrategic Business Group, says forecasting is about adapting strategies to the future.
For example, she says, “if you make an investment into a country, if you change your supply chains, where should you go less and where should you go more? If you acquire a new business in a certain country, what is the likelihood that the government might intervene on your transaction?”
To help illustrate what makes a superforecaster worthy of the name, the FT asked both readers and Good Judgment superforecasters to make predictions on ten questions early this year, spanning from GDP growth rates to viewership of the Fifa women’s world cup final.
A total of 95 superforecasters answered questions, while the reader poll had approximately 8,500 respondents. The latter was more casual than scientific and only lasted for one week rather than the whole year. Still, the results shed interesting light on what we think about when we think about the future.
FT readers vs superforecasters
The FT readers were admittedly at a disadvantage; unlike at Good Judgment, our poll didn’t come with any training, formal community, ongoing feedback or stakes. Over 10 questions, it is clear that those things make a difference.
Using a scoring system that ranges from 0.5 (or pure guessing) to 1 (perfect choices) over nine questions, FT readers scored a 0.73 while superforecasters scored 0.91, using their responses from February 2 2023.
(Of the 10 questions that were posed at the end of January, we could only confidently resolve six by the time of publication, though we can fairly confidently assume three more.)
Unlike readers, the Good Judgment superforecasters were also able to update their predictions as the year went on. So although it wasn’t exactly a fair contest, our experiment does give some insight into what goes into a real superforecast — and what amateur prognosticators need to bear in mind. Here, we break it down question by question:
What will the price of bitcoin be on December 31 2023?
Some questions feel harder to predict than others. Bitcoin’s volatility is far higher than even the most unstable fiat currencies. In the month of January, bitcoin rose from $16,000 to $23,000 and as of the time of publication, it is now worth over $42,000.
If those numbers are surprising, that could be an advantage. Tetlock’s early research showed that having subject matter expertise didn’t equate to having an upper hand. Being an expert often anchored people in opinions and made it harder to synthesise additional information.
“I think the value of our group is that we’re not all trying to come to the exact same number, but rather referencing different viewpoints. It’s what together all of that represents that becomes important”, says Alice Dorman, a superforecaster who says she’s not a crypto-savvy person.
Sharing information and encouraging open-minded thinking was what made teams of superforecasters so powerful. Research from Adam Grant at the University of Pennsylvania shows that those who are good at helping others in teams often encourage others to share as well — and it tends to benefit everyone, including the original sharer.
“We have the advantage of some subject matter experts chiming in,” Dorman says. “And that’s hugely helpful to those of us who are generalists.”
What will the global annual inflation rate be in October 2023?
In February, 43 per cent of superforecasters correctly predicted that the most likely range for the global annual inflation rate in October 2023 was 6 to 7.5 per cent. Only 17 per cent of FT readers made the same guess.
Warren Hatch, Good Judgment’s chief executive, says that the key to keep in mind with a question that involves a change over time is the timeframe in which it happens. He points out that both FT readers and superforecasters generally agreed that inflation was transitory and that “we were not entering an era of multiyear, 70s-style hyperinflation”. What was different, however, was how forecasters looked at the timeline.
In 1992, Daniel Kahneman and Jack Knetsch introduced the concept of “scope insensitivity.” For a question that involves a factor that changes over time, this means that the probability of an event happening by a certain date should not be the same probability as the same event happening at a later date.
“[Experienced forecasters] will ask themselves, ‘will inflation be below a level X by date Y?’ They’ll immediately think about different date Y’s. And they’ll also think about different level X’s. So they’ll almost have a mental matrix that they’re putting together. Now, you can do this really quickly, even just a minute or two so you can get a slightly better informed view about what the probabilities look like in addition to the direction,” Hatch says. “Both forecasting groups were correct that inflation would come down. The question that remained was what the timing looks like.”
How many European countries and territories will set a record high temperature in 2023?
One skill to hone as a superforecaster is how to react to news. It can be detrimental to either overreact or underreact. How someone might respond can be linked with a variety of psychological biases, based on longstanding beliefs or recent personal experiences.
This summer offers a good example. “I had been to Spain in May just after the tremendous heatwave so I was aware of what was going on. But again, I didn’t want to let those kinds of subjective feelings get in the way of a projection,” says Steve Roth, a superforecaster based in New York who began forecasting in 2014 with Good Judgment during the IARPA tournament.
Roth says that if he had no information and went just on his initial gut feeling, he would have predicted more than two temperature records would have been broken. Instead, he looked at eight years of temperature records. In four of the last eight years, one or two records were broken. In the other years, either none or more than two had been broken. He and 37 per cent of superforecasters correctly predicted in February that only one or two broken records was most probable.
He says that regardless of his own point of view, he had to put that aside for the data, avoiding what psychologists call “belief perseverance,” maintaining a belief even when there is proof against it.
While the superforecasters had seen news stories throughout the summer about records being set in specific cities and average temperatures rising, average rises do not equate to records hit. Neither do records in cities necessarily translate to records in countries.
“People are reading about the warming climate, the hottest year of the world, etc, etc, etc and that encompasses everything as opposed to very, very specific locations and events,” said Roth.
Will Vladimir Putin cease to be the president of the Russian Federation before January 1 2024?
Both FT readers and superforecasters generally agreed that it was unlikely that Putin would no longer be president by the end of the year, but more superforecasters were sure Putin would stay in post (which we assume will be the case with only a couple of days remaining of 2023).
Hatch says that to avoid bias, it is best to establish a base rate. “What we really want to do is zoom out. How often do leaders in that strong of a position disappear over the course of a year for any reason? That is super, super low. It just doesn’t happen that often,” Hatch says.
Kahneman and longtime collaborator Amos Tversky call this balancing act the “outside and inside” perspectives. The outside view refers to the general case and the inside view refers to the specifics. Kahneman and Tversky found that people tend to anchor their judgments around the first number they consider, so to avoid anchoring biases, superforecasters are trained to anchor their probabilities first on the general, then adjust to the specifics of the case.
This perspective meant that for Stamp, his probabilities didn’t change much even after the Wagner uprising in June, when paramilitary forces led by Yevgeny Prigozhin staged a brief armed insurgency against Russian forces.
“For Russia, there’s little history of coups working,” Stamp says. “Wagner itself has never been part of the system and Prigozhin has always been an outsider in the Russian elite, with no real friends except for perhaps some generals or other criminals or whatever else. So . . . the chances seemed very low to me.”
What the superforecasters see for 2024
This weekend, senior writers from the FT will publish some of their predictions for 2024. Superforecasters are also scrutinising what the next twelve months will bring.
We spoke with six in mid-December, and asked them if they thought next year would be easier or harder to forecast, or about the same as 2023. They all agreed it might be about the same, but possibly more difficult with the large number of elections coming up around the world.
Stamp, who gave the year an 80 per cent chance of being the same difficulty and 10 per cent chances of it being respectively less or more difficult to predict, explained his reasoning in a lengthy email, with factors including the ages of China and Iran’s presidents. “Sorry for the long answer,” he concluded, “but in a sense you were lucky: this is only a short and quick version of a forecast. Usually, they are longer.”
Comments