Warren Hatch
Appearances
3 Takeaways
The Science and Skill of Superforecasting (#230)
And tariffs, there's an incoming administration with a wish list, and that's high on the wish list. So some sort of tariffs seems in the bag. But the more significant question, certainly for people who are asking us for our forecast, is where, on what categories of goods, which countries are some of them gonna get exempted, or will it be an across the board thing? Exactly what does that look like?
3 Takeaways
The Science and Skill of Superforecasting (#230)
And what we're generally concluding for the moment is that just a blanket tariff, maybe a 10% probability of that occurring at the moment, We'll know more as we see who gets staffed in the different positions. But the higher risks are in specific categories and specific countries. And we're just unfolding those questions now.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Well, I think one thing that I get asked a lot is the role of artificial intelligence in forecasting. And there are certainly an awful lot of firms popping up. But here's the thing too, just like with experts, there's an artificial tension between AI and humans, which from our point of view is misplaced.
3 Takeaways
The Science and Skill of Superforecasting (#230)
We do know, because the studies have been out there, that at least for now, the humans continue to do better than artificial intelligence. Phil Tetlock did a big research project that came out a couple of months ago where they compared lots of different models None of them did better than the superforecasters and the best of them still lag by 20%. But what we've seen is that you want to have both.
3 Takeaways
The Science and Skill of Superforecasting (#230)
You can use the LLMs, you can use the AI models to help humans get to a better number faster. So I'm very optimistic about the future with AI from that point of view.
3 Takeaways
The Science and Skill of Superforecasting (#230)
First, slow down. We need to make lots of decisions every day, thousands, and most of those are going to be from the gut. But for the ones that really matter, slow down and use what Daniel Kahneman calls system two thinking. And you basically start having a dialogue with yourself. And that way you can challenge your thinking yourself.
3 Takeaways
The Science and Skill of Superforecasting (#230)
You can double check your assumptions, make sure you're not overly confident about what you think you know. The next thing is to change your mind. Expect to change your mind as new information becomes available. Number three is to keep score. At the end of it all, to get better, you need feedback. You need to see if your forecasts occur with the frequency that you think they do.
3 Takeaways
The Science and Skill of Superforecasting (#230)
You need to see if your reasoning aligns with how reality unfolds. So keep score, both for yourself to get better and better calibrated as a forecaster and to make, therefore, better decisions and extend that accountability to thought leaders, too.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Thank you, Lynn. It's a pleasure to be here. Thanks for having me on.
3 Takeaways
The Science and Skill of Superforecasting (#230)
In a sense, every decision that we make is a forecast because we're going to be taking action and doing things to improve the odds that we're going to get our desired outcome, whatever that might be. So we're doing that all day, all week, all year long, or thousands of decisions that way. So we want to make the best possible forecast.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And super forecasting is a process to get to the best possible forecast and therefore to get to the best possible decision.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Well, most people, and it's the dominant way, is they'll use language to express their views about the future. Somebody will ask them, well, do you think this will happen? And they'll say, well, maybe, maybe it will. Or they'll use fancier words like, well, there's a possibility or a distinct possibility. Now, here's the problem with that, many problems.
3 Takeaways
The Science and Skill of Superforecasting (#230)
One problem is we're all going to understand that in different ways. There is a famous example when Kennedy came into office and inherited a plan to invade Cuba and topple the regime. He asked his advisors, will this succeed? And they said, there's a fair chance it will succeed. Now, it turns out Kennedy had in mind north of 50%. The analysts had in mind something more like 25%.
3 Takeaways
The Science and Skill of Superforecasting (#230)
By using language, there was noise in that decision process. I imagine if Kennedy had known they had in mind 25%, history might have been a little different. So that is true for all kinds of words like that, where we're going to interpret it differently. Another bad thing about it is that it straddles for the 50-50 line. So if it happens, you say, aha, I said there'd be a distinct possibility.
3 Takeaways
The Science and Skill of Superforecasting (#230)
If it doesn't, well, I only said it was a distinct possibility. One other bad thing about it is it's impossible to put different views together. You can't crowdsource language like that because we're all using different words and in different ways. How much better to use a number? We all know what 72% is. We all know what that means.
3 Takeaways
The Science and Skill of Superforecasting (#230)
That's a great question, because sometimes there's an apparent tension between experts and good forecasters. where it's one is better than the other. We really are of the view that you want both. You want hybrid models when it comes to experts and things like that. And one core reason for that is experts might be good forecasters. But we don't know that until we see their track record.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Just being an expert does not make you a good forecaster. We want to see if when you say an 80% probability of something occurring, it occurs eight times out of 10 and doesn't two times out of 10. That's accuracy in a probabilistic sense. Most experts do not go through that process.
3 Takeaways
The Science and Skill of Superforecasting (#230)
They're very good at telling us what we need to understand, how we got where we are, some of the things we might want to watch. going forward. But what we've seen time and again, when you ask them for a probabilistic forecast, experts generally tend to assign higher probabilities to events in their area of expertise than actually occur.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And so you're better off by going to a crowd who are very skilled at assigning probabilities that occur with that frequency to give a forecast for that particular thing. Now, best of all is if you get an expert who applies themselves and becomes a good forecaster, that's what we really want to see. But experts, not necessarily good forecasters, just by being an expert.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Oh, I love that. That's a great question. And that's half the work is getting the question right. You want to make sure that everybody understands it in the same way. You want to be sure that it's actionable, it's useful. Why go to all this effort if it's just, you know, a parlor game?
3 Takeaways
The Science and Skill of Superforecasting (#230)
And you also want to be able to say, well, when that date occurs and you look back, we will agree it happened or it didn't. And a lot of forecast questions that are out there don't meet those criteria. There was a great example because in the original part of the project, the government wrote the questions themselves.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And in that first year or so, they had questions of the sort like, will construction begin on a canal through Nicaragua? And this was back when there was a guy in Hong Kong who was going to build a big, giant canal for the Supermax boats. And will construction begin there? was the question that they posed. Now, if you reflect a little bit, you can see that there's a problem with that.
3 Takeaways
The Science and Skill of Superforecasting (#230)
What counts as construction? Is it built with boats going through? Or is it big roads going through the jungle? Or is it a golden shovel in the ground? Which was my view. And sure enough, there was a golden shovel in the ground. But that was it. So did construction begin? We need to agree on what the question's asking. So the head of our question team
3 Takeaways
The Science and Skill of Superforecasting (#230)
The fail rate we have in our questions now is like 0% this year, thanks to his hard work. And what he'll do is in addition to writing a careful question, he'll include, it's almost like a little contract. He's trained as an attorney where this is how the question will resolve under these circuits. And for things that matter, it's worth going to that effort.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Not everything matters that much, but for things that do, then you want to make sure it's crisp and tight. It's actionable, it's verifiable, and we all can agree if it happened or it did not.
3 Takeaways
The Science and Skill of Superforecasting (#230)
For the higher order questions, things that really impact our lives, it's often not just a single decision. There might be a series of smaller decisions that go into it.
3 Takeaways
The Science and Skill of Superforecasting (#230)
and so breaking that down into what are the pieces that are really going to be impactful for how i think about this issue so where to go to school that's more than just a single decision there's a series of them do they provide the support that i would want as a student do they have the right kind of curriculum what sorts of career trajectory might i get on those are all smaller sub decisions and those are all things that you can then analyze as a forecasting question too
3 Takeaways
The Science and Skill of Superforecasting (#230)
Now, one thing that people tend to do, both when they're thinking about the question as well as the forecast, is focus on the particulars themselves. Like who's going to win the next presidential election? Most people immediately start thinking about the candidates. And what we want to do is instead of going to the particulars right away, we want to zoom out. How do things like this usually go?
3 Takeaways
The Science and Skill of Superforecasting (#230)
What Daniel Kahneman calls the outside view to get a sense of,
3 Takeaways
The Science and Skill of Superforecasting (#230)
what history can tell us what are the comparison classes out in the world where we can begin our forecasting where we can anchor ourselves most people will get anchored that's what happens so you want to anchor in the best possible place and then incrementally update from there so if you're thinking about schools and what the career trajectories are
3 Takeaways
The Science and Skill of Superforecasting (#230)
You'd want to go and take a look at all the data for comparable schools and see what the history of graduation rates and the like have been, and then go narrower and narrower into what Kahneman calls the inside view to make the decision itself.
3 Takeaways
The Science and Skill of Superforecasting (#230)
That's actually step one is defined. Another word for that is base rate. How do things like this usually go? And by going to a base rate or outside view, that was basically synonyms, you'll get an immediate boost just by doing that compared to the rest of the crowd.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Another good one for anyone to have on their checklist is to make a comment. When you've come up with an estimate about the future, jot down your rationale. One reason for that is it crystallizes your thinking. It also allows you to, in the future, look back and say, was I right for the right reasons or did I miss something? And it also allows you to share information with others.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And that's how you get a high quality forecast from a crowd. Most wisdom of the crowd approaches. You ask everybody. Then you take an average and you're done. That's where we start.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And what we'd like to do is have everyone make a comment and then exchange those ideas and see if somebody had a point that maybe I missed and then make an update, which is the next very crucial thing is expect to change your view, make an update. And you just do those few things. You're going to end up with a better number. Start with a base rate, make a comment, exchange views, update.
3 Takeaways
The Science and Skill of Superforecasting (#230)
One area where we learned a lot was during the pandemic. We were talking about experts earlier. And one thing about experts as well as artificial intelligence is they have models. That's the way they work. And if you have a model, then you're using backward-looking data to build your model. And that works great when we're in moments of relative equilibrium.
3 Takeaways
The Science and Skill of Superforecasting (#230)
But when things get upended and there is a lot of flux, those models don't work so well. And almost by definition, the experts are going to be slower to recognize that because their models will be filtering out the subtle changes that are eroding their models. And an excellent example of that was when COVID began to go global.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And many of the experts with their models were downplaying the significant, not all of them, but many of them. I think even most of them were saying that this thing will be contained. And instead, of course, it became something much more significant.
3 Takeaways
The Science and Skill of Superforecasting (#230)
Our forecasters who were not bound by a particular model and were skilled at identifying these subtle shifts did much better at identifying how quickly the pandemic would spread They also did very well at identifying when a vaccine would become available. They saw it becoming available far sooner than the experts did.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And that, of course, getting those two pieces has enormous implications for public policy, for investing, and for personal life decisions. So for us, that was a really critical example where we could also contribute to the public discourse on an important issue of about how to think about applying this process in real world situations, especially when there's a lot of flux going on.
3 Takeaways
The Science and Skill of Superforecasting (#230)
That's where this kind of a process can really shine is when things get upended, when there's more uncertainty than we recognized.
3 Takeaways
The Science and Skill of Superforecasting (#230)
it's a process that can be useful in everyday decision-making, but I think where it really stands out relative to other ways of thinking about the world is when there is a lot of flux and maybe not so much dark black swans as very dark gray swans. Because some of these things may be small probability, but high impact.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And somehow getting those small probability items on your radar can help you be better prepared. And in the case of the pandemic, there were some people who were identifying that as a possibility, not something that they were attaching high probabilities to, but it was definitely on their radar. And I think identifying those kinds of dark gray swans
3 Takeaways
The Science and Skill of Superforecasting (#230)
is part of a good, successful forecasting process. But once it hits, you're absolutely right. Now it's here. Now what do we do? Well, one thing we should do is discount how much reliance we have on the static models. We don't want to get rid of them completely, but the importance and the reliance we've previously had on them should be discounted.
3 Takeaways
The Science and Skill of Superforecasting (#230)
And we should instead factor in more weight to probabilistic judgment from people who are recognizing the small subtle factors that will eventually lead to new models.
3 Takeaways
The Science and Skill of Superforecasting (#230)
One is, yeah, another pandemic could be a dark gray swan, wildfires. has become such a thing where the frequency is much higher than it was. But where are they going to hit? That's the challenge. I think a lot of the AI developments can also fall into that category, what some of those risks might look like. And also possible global conflicts. One that we're looking at more is in Korea.
3 Takeaways
The Science and Skill of Superforecasting (#230)
We started looking at this a couple of months ago because there have been some changes. There have been some changes to North Korea's doctrine. North Korea sent troops to fight on Russia's behalf around Kursk. These are, in the global scheme of things, small but significant. So we're paying much more attention to how events on the Korean Peninsula may unfold as a possible dark, gray swan.