John List
speaker
84 appearances
2 recordings
2 series
first heard Dec 2024
last heard Apr 2025
John List’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
They are the people who are the fraction or the group of people who receive the largest program benefits.
That's one piece of it. Another piece is... who will sign their kids up for Head Start or for a program in a neighborhood that advances the reading skills of the child? Who's going to be first in line? The people who really care about education and the people who think their child will receive the most benefits from the program.
Now, another way to get it is sort of along the lines that you talked about. It could be the researcher knows something about the population. That other people don't know. Like, I want to give my program its best shot of working. Okay. And what's in your third bucket of scaling failures? The third bucket is something that we call the wrong situation was used.
And what I mean by that is that certain aspects of the situation change when you go from the original research to the scaled research program. we don't understand what properties of the situation or features of the environment will matter. There are a really large group of implementation scientists who have explored this question for years.
Now, what they emphasize and focus on is something called voltage drop. And voltage drop essentially means I found a really good result in my original research study. But then when they do it at scale, that voltage drop ends up being, for example, a tenth of the original result or a quarter of the original result.
An example of this is when you look at Head Start's home visiting services, what they do there is this is an early childhood intervention.
that found huge improvements in both child and parent outcomes in the original study, except when they tried to scale that up into home visits at a much larger scale, what they found is that, for example, home visits for at-risk families involved a lot more distractions in the house and there was less time on child-focused activities. So this is sort of
The wrong dosage or the wrong program is given at scale.
When you think about the chef, if a restaurant succeeds because of the magical work of the chef, and you think about scaling that, if you can't scale the magic in the chef, that's not scalable. Now, if the magic is because of the mix of ingredients, And the secret sauce, like Domino's, for example, the secret sauce or Papa John's is the actual ingredients, then that will be scalable.
Now, our proposal is that we do not believe that we should scale a program until you're 95% certain the result is true. So essentially what that means is we need the original research and then three or four well-powered, independent replications of the original findings.
My intuition is that they're probably not far away from three or four well-powered independent replications. In the hard sciences, in many cases, you not only have the original research, but you have a first replication also published in science. You know, the current credibility crisis in science is a serious one that major results are not replicating.
The reason why is because we weren't serious about replication in the first place. So this sort of puts the onus on policymakers and funding agencies in a sense of saying, we need to change the equilibrium.
Well, I think it's sort of a mix. I think it's fair to say that some policymakers are out looking for evidence to to base their preferred program on, what this will do is slow that down. If you have a pet project that you want to get through, fund the replications and let's make sure the science is correct. We think we should actually be rewarding scholars for attempting to replicate.
You know, right now in my community, if I try to replicate someone else, guess what I've just made? I've just made a mortal enemy for life. If you find a publishable result, what result is that? You're refuting previous research. Now I've doubled down on my enemy. So that's like a first step in terms of rewarding scholars who are attempting to replicate.
Now, to complement that, I think we should also reward scholars who have produced results that are independently replicated. You know what I'm talking about? Tying tenure decisions, grant money, and the like to people who have given us credible research that replicates.
Say I'm doing an experiment in Chicago Heights on early childhood and I find a great result. How confident should I be that when we take that result to all of Illinois or all of the Midwest or all of America, is that result still going to find that important benefit cost profile that we found in Chicago Heights? We need to know what is the magic sauce.
Was it the 20 teachers you hired down in Chicago Heights where if we go nationally, we need 20,000? So it should behoove me as an original researcher to say, look, if this scales up, we're going to need many more teachers. I know teachers are an important input. Is the average teacher in the 20,000 the same as the average teacher in the 20?
And the implementation scientists have focused on fidelity as a core component behind the voltage drop.
Thank you.
you Thank you.
Showing 21–40 of 84 · page 2 of 5
← Previous
Next →