Adam Kucharski

Yeah I think that the story of of kind of how we rank evidence in medicine is a fascinating one I mean even just how long it took for people to think about these elements of randomization and fundamentally the what we're trying to do when we we have evidence here in medicine or science is prevent ourselves from confusing randomness for a signal I mean that's that's fundamentally you know we don't want to mistake something don't think it's going on it's not um

And the challenge, particularly with any intervention, is you only get to see one version of reality.

You can't give someone a drug, follow them, rewind history, not give them the drug, and then follow them again.

So one of the things that essentially randomization allows us to do is if you have two groups, one that's been randomized, one that hasn't, on average, the difference in outcomes between those groups is going to be down to the treatment effect.

It doesn't necessarily mean in reality that would be the case, but on average, that's the expectation that you'd have.

And it's kind of interesting, actually, that the first modern randomized controlled trial in medicine in 1947, this is for TB and streptomycin, the randomization element, actually, it wasn't so much statistical as behavioral.

If you have people come into hospital, you could...

some extent just say we'll just alternate we're not going to randomize we're just going to first patient we'll say is a control second patient treatment but what they found in a lot of previous studies was you know doctors have bias maybe that patient looks a little bit ill or that one yeah maybe is on borderline for eligibility and often you got these quite striking imbalances when you allowed it for human judgment so it's really about kind of shielding against those behavioral um

elements but I think there's a few situations it's a really powerful tool for a lot of these questions but as you mentioned one is this issue of you know you have the population you study on and then perhaps in reality how that translates elsewhere and we see I mean things like flu vaccines are a good example which are very dependent on immunity and evolution and what goes on in different populations sometimes you've had a result on a vaccine in one place and then the effectiveness doesn't translate in the same way to somewhere else I think the other really important thing to bear in mind is

as I said, it's the averaging that you're getting an average effect between two different groups.

And I think we see a certain lot of development around things like personalized medicine, where actually you want, you're much more interested in the outcome for the individual.

And so, you know, what a trial can give you evidence of is on average across a group, this is the effect that I can expect this intervention to have.

But we've now seen more of the emergence of things like N equals one studies where you can actually, you know, over the same individual, particularly kind of chronic conditions, look at those kind of interventions.

And also there's just these extreme examples where you're ethically not going to run a trial.

You know, there's never been a trial of whether it's a good idea to have intensive care units in hospitals.

Or there's a lot of these kind of historical treatments which are just so overwhelmingly effective.

that we're not going to run trials.

So almost this hierarchy over time, you can see it getting shifted because actually you do have these situations where other forms of evidence can get you either closer to what you need or just more feasibly an answer where it's just not ethical or practical to do an RCT.

Yeah, I think this is, as emerged, is a really valuable tool.

It's kind of interesting in the book, you're talking to economists like Josh Angrist, that a lot of these ideas emerged in epidemiology, but I think were really then taken up by economists, particularly as they wanted to add more credibility to a lot of these kind of policy questions.