Rob Wiblin

Well, I guess they have a very strong record of making mistakes.

They really do.

Yeah, I guess the net benefit explodes in this scenario.

I guess it's kind of a closer call if the UK goes alone.

I think it was within the error bounds that perhaps it wouldn't be worth it.

But if the UK demonstrates that this is possible to do technically, that it's not too expensive, not too impractical, then I suppose people have talked about this since I guess I was in high school and studying biology.

I remember people were worried about the risk from DNA synthesis.

Yes.

I guess centuries ago when I was in high school.

But I think interest is like really increasing around because it's all of the things that we've been talking about that the risk of people successfully pulling something like this off is going up.

So the chance that the EU and the US and China are all interested in emulating a successful program like this seems meaningful.

I guess it's certainly if we have like a near miss scare or something like that, then you could really see people rushing to do it.

And the reason that it works even if you're not at 100% is that imagine that you're a postdoc or something at a university lab and you're hoping to do something surreptitiously that's very dangerous.

But like almost all of the main players are screening and then you order from one of the handful from like Venezuela or something that aren't doing it.

Then that's like super sus to your university for like, well, maybe that would just like block you from getting anything from there or from bringing anything in.

And I guess alternatively to the intelligence services, they might well be able to find out that you're doing this and that puts you on a watch list, basically.

So I guess the biggest risk or the biggest challenge with this is that as the AI tools get better at obfuscating what proteins are, where they come from and what they might be used for, in order for this screening to be highly effective against a smart adversary, you need it to be extremely good at figuring out whether a given sequence is potentially able to contribute to a biological weapons program or to some extremely dangerous pandemic.

Yeah.

And then that model itself is an information hazard because it's basically a meter for determining whether something is dangerous and harmful or not.

So within the defensive acceleration framework, why isn't this actually not advantaging defenders?