Tim Fist

Yeah, so I think it's interesting to put this in the context of current events.

I think over the last few months, we've had all three of the leading US AI labs say that having the option for a global slowdown or pause in AI development is something that they would support.

So this is coming from DeepMind, Anthropic, and OpenAI, which is kind of like a big deal.

if we take them at their word for why they said they want this kind of thing, they think they're not that far off from building AI systems that can exhibit what's called recursive self-improvement or RSI.

And what that means is like an AI system that's capable of autonomously designing and then building its own successor.

And I think the risks that these people point to is if this happens, it could have two big consequences.

So one is on like the misuse of AI.

So if we see this rate of capability growth happening far exceeding what we've seen over the past few years, it could lead to sort of much greater risks in the near term future of people like using AI to do dangerous stuff.

And the other risks that these people point to is the risk of like loss of control.

So humans losing understanding of the AI systems that they're building, leading to the creation of a model that we can't control and we also don't understand how it works.

And so it could be misaligned with human interests.

And so, yeah, what...

These labs are calling for what we might want in such a situation is time for the world to take coordinated action so that like societal institutions and alignment research can keep up.

And what you really need for that is some way to verify that everyone's following those same rules and actually engaging in that kind of like coordinated slowdown.

So there's a couple of examples from the nuclear space about this fundamental idea of a technology enabled an agreement to happen.

One is the seismic monitoring system that allows treaties like the Comprehensive Test Brand Treaty, the CDBT, to happen, where because we have the technology to detect underground tests, 300 monitoring stations distributed globally across, you know, up to 100 countries,

those monitoring stations allow us to detect underground tests, which then allow you to have an agreement that bans underground tests.

Because without that technology, you would not be able to verify whether the agreement was being complied with.

And we did see that every single signatory to this treaty has not engaged in nuclear testing, which is sort of, in my view, a very big success story.

Another really interesting one is from the Intermediate-Range Nuclear Forces Treaty.