Ajeya Cotra

Yeah, there's epistemic disruption, which you mentioned.

I guess new competitive pressures kind of concerns that you can end up in a sort of Malthusian situation if you have competition between many different AIs and possibly some others that are missing here.

But there's, I guess, many other, like, I guess we don't know which of these are going to loom large at the time.

Some of them might feel like they've kind of been addressed or perhaps that we were hallucinating issues that aren't so severe.

But yeah, there's many different ways that we could potentially apply it.

So you said in your notes that you think this approach is basically what all of the frontier AI companies say.

This is their safety plan, more or less.

Is that right?

Yeah, what assumptions are necessary for this approach to make sense?

Or what kinds of setups could actually just make it a bad plan?

There's something that's a bit self-contradictory about that because an AI that can – it's like extremely smart, but all it can do is improve the sample efficiency of the next model is in a sense like not very troubling, but in itself because it doesn't have like general capabilities.

That kind of model isn't going to be able to take over or invent other technologies.

It's only at the point that it has the broader –

capabilities, the broader agency that actually is able to make problems.

But I guess you're saying you can have a long lead up where that's all that it can do.

And then at the last stage.

Yeah, I guess many of the problems that we'd like it to help with are issues like social issues, political issues, philosophical issues in some cases.

What do you think is the chances that AI, I mean, the companies, I think they're working harder to make them good at coding and to make them good at AI research than any other particular thing.

And I guess those are more concrete measurable problems than solving philosophical questions.

So it seems like it is really a live risk that unfortunately the balance of capabilities will end up being pretty disadvantageous for this plan.