John Schulman

So even if the models are good enough to actually run a successful business themselves,

Um, so, uh, yeah, to some extent there might be, uh, choices there.

Um, and, uh, I think people will still have different interests, uh, and what they want to different ideas for what kind of, uh, interesting pursuits they want to direct their AIs at.

And, uh, like they can, people, people could, uh, like, um, you

Yeah, do a lot of, AI doesn't necessarily have an intrinsic, like any kind of intrinsic desire.

Not yet.

Unless we put it in the system.

So I think, so people can still end up being, even if AI's like become extremely capable, I would hope that people are still the drivers of like what the AI's end up doing.

if we wanted to keep, uh, humans in the loop, uh, which seems reasonable.

Um, and, uh, it turned out that, um, firms with any humans in the loop were out competed with by firms that didn't have any humans.

Then I think then you would obviously need some kind of regulation that, uh, like disallowed, um, having no humans in the loop for running a whole company.

Right.

Yeah, you would either have to have every country agree to this regulatory regime, or you would need all of the model infrastructure or the model providers to agree to this kind of requirement.

So it's definitely going to be non-trivial.

So I guess...

Yeah, this is looking a ways ahead, so it's a little hard to imagine this world before seeing anything like it.

Uh, so for example, uh, like there's some questions like, would, uh, are we actually confident that, uh, AI run companies are, uh, better in every way?

Or, uh, do we think they're better most of the time, but occasionally they, um, malfunction because AIs are still like, they're still less sample efficient in certain ways, like dealing with very wacky situations.

So, um,

So actually, AI-run firms have higher tail risk because they're more likely to malfunction in a big way.