Scott Alexander

What does targeting along mean?

So from the perspective of misaligned AIs, you wouldn't want to kill the humans or get into a war with them if you're going to get wrecked because you need the humans to maintain your computers, right?

So yeah, in our scenario, once they are completely self-sufficient, then they can start being more blatantly misaligned.

And so I'm curious, when would they be fully self-sufficient?

Not in the sense of like,

they're not literally using the humans at all, but in the sense of like, they don't really need the humans anymore.

Like they can get along pretty fine without them.

They can continue to like do their science.

They can continue to expand their industry.

They can continue to have a flourishing civilization, you know, indefinitely into the future without any humans.

Like 10 years, basically, instead of one year.

I think we agree on the core model.

This is why we didn't depict something more like the bathtub nanotech scenario, where they just don't need to do the experiments very much, and they just immediately jump to the right answers.

We are imagining this process of learning by doing,

through this distributed across the economy, lots of different laboratories and factories building different things, learning from them, et cetera.

We're just imagining that this overall goes much faster than it would go if humans were in charge.

And then we do have, in fact, lots of uncertainty, of course.

Dividing up this part period into two chunks, the early 2028 until fully autonomous robot economy part, and then the fully autonomous robot economy to cancer cures, nanobots, all that crazy sci-fi stuff,

I want to separate them because, like, the important parts for a scenario only depend on the first part, really.

If you think that it's going to take, like, 100 years to get to nanobots, that's fine, whatever.