Scott Alexander

Happily, we went the way of LLMs first where the broad world understanding came first.

And then now we're trying to turn them into agents.

Not saying that.

I mean, I think I also don't want China to get to superintelligence before the U.S.

That would be quite bad.

Yeah, it's a tricky thing that we're going to have to do.

Like, people ask about PDoom, right?

And, like, my PDoom is sort of infamously high, like 70%.

Oh, wait, really?

Oh, well, that's what it is.

And, like, part of the reason for that is just that, like, I feel like a bunch of stuff has to go right.

Like, I feel like we can't just, like, unilaterally slow down and have China go take the lead.

That also is a terrible future.

Yeah.

But, like...

we can't also just like completely race because for the reasons I mentioned previously about alignment, I think that if we just, you know, go all out on racing, we're going to lose control of our AIs, right?

And so we have to somehow like thread this needle of like pivoting and doing more, you know, alignment research and stuff, but not too much that like...

Helps China win, you know?

And that's all just for the alignment stuff, but then there's like the concentration of power stuff where like somehow in the middle of doing all of that, the powerful people who are involved need to like somehow negotiate a truce between themselves to share power and then ideally spread that power out amongst the government and get the legislative branch involved.

You know, somehow that has to happen too.