Scott Alexander

What I would like to see is society as a whole starting to freak out as the trend lines start upwards and things get automated and you have these fully autonomous agents and they start using neural leads and hive minds.

As all that exciting stuff starts happening in the data centers, I would like it to be the case that the public is following along and then getting activated and all of these other researchers are like,

you know, reading the safety case and critiquing it and like doing little ML experiments on their own tiny compute clusters to like examine some of the assumptions in the safety case and so forth.

And, you know, basically like I think that a sort of one way of summarizing it is that like currently there's going to be like 10 alignment experts in whatever inner silo of whatever company is in the lead.

And like the technical issue of making sure that AIs are actually aligned is going to fall roughly to them.

But what I would like to be is a situation where it's more like 100 or like 500 alignment experts spread out over different companies and in nonprofits that are sort of like all communicating with each other and working on this together.

I think we're substantially more likely to make things, you know, get the technical stuff right if it's something like that.

Totally.

I mean, I also have worried about that exact example.

I would summarize the situation as the government lacks the expertise and the companies lack the right incentives.

And so...

It's a terrible situation.

I think that if the government wades in and tries to make more specific regulations along the lines of what you mentioned, it's very plausible that it'll end up backfiring for reasons like what you mentioned.

On the other hand, if we just trust it to the companies, they're in a race with each other, and they're full of people who have convinced themselves that this is not a big deal.

There's other types of transparency too.

So transparency about capabilities and transparency about the spec and the governance structure.

So for the capabilities thing, that's pretty simple.

It's like, if you're doing an intelligence explosion, you should keep the public informed about that.

When you've finally got your automated army of AI researchers that are completely automating the whole thing on the data center, you should tell everyone, like, hey guys, FYI, this is what's happening now.

It really is working.