Eliezer Yudkowsky

Eventually.

You shouldn't do it first.

If you already have giant nuclear stockpiles, don't build more.

If some other country starts building a larger nuclear stockpile, then sure, even then, maybe just have enough nukes.

These things are not quite like nuclear weapons.

They spit out gold until they get large enough and then ignite the atmosphere and kill everybody.

Yeah.

And there is something to be said for not destroying the world with your own hands, even if you can't stop somebody else from doing it.

But open sourcing, no, that's just sheer catastrophe.

The whole notion of open sourcing, this was always the wrong approach, the wrong ideal.

There are places in the world where open source is a noble ideal and building stuff you don't understand that is difficult to control, that where if you could align it, it would take time.

You'd have to spend a bunch of time doing it.

That is not a place for open source because then you just have powerful things that just go straight out the gate without anybody having had the time to have them not kill everyone.

I do not believe in the practice of steel manning.

There is something to be said for trying to pass the ideological Turing test, where you describe...

your opponent's position, the disagreeing person's position well enough that somebody cannot tell the difference between your description and their description.

but steel manning, no.

Like- Okay, well, this is where you and I disagree here.

That's interesting.

Why don't you believe in steel manning?