Max Tegmark

In a nuclear reaction, if each one can make more than one, then you get an exponential growth in that.

We call it a nuclear explosion.

All explosions are like that.

And an intelligence explosion, it's just exactly the same principle, that some amount of intelligence can make more intelligence than that, and then repeat.

You always get exponentials.

It's obviously gonna stop when it bumps up against the laws of physics.

There are some things you just can't do no matter how smart you are, right?

Because we don't know the full laws of physics yet, right?

Seth Lloyd wrote a really cool paper on the physical limits on computation, for example, if you make it...

too much energy into it and the finite space it'll turn into a black hole you can't move information around faster than the speed of light stuff like that but uh it's hard to store way more than than a modest number of bits per atom etc but you know those limits are just astronomically above like 30 orders of magnitude above where we are now so

bigger different bigger jump in intelligence than if you go from a from an ant to a human i think of course what we want to do is have have a controlled thing a nuclear reactor you put moderators in to make sure exactly it doesn't blow up out of control right when we do um

experiments with biology and cells and so on.

We also try to make sure it doesn't get out of control.

We can do this with AI too.

The thing is, we haven't succeeded yet.

And Moloch is exactly doing the opposite, just fueling, just egging everybody on faster, faster, faster, or the other company is going to catch up with you or the other country is going to catch up with you.

We have to want this stuff.

And I don't believe in this, just asking people to look into their hearts and do the right thing.

It's easier for others to say that.

But if you're in a situation where your company is going to get screwed, by other companies that are not stopping, you're putting people in a very hard situation.