Stephen Wolfram

It's like if you're running a company.

You can make a company be very stable for a certain period of time.

If your company gets bought by some private investment group, then you can run a company just fine for five years by just taking what it does and removing all R&D.

And the company will burn out after a while, but it'll run just fine for a little while.

So if you tell the AI, keep the humans okay for a thousand years, there's probably a certain set of things that one would do to optimize that.

Many of which one might say, well, that would be a pretty big shame for the future of history, so to speak, for that to be what happens.

But I think in the end, as you start thinking about that question, it is...

what you realize is there's a whole sort of raft of undecidability, computational irreducibility.

In other words, one of the good things about what our civilization has gone through and what we humans go through is that there's a certain computational irreducibility to it in the sense that it isn't the case that you can look from the outside and just say, the answer is going to be this.

At the end of the day, this is what's going to happen.

You actually have to go through the process to find out.

And I think that feels better in the sense that something is achieved by going through all of this process.

But it also means that telling the AI, go figure out what will be the best outcome.

Well, unfortunately, it's going to come back and say it's kind of undecidable what to do.

We'd have to run all of those scenarios to see what happens.

And if we want it for the infinite future, we're thrown immediately into sort of standard issues of kind of infinite computation and so on.

Right.

Well, I think it's saying, to summarize, this is the result of the universe.

If that is possible, it tells us the whole sort of structure of thinking about computation and so on and thinking about how stuff works.

If it's possible to say, and the answer is such and such, you're basically saying there's a way of going outside the universe.