Eliezer Yudkowsky

If you're like breeding on restrained populations.

And

So that's like the sort of... So this is not a smart optimization process.

Natural selection is like so incredibly stupid and simple that we can actually quantify how stupid it is if you like read the textbooks with the math.

Nonetheless, this is the sort of basic thing of you look at this alien optimization process and there's the thing that you...

hope it will produce.

And you have to learn to clear that out of your mind and just think about the underlying dynamics and where it finds the maximum from its standpoint that it's looking for, rather than how it finds that thing that leapt into your mind as the beautiful aesthetic solution that you hope it finds.

And this is something that has been fought out historically as the field of biology was coming to terms with evolutionary biology.

And you can look at them fighting it out as they get to terms with this very alien inhuman optimization process.

And indeed, something smarter than us would be also much like smarter than natural selection.

So it doesn't just automatically carry over.

But there's a lesson there.

There's a warning.

Well, it's kind of stupid.

It has to run hundreds of generations to notice that something is working.

It doesn't be like, oh, well, I tried this in one organism.

I saw it worked.

Now I'm going to duplicate that feature onto everything immediately.

It has to run for hundreds of generations for a new mutation to rise to fixation.

It runs for a long time and eventually manages to optimize things.