Eliezer Yudkowsky

Aren't you supposing something that's smart enough to be dangerous, but also stupid enough that it will just make paperclips and never question that?

In some cases, people are like, well, even if you like misspecify the objective function, won't you realize that what you really wanted was X?

Are you supposing something that is like smart enough to be dangerous, but stupid enough that it doesn't understand what the humans really meant when they specified the objective function?

Well, I'm saying that it's that, well, what I'm saying is like, what you think about artificial intelligence depends on what you think about intelligence.

So how do we think about intelligence correctly?

And also there's like, is made of John von Neumann and has like, and there's lots of them.

Because we understand that, yeah, we understand, like, Jonathan Newman is a historical case, so you can, like, look up what he did and imagine based on that.

And we know, like, people have, like, some intuition for, like, if you have more humans, they can solve tougher cognitive problems.

Although, in fact, like, in the game of Kasparov versus the world, which was, like, Garry Kasparov on one side, and...

an entire horde of internet people led by four chess grandmasters on the other side, Kasparov won.

So, like, all those people aggregated to be smarter.

It was a hard-fought game.

So, like, all those people aggregated to be smarter than any individual one of them, but they didn't aggregate so well that they could defeat Kasparov.

So humans aggregating don't actually get, in my opinion, very much smarter, especially compared to running them for longer.

The difference between capabilities now and a thousand years ago is a bigger gap than the gap in capabilities between 10 people and one person.

But even so, pumping intuition for what it means to augment intelligence, John von Neumann, there's millions of him.

He runs at a million times the speed and therefore can solve tougher problems, quite a lot tougher.

If one studies evolutionary biology with a bit of math,

And in particular, books from when the field was just sort of properly coalescing and knowing itself.

Not the modern textbooks, which are just like, memorize this legible math so you can do well on these tests.