Ilya Sutskever

We got to the point where we are in a world where there are more companies than ideas by quite a bit.

Actually on that, you know, there is the Silicon Valley saying that says that ideas are cheap, execution is everything.

And people say that a lot.

And there is truth to that.

But then I saw someone say on Twitter something like, if ideas are so cheap, how come no one's having any ideas?

And I think it's true too.

I think if you think about research progress in terms of bottlenecks,

There are several bottlenecks.

And one of them is ideas, and one of them is your ability to bring them to life, which might be compute, but also engineering.

So if you go back to the 90s, let's say, you had people who had pretty good ideas.

And if they had much larger computers, maybe they could demonstrate that their ideas were viable, but they could not.

So they could only have very, very small demonstration that did not convince anyone.

So the bottleneck was compute.

Then in the age of scaling, computers increased a lot.

And of course there is a question of how much compute is needed, but compute is large.

So compute is large enough such that it's like not obvious that you need that much more compute to prove some idea.

Like I'll give you an analogy.

AlexNet was built on two GPUs.

That was the total amount of compute use for it.

The Transformer