Gwern Branwen

and they just don't know that BigGAN handled 300 million images without a sweat.

If you don't know that, then I think you'd probably easily think, oh, GANs are broken.

But if you do know that, then you think to yourself, how can algorithms be so important when all these different generative architectures all work so well, as long as you have lots and lots of GPUs?

That's the common ingredient, right?

You have to have lots and lots of GPUs.

Yeah, I would say it was very far away from 2005 to 2010.

It was somewhere well past 2050.

It was close enough that I thought I might live to see it, but I was not actually sure if there was any reasonable chance.

But once AlexNet and DanNet came out, then it just kind of kept dropping at a rate of like two years per year, every year, basically until now.

We just kept hitting on barriers to deep learning doing better.

And I think regardless of how it was doing it, it was obviously getting way better.

It just seemed like none of the alternative paradigms were really doing that well, and this one was doing super well.

Yeah, there were a few times where I thought I had overshot.

I thought people over-updated on AlphaGo.

They went too far on AI hype with AlphaGo, I think.

And then afterwards, when pushes into big reinforcement learning efforts had kind of fizzled out, like post-Dota, as the reinforcement learning wasn't working out for solving all those hard problems outside of the simulated game universes.

Then I started thinking, okay, maybe we kind of overshot.

But then GPT came out of nowhere and basically erased all of that.

It was kind of this like, oh shit, here's how RL is going to work.

It's going to be the cherry on this cake.