Dwarkesh Patel

We need X, Y and Z capabilities in these models.

Models keep getting more impressive at the rate that the short timelines people predict, but more useful at the rate that the long timelines people predict.

It's worth asking, what are we scaling?

With pre-trading, we had this extremely clean and general trend in improvement in loss across multiples orders of magnitude in compute.

Albeit, this was on a power law, which is as weak as exponential growth is strong.

But people are trying to launder the prestige that three-training scaling has, which is almost as predictable as a physical law of the universe, to justify bullish predictions about reinforcement learning from verifiable reward, for which we have no well-fit publicly known trend.

And when intrepid researchers do try to piece together the implications from scarce public data points, they get pretty bearish results.

For example, Toby Board has a great post where he cleverly connects the dots between the different O-series benchmarks.

And this suggested to him that, quote, we need something like a million X scale up in total RL compute to give a boost similar to a single GPT level, end quote.

So people have spent a lot of time talking about the possibility of a software-only singularity, where AI models will write the code that generates a smarter successor system, or a software plus hardware singularity, where AIs also improve their successor's computing hardware.

However, all these scenarios neglect what I think will be the main driver of further improvements atop AGI, continual learning.

Again, think about how humans become more capable than anything.

It's mostly from experience in the relevant domain.

Over conversation, Baron Millage made this interesting suggestion that the future might look like continual learning agents who are all going out and they're doing different jobs and they're generating value.

And then they're bringing back all their learnings to the hive mind model, which does some kind of batch distillation on all of these agents.

The agents themselves could be quite specialized, containing what Karpathy called the cognitive core, plus knowledge and skills relevant to the job they're being deployed to do.

Solving continual learning won't be a singular one-and-done achievement.

Instead, it will feel like solving in-context learning.

Now, GPT-3 already demonstrated in-context learning could be very powerful in 2020.

Its in-context learning capabilities were so remarkable, the title of the GPT-3 paper was Language Models Are Few-Shot Learners.