Dwarkesh Patel

The agent is in no substantial way learning from organic and self-directed engagement with the world.

Having to learn only from human data, which is an inelastic and hard to scale resource, is not a scalable way to use compute.

Furthermore, what these LLMs learn from training is not a true world model, which would tell you how the environment changes in response to different actions that you take.

Rather, they're building a model of what a human would say next.

And this leads them to rely on human-derived concepts.

A way to think about this would be, suppose you trained an LLM on all the data up to the year 1900.

That LLM probably wouldn't be able to come up with relativity from scratch.

And maybe here's a more fundamental reason to think this whole paradigm will eventually be superseded.

LLMs aren't capable of learning on the job, so we'll need some new architecture to enable this kind of continual learning.

And once we do have this architecture, we won't need a special training phase.

The agents will just be able to learn on the fly, like all humans, and in fact, like all animals are able to do.

And this new paradigm will render our current approach with LLMs and their special training phase that's super sample and efficient totally obsolete.

So that's my understanding of Rich's position.

My main difference with Rich is just that I don't think the concepts he's using to distinguish LLMs from true intelligence or animal intelligence are actually that mutually exclusive or dichotomous.

For example, I think imitation learning is continuous with and complementary to RL.

And relatedly, models of humans can give you a prior which facilitates learning quote-unquote true world models.

I also wouldn't be surprised if some future version of test-time fine-tuning could replicate continual learning, given that we've already managed to accomplish this somewhat with in-context learning.

So let's start with my claim that imitation learning is continuous with and complementary to RL.

So I tried to ask Richard a couple of times whether free-trained LLMs can serve as a good prior on which we can accumulate the experiential learning, aka do the RL, which would lead to AGI.

So Ilya Seskovor gave a talk a couple months ago that I thought was super interesting, and he compared pre-training data to fossil fuels.