Dwarkesh Patel

An LLM that's being RL'd on outcome-based rewards learns on the order of one bit per episode, and an episode might be tens of thousands of tokens long.

Now, obviously, animals and humans are clearly extracting more information from interacting with our environment than just the reward signal at the end of an episode.

Conceptually, how should we think about what is happening with animals?

I think we're learning to model the world through observations.

This outer loop RL is incentivizing some other learning system to pick up maximum signal from the environment.

In Richard's oak architecture, he calls this the transition model.

And if we were trying to pigeonhole this feature spec into modern LLMs, what you do is fine tune on all your observed tokens.

From what I hear from my researcher friends, in practice, the most naive way of doing this actually doesn't work very well.

Now, being able to learn from the environment in a high throughput way is obviously necessary for true AGI.

And it clearly doesn't exist with LLMs trained on RLVR.

But there might be some other relatively straightforward ways to shoehorn continual learning atop LLMs.

For example, one could imagine making supervised fine tuning a tool call for the model.

So the outer loop RL is incentivizing the model to teach itself effectively using supervised learning in order to solve problems that don't fit in the context window.

Now, I'm genuinely agnostic about how well techniques like this will work.

I'm not an AI researcher.

but I wouldn't be surprised if they basically replicate continual learning.

And the reason is that models are already demonstrating something resembling human continual learning within their context windows.

The fact that in-context learning emerged spontaneously from the training incentive to process long sequences makes me think that if information could just flow across windows longer than the context limit, then models could meta-learn the same flexibility that they already show in context.

Okay, some concluding thoughts.

Evolution does meta-RL to make an RL agent, and that agent can selectively do imitation learning.