Dwarkesh Patel

You two did some very interesting things about this.

Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment and in ways that seems almost irrespective of the final reward at the end of the episode?

If somebody's starting to start a business and at the end of 10 years she finds out whether the business succeeded or failed,

We say that she's earned a bunch of wisdom and experience, but it's not because like the log probs of every single thing that happened over the last 10 years are up-weighted or down-weighted.

It's something much more deliberate and rich is happening.

What is the ML analogy and how does that compare to what we're doing with other ones right now?

But you're so good at coming up with evocative phrases.

Sucking supervision through a straw is, like, so good.

Why hasn't—so you're saying, like, your problem with outcome-based reward is that you have this huge trajectory, and then at the end, you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit.

Why hasn't—given the fact that this is obvious—why hasn't process-based supervision—

as an alternative been a successful way to make models more capable?

What has been preventing us from using this alternative paradigm?

You're basically training the LLM to be a prompt injection model.

So to the extent you think this is the bottleneck to making RL more functional, then that will require making LLMs better judges if you want to do this in an automated way.

And then so is it just going to be like some sort of GAN-like approach where you had to train models to be more robust?

Interesting.

Do you have some shape of what the other idea could be?

Yeah.

So I guess I see a very, not easy, but like I can conceptualize how you would be able to train on synthetic examples or synthetic problems that you have made for yourself.

But there seems to be another thing humans do, maybe sleep is this, maybe daydreaming is this, which is not necessarily come up with fake problems, but just like reflect.