Dwarkesh Patel

Now, obviously these engineers evaluated these interactions on a pass-fail basis, but they also rated every single response on a bunch of different dimensions like readability and performance.

And they wrote down their thought processes for every single rating that they gave.

So you're basically showing every single step an engineer takes and every single thought that they have while they're doing their job.

And this is just something you could never get from usage data alone.

And so LabelBox packaged up all these evaluations and included all the agent trajectories and the corrective human edits for the customer to train on.

This is just one example, so go check out how Labelbox can get you high-quality frontier data across domains, modalities, and training paradigms.

Reach out at labelbox.com slash thwarkesh.

Let's talk about RL a bit.

You two did some very interesting things about this.

Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment and in ways that seems almost irrespective of the final reward at the end of the episode?

If somebody's starting to start a business and at the end of 10 years she finds out whether the business succeeded or failed,

We say that she's earned a bunch of wisdom and experience, but it's not because like the log probs of every single thing that happened over the last 10 years are up-weighted or down-weighted.

It's something much more deliberate and rich is happening.

What is the ML analogy and how does that compare to what we're doing with other ones right now?

But you're so good at coming up with evocative phrases.

Sucking supervision through a straw is, like, so good.

Why hasn't—so you're saying, like, your problem with outcome-based reward is that you have this huge trajectory, and then at the end, you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit.

Why hasn't—given the fact that this is obvious—why hasn't process-based supervision—

as an alternative been a successful way to make models more capable?

What has been preventing us from using this alternative paradigm?