Sam Altman

You know, it wasn't the underlying model that mattered.

It was the usability of it, both the RLHF and the interface to it.

So we train these models on a lot of text data.

And in that process, they learn the underlying something about the underlying representations of what's in here.

are in there and they can do amazing things but when you first play with that base model that we call it after you finish training it can do very well on evals it can pass tests it can do a lot of you know there's knowledge in there but it's not very useful

or at least it's not easy to use, let's say.

And RLHF is how we take some human feedback.

The simplest version of this is show two outputs, ask which one is better than the other, which one the human raters prefer, and then feed that back into the model with reinforcement learning.

And that process works remarkably well with, in my opinion, remarkably little data to make the model more useful.

So RLHF is how we

align the model to what humans want it to do.

Maybe just because it's much easier to use.

It's much easier to get what you want.

You get it right more often the first time, and ease of use matters a lot, even if the base capability was there before.

To be fair, we understand the science of this part at a much earlier stage than we do the science of creating these large pre-trained models in the first place, but yes, less data.

Much less data.

That's so interesting.

We spend a huge amount of effort pulling that together from many different sources.

There are open source databases of information.

We get stuff via partnerships.