Andrej Karpathy

And they think through things.

There's nothing in current LLMs that does this.

There's no equivalent of it.

But I do see papers popping out that are trying to do this because it's obvious to everyone in the field.

So I kind of see it as like, the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can fine-tune by imitation in humans.

And that was incredible.

Because in the beginning, all we had was base models.

Base models are autocomplete.

And it wasn't obvious to me at the time, and I had to learn this, and the paper that blew my mind was InstructGPT.

Because it pointed out that, hey, you can take the pre-trained model, which is autocomplete,

And if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become very conversational.

And it keeps all the knowledge from pre-training.

And this blew my mind because I didn't understand that this just like stylistically can adjust so quickly and become an assistant to a user through just a few loops of fine-tuning on that kind of data.

It was very miraculous to me that that worked.

So incredible.

And that was like two years, three years of work.

And now came RL.

And RL allows you to do a bit better than just imitation learning, right?

Because you can't have these reward functions and you can hill climb on the reward functions.

And so some problems have just correct answers.