Dwarkesh Patel

You're basically training the LLM to be a prompt injection model.

So to the extent you think this is the bottleneck to making RL more functional, then that will require making LLMs better judges if you want to do this in an automated way.

And then so is it just going to be like some sort of GAN-like approach where you had to train models to be more robust?

Interesting.

Do you have some shape of what the other idea could be?

Yeah.

So I guess I see a very, not easy, but like I can conceptualize how you would be able to train on synthetic examples or synthetic problems that you have made for yourself.

But there seems to be another thing humans do, maybe sleep is this, maybe daydreaming is this, which is not necessarily come up with fake problems, but just like reflect.

Yeah.

And I'm not sure what the ML analogy for, you know, daydreaming or sleeping, but just like just reflecting.

I haven't come up with a new problem.

Yeah, yeah.

I mean, obviously, the very basic analogy would just be fine-tuning on reflection bits, but I feel like in practice that probably wouldn't work that well.

So I don't know if you have some take on what the analogy of this thing is.

Just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution?

I guess what I'm saying is...

You can't just keep scaling, quote-unquote, reflection on the same amount of prompt information and then get returns from that.

Have you seen this super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse?

That the reason dreaming is an evolutionary adaptive is to put you in weird situations that are very unlike your day-to-day reality to prevent this kind of overfitting?

This is a very ill-formed thought, so I'll just put it out and let you react to it.