Dwarkesh Patel

The best learners that we are aware of, which are children, are extremely bad at recollecting information.

In fact, at the very earliest stages of childhood, you will forget everything.

You're just an amnesiac about everything that happens before a certain year date.

But you're extremely good at picking up new languages and learning from the world.

And maybe there's some element of being able to see the forest for the trees.

Whereas if you compare it to the opposite end of the spectrum, you have...

LLM pre-training, which these models were literally able to regurgitate word for word what is the next thing in a Wikipedia page.

But their ability to learn abstract concepts really quickly the way a child can is much more limited.

And then adults are somewhere in between where they don't have the flexibility of childhood learning, but they can, you know, adults can memorize facts and information in a way that is harder for kids.

And I don't know if there's something interesting about that

And this is also relevant to preventing model collapse.

Let me think.

Hmm.

What is a solution to model collapse?

I mean, there's very naive things you could attempt.

It's just like the distribution over logits should be wider or something.

Like, there's many naive things you could try.

What ends up being the problem with the naive approaches?

In fact, it's actively penalized, right?

If you're like super creative in RL, it's like not good.