Dwarkesh Patel

The difference, I guess, is that evolution...

has to be titrated in the case of humans through three gigabytes of DNA.

And so that's very unlike the weights of a model.

I mean, literally the weights of the model are a brain, which obviously is not encoded in the sperm and the egg, or does not exist in the sperm and the egg.

So it has to be grown.

And also the information for every single synapse in the brain simply cannot exist in the three gigabytes that exist in the DNA.

Evolution seems closer to finding the algorithm

which then does the lifetime learning.

Now, maybe the lifetime learning is not analogous to RL, to your point.

Is that compatible with the thing you were saying, or would you disagree with that?

Just to steel man the other perspective, because after doing this in an interview and thinking about it a bit, he has an important point here.

Evolution does not give us the knowledge, really, right?

It gives us the algorithm to find the knowledge.

And that seems different from pre-training.

So if perhaps the perspective is that pre-training helps build the kind of entity which can learn better, it teaches meta-learning.

and therefore it is similar to like finding an algorithm.

But if it's like evolution gives us knowledge and pre-training gives us knowledge, that analogy seems to break down.

There's so much interesting stuff there.

Okay, so let's start with in-context learning.

This is an obvious point, but I think it's worth just like saying it explicitly and meditating on it.