Andrej Karpathy

speaker

36 appearances 1 recordings 1 series first heard Dec 2024 last heard Dec 2024

Andrej Karpathy’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

No recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.

newest first · ▶ plays the moment

think so, hundred percent.

And this is not like an obvious point or something that people maybe agree on right now, but I think hundred percent.

I I guess one thing I've been wondering about a little bit

small models, you know.

I think it can be surprisingly small.

And I do think that the current models are wasting a ton of capacity remembering stuff that doesn't matter.

Like they remember SHA hashes, they remember like the ancient.

Like and I think this will go away.

And I think we just need to get to the cognitive core.

And I think the cognitive core can be extremely small and is just this thing that thinks.

And if it needs to look up information, it knows how to use different tools.

I think even a billion billion suffices we'll probably get to that point.

And uh the models can be very, very small.

And I think the reason they can be very small is fundamentally I think just like distillation works is maybe like the only thing I would say.

Distillation works like surprisingly well.

Uh distillation is where you get a really big model or a huge amount of compute or something like that, um, supervising a very small model.

Showing 21–36 of 36 · page 2 of 2