Dwarkesh Patel

Like, okay, start with a lookup table and then go to a transformer, and then each piece is motivated.

Why would you add that?

Why would you add the next thing?

You couldn't memorize this sort of attention formula, but just like having an understanding of why every single piece is relevant, what problem it solves.

Because if you try to come up with it yourself, I guess you get a better understanding of, like, what is the action space and then what is the sort of, like, objective?

Then, like, why does only this action fulfill that objective, right?

Why do you think, by default, people who are genuine experts in their field are often bad at explaining it to somebody ramping up?

Another trick like that that just works astoundingly well.

If somebody writes a paper or a blog post or an announcement, it is in 100% of cases true that just the narration or the transcription of how they would explain it to you over lunch

is way more not only understandable, but actually also more accurate and scientific in the sense that people have a bias to explain things in the most abstract, jargon-filled way possible and to clear their throat for four paragraphs before they explain the central idea.

Yeah.

But there's something about communicating one-on-one with a person which compels you to just say the thing.

Right.

Exactly.

This is coming from the perspective of how somebody who's trying to explain an idea should formulate it better.

What is your advice to...

As a student to other students, where if you don't have a Karpathy who is doing the exposition of an idea, if you're reading a paper from somebody or reading a book, what strategies do you employ to learn material you're interested in in fields you're not an expert in?

Oh, yeah.

I think that's an excellent note to close on.

Yeah.