Andrej Karpathy

And there's a huge amount of work to do efficiency.

You need your tensors, you lay them out, you stride them, you make sure your kernels are orchestrating memory movement correctly, et cetera.

It's all just efficiency, roughly speaking.

But the core intellectual sort of piece of neural network training is micrograph.

It's 100 lines.

You can easily understand it.

You're chaining.

It's a recursive application of chain rule to derive the gradient, which allows you to optimize any arbitrary differential function.

So I love finding these, like, you know, the smaller terms and serving them on a platter and discovering them.

And I feel like education is, like, the most intellectually interesting thing because you have a tangle of understanding, and you're trying to lay it out in a way that creates a ramp

where everything only depends on the thing before it.

And I find that this, like, you know, untangling of knowledge is just so intellectually interesting as a cognitive task.

And so I love doing it personally, but I just have fascination with trying to lay things out in a certain way.

Maybe that helps me.

Yeah, yeah.

Yeah, you're presenting the pain before you present a solution.

And how clever is that?

And you want to take the student through that progression.

So there's a lot of other small things like that that I think make it nice and engaging and interesting.

And always prompting the student.