Andrej Karpathy

And everything has to be very precisely arranged.

And the models are always trying to, they kept trying to, I mean, they have so many cognitive deficits, right?

So one example, they keep trying to, they keep misunderstanding the code because they have too much memory from all the typical ways of doing things on the internet that I just wasn't adopting.

So the models, for example, I mean, I don't know if I want to get into the full details, but they keep thinking I'm writing normal code and I'm not.

Maybe one example.

Maybe one example is, so the way to synchronize, so we have eight GPUs that are all doing forward backwards.

The way to synchronize gradients between them is to use a distributed data parallel container of PyTorch, which automatically does all the, as you're doing the backward, it will start communicating and synchronizing gradients.

I didn't use DDP because I didn't want to use it because it's not necessary.

So I threw it out.

And I basically wrote my own synchronization routine that's inside the step of the optimizer.

And so the models were trying to get me to use the DDP container, and they were very concerned about, okay, this gets way too technical, but I wasn't using that container because I don't need it, and I have a custom implementation of something like it.

Yeah, they couldn't get past that.

and then um they kept trying to like mess up the style like they're way too over defensive they make all these try catch statements they keep trying to make a production code base and i have a bunch of assumptions in my code and it's okay and uh and it's just like i don't need all this extra stuff in there and so i just kind of feel like they're bloating the code base they're bloating the complexity they keep misunderstanding they're using deprecated apis a bunch of times so it's total mess um

and it's just not that useful.

I can go in and I can clean it up, but it's not that useful.

I also feel like it's kind of annoying to have to, like, type out what I want in English because it's just too much typing.

Like, if I just navigate to the part of the code that I want and I go where I know the code has to appear and I start typing out the first three letters, autocomplete gets it and just gives you the code.

And so I think it's... This is a very high-information bandwidth to specify what you want.

If you point to the code where you want it and you type out the first few pieces, and the model will complete it.

So I guess what I mean is...