Andrej Karpathy

We're probably going to have a lot better hardware.

We're probably going to have a lot better kernels and software.

We're probably going to have better algorithms.

And all of those, it's almost like no one of them is winning too much.

All of them are surprisingly equal.

And this has kind of been the trend for a while.

So I guess to answer maybe your question, I expect differences algorithmically to what's happening today.

But I do also expect that some of the things that have stuck around for a very long time will probably still be there.

It's probably still a giant neural network trained with gradient descent.

That would be my guess.

But I guess what was shocking to me is everything needs to improve across the board.

Architecture, optimizer, loss function, and also has improved across the board forever.

So I kind of expect all those changes to be alive and well.

Building NanoChat?

So NanoChat is a kind of a repository I released.

Was it yesterday or the day before?

I can't remember.

We can see this lead generation that went into the... Well, it's just trying to be a...

It's trying to be the simplest, complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone.

And so, you know, you have all of the steps, not just any individual step, which is a bunch of... I worked on all the individual steps sort of in the past and released small pieces of code that kind of show you how that's done in algorithmic sense in like simple code.