Andrej Karpathy

You might think that...

Basically, you don't want to endow it with too much meaning with respect to the brain and how it works.

It's really just a complicated mathematical expression with knobs, and those knobs need a proper setting for it to do something desirable.

Yeah, I think that's fair.

So basically, I'm underselling it by a lot because you definitely do get very surprising emergent behaviors out of these neural nets when they're large enough and trained on complicated enough problems.

Like say, for example, the next word prediction in a massive dataset from the internet.

And then these neural nets take on pretty surprising magical properties.

Yeah, I think it's kind of interesting how much you can get out of even very simple mathematical formalism.

Well, it's definitely some kind of a generative model that's GPT-like and prompted by you.

So you're giving me a prompt, and I'm kind of responding to it in a generative way.

Well, it definitely feels like you're referencing some kind of a declarative structure of memory and so on.

And then...

you're putting that together with your prompt and giving away some answers.

Nothing, basically, right?

Yeah, could be.

I mean, I'm using phrases that are common, etc., but I'm remixing it into a pretty sort of unique sentence at the end of the day.

But you're right, definitely there's like a ton of remixing.

I mean, it's kind of interesting because I'm simultaneously underselling them, but I also feel like there's an element to which I'm over, like, it's actually kind of incredible that you can get so much emergent magical behavior out of them despite them being so simple mathematically.

So I think those are kind of like two surprising statements that are kind of juxtaposed together.

And I think basically what it is, is we are actually fairly good at optimizing these neural nets.