Stephen Wolfram

That's, you know, that messed the whole thing up.

But for us humans, we make distinctions which seem to correspond to the kinds of distinctions that neural nets make.

So then the thing that is just amazing to me about ChatGPT is how similar the structure it has is to the very original way people imagined neural nets might work back in 1943.

And there's a lot of detailed engineering, great cleverness, but it's really the same idea.

In fact, even the sort of elaborations of that idea where people said, let's put in some actual particular structure to try and make the neural net more elaborate to be very clever about it, most of that didn't matter.

When you train this neural net, the one thing this kind of transformer architecture, this attention idea, that really has to do with does every one of these neurons connect to every other neuron, or is it somehow causally localized, so to speak?

Is it like we're making a sequence of words and the words depend on previous words rather than just everything can depend on everything?

And that seems to be important in just organizing things so that you don't have a sort of a giant mess.

But the thing worth understanding about what is ChatGPT in the end?

I mean, what is a neural net in the end?

A neural net in the end is each neuron has a – it's taking inputs from a bunch of other neurons.

It's eventually – it's going to have a numerical value.

It's going to compute some number.

And it's saying, I'm going to look at the neurons above me.

It's kind of a series of layers.

It's going to look at the neurons above me, and it's going to say, what are the values of all those neurons?

Then it's going to add those up and multiply them by these weights.

And then it's going to apply some function that says if it's bigger than zero or something, then make it one, and otherwise make it zero, or some slightly more complicated function.

Right.

The real issue is, you know, this architecture of a neural net where what's happening is, you know, you've turned... So neural nets always just deal with numbers.