Stephen Wolfram

And so, you know, you've turned the sentence that you started with into a bunch of numbers.

Like, let's say, by mapping, you know, each word of the 50,000 words in English, you just map each word or each part of a word into some number.

You feed all those numbers in.

And then the thing is going to... And then those numbers just go into the values of these neurons.

And then what happens is it's just rippling down, going layer to layer until it gets to the end.

I think ChatGPG has about 400 layers.

And it just goes once through.

Every new word it's going to compute just says, here are the numbers from the words before.

Let's compute...

What does it compute?

It computes the probabilities that it estimates for each of the possible 50,000 words that could come next.

And then it decides, sometimes it will use the most probable word, sometimes it will use not the most probable word.

It's an interesting fact that there's this so-called temperature parameter, which at temperature zero, it's always using the most probable word that it estimated was the most probable thing to come next.

You know, if you increase the temperature, it'll be more and more kind of random in its selection of words.

It'll go down to lower and lower probability words.

The thing I was just playing with actually recently was the transition that happens as you increase the temperature.

The thing goes bonkers at a particular, you know, sometimes at a particular temperature, maybe about 1.2 ish.

the thing I was noticing from yesterday, actually, that usually it's giving reasonable answers.

And then at that temperature, with some probability, it just starts spouting nonsense.

And nobody knows why this happens.