Dr. Terry Sejnowski

Yes. I think that that's an example, but it turns out that every word is ambiguous. It has like three, four meanings. And so you have to figure that out from context. And so in other words, there are words that live together. and that come up often. And you can learn that from just by predicting the next word in a sentence. That's how a transformer is trained.

Yes. I think that that's an example, but it turns out that every word is ambiguous. It has like three, four meanings. And so you have to figure that out from context. And so in other words, there are words that live together. and that come up often. And you can learn that from just by predicting the next word in a sentence. That's how a transformer is trained.

You give it a bunch of words and it keeps predicting the next word in a sentence.

You give it a bunch of words and it keeps predicting the next word in a sentence.

You give it a bunch of words and it keeps predicting the next word in a sentence.

Okay, well, that's because it's a very primitive version of this algorithm. What happened is if you train it up on enough, not only can it answer the next word, it internally builds up a semantic representation in the same way you describe the words that are related to each other, having associations.

Okay, well, that's because it's a very primitive version of this algorithm. What happened is if you train it up on enough, not only can it answer the next word, it internally builds up a semantic representation in the same way you describe the words that are related to each other, having associations.

Okay, well, that's because it's a very primitive version of this algorithm. What happened is if you train it up on enough, not only can it answer the next word, it internally builds up a semantic representation in the same way you describe the words that are related to each other, having associations.

It can figure that out and it has representations inside this very large network with trillions of parameters. It's unbelievable how big they've gotten. And those associations now form an internal model of the meaning of the sentence. Literally, this is something that now we've probed these transformers, and so we pretty much are pretty confident.

It can figure that out and it has representations inside this very large network with trillions of parameters. It's unbelievable how big they've gotten. And those associations now form an internal model of the meaning of the sentence. Literally, this is something that now we've probed these transformers, and so we pretty much are pretty confident.

It can figure that out and it has representations inside this very large network with trillions of parameters. It's unbelievable how big they've gotten. And those associations now form an internal model of the meaning of the sentence. Literally, this is something that now we've probed these transformers, and so we pretty much are pretty confident.

And that means that it's forming an internal model of the outside world, in this case, a bunch of words. And that's how it's able to actually respond to you in a way that is sensible, that makes sense and actually is interesting and so forth. And it's all the self-attention I'm talking about. So in any case, my pioneer proposal is to figure out how does the brain do self-attention, right?

And that means that it's forming an internal model of the outside world, in this case, a bunch of words. And that's how it's able to actually respond to you in a way that is sensible, that makes sense and actually is interesting and so forth. And it's all the self-attention I'm talking about. So in any case, my pioneer proposal is to figure out how does the brain do self-attention, right?

And that means that it's forming an internal model of the outside world, in this case, a bunch of words. And that's how it's able to actually respond to you in a way that is sensible, that makes sense and actually is interesting and so forth. And it's all the self-attention I'm talking about. So in any case, my pioneer proposal is to figure out how does the brain do self-attention, right?

It's got to do it somehow. And I'll give you a little hint. Basal ganglia.

It's got to do it somehow. And I'll give you a little hint. Basal ganglia.

It's got to do it somehow. And I'll give you a little hint. Basal ganglia.

That's my hypothesis. Well, we'll see. I mean, you know, I'll be working with experimental people. I've worked with John Reynolds, for example, who studies primate visual cortex. And we've looked at traveling ways there. And there are other people that have looked at in primates.

That's my hypothesis. Well, we'll see. I mean, you know, I'll be working with experimental people. I've worked with John Reynolds, for example, who studies primate visual cortex. And we've looked at traveling ways there. And there are other people that have looked at in primates.

That's my hypothesis. Well, we'll see. I mean, you know, I'll be working with experimental people. I've worked with John Reynolds, for example, who studies primate visual cortex. And we've looked at traveling ways there. And there are other people that have looked at in primates.