Yann LeCun
speaker
384 appearances
5 recordings
4 series
first heard Mar 2024
last heard 20 Jun
Yann LeCun’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 4 in all, peaking in Jan 2026 with 2.
Appearances
We're not going to be able to do this with the type of LLMs that we are working with today. And there's a number of reasons for this. But the main reason is The way LLMs are trained is that you take a piece of text, you remove some of the words in that text, you mask them, you replace them by blank markers, and you train a genetic neural net to predict the words that are missing.
And if you build this neural net in a particular way so that it can only look at words that are to the left of the one it's trying to predict, then what you have is a system that basically is trying to predict the next word in a text, right? So then you can feed it a text, a prompt, and you can ask it to predict the next word. It can never predict the next word exactly.
And so what it's going to do is produce a probability distribution over all the possible words in your dictionary. In fact, it doesn't predict words, it predicts tokens that are kind of subword units. And so it's easy to handle the uncertainty in the prediction there, because there is only a finite number of possible words in the dictionary, and you can just compute a distribution over them.
Then what the system does is that it picks a word from that distribution. Of course, there's a higher chance of picking words that have a higher probability within that distribution. So you sample from the distribution to actually produce a word. And then you shift that word into the input. And so that allows the system not to predict the second word, right?
And once you do this, you shift it into the input, et cetera. That's called autoregressive prediction, which is why those LLMs should be called autoregressive LLMs. But we just call them LLMs. And there is a difference between this kind of process and a process by which before producing a word, when you talk, when you and I talk, You and I are bilingual.
We think about what we're going to say, and it's relatively independent of the language in which we're going to say it. When we talk about, I don't know, let's say a mathematical concept or something, the kind of thinking that we're doing and the answer that we're planning to produce is is not linked to whether we're going to see it in French or Russian or English.
Right. It's certainly true for a lot of thinking that we do.
Yeah, pretty much.
Well, it depends what kind of thinking, right? If it's just, if it's like producing puns, I get much better in French than English about that.
There is an abstract representation of imagining the reaction of a reader to that text.
Or figure out a reaction you want to cause and then figure out how to say it so that it causes that reaction. But that's really close to language. But think about a mathematical concept or imagining something you want to build out of wood or something like this, right? the kind of thinking you're doing is absolutely nothing to do with language, really.
It's not like you have necessarily an internal monologue in any particular language. You're imagining mental models of the thing, right? If I ask you to imagine what this water bottle will look like if I rotate it 90 degrees, that has nothing to do with language.
And so clearly there is a more abstract level of representation in which we do most of our thinking and we plan what we're gonna say if the output is you know, uttered words as opposed to an output being, you know, muscle actions, right? We plan our answer before we produce it. And LLMs don't do that. They just produce one word after the other instinctively, if you want.
It's like, it's a bit like the, you know, subconscious actions where you don't, Like you're distracted, you're doing something, you're completely concentrated, and someone comes to you and asks you a question, and you kind of answer the question. You don't have time to think about the answer, but the answer is easy, so you don't need to pay attention. You sort of respond automatically.
That's kind of what an LLM does, right? It doesn't think about its answer, really. It retrieves it because it's accumulated a lot of knowledge, so it can retrieve some things, but it's going to just spit out one token after the other without planning the answer.
Okay, but then that assumes that those systems actually possess an eternal world model.
Yeah. So can you build this, first of all, by prediction? Right. And the answer is probably yes. Can you build it by predicting words? And the answer is most probably no, because language is very poor in terms of weak or low bandwidth, if you want. There's just not enough information there.
So building world models means observing the world and understanding why the world is evolving the way it is. And then the extra component of a world model is something that can predict how the world is going to evolve as a consequence of an action you might take, right? So one model really is, here is my idea of the state of the world at time t. Here is an action I might take.
What is the predicted state of the world at time t plus 1? Now that state of the world does not need to represent everything about the world. It just needs to represent enough that's relevant for this planning of the action, but not necessarily all the details. Now here is the problem. You're not going to be able to do this with generative models.
So a generative model that's trained on video, and we've tried to do this for 10 years. You take a video, show a system a piece of video, and then ask it to predict the reminder of the video. Basically, predict what's going to happen.
Showing 41–60 of 384 · page 3 of 20
← Previous
Next →