Yann LeCun
speaker
384 appearances
5 recordings
4 series
first heard Mar 2024
last heard 20 Jun
Yann LeCun’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 4 in all, peaking in Jan 2026 with 2.
Appearances
You have a model of the system you want to control that can predict the sequence of states corresponding to a sequence of commands. And you're planning a sequence of commands so that, according to your world model, the end state of the system will satisfy an objective that you fix. This is the way...
Rocket trajectories have been planned since computers have been around, so since the early 60s, essentially.
Well, so no, you will have to build a specific architecture to allow for hierarchical planning. So hierarchical planning is absolutely necessary if you want to plan complex actions. If I want to go from, let's say, from New York to Paris, it's the example I use all the time, and I'm sitting in my office at NYU, my objective that I need to minimize is my distance to Paris at a high level.
a very abstract representation of my location, I would have to decompose this into two sub-goals. First one is go to the airport. Second one is catch a plane to Paris. Okay, so my sub-goal is now going to the airport. My objective function is my distance to the airport. How do I go to the airport? Well, I have to go in the street and hail a taxi, which you can do in New York.
Okay, now I have another sub-goal. Go down on the street. What that means, going to the elevator, going down the elevator, walk out the street. How do I go to the elevator? I have to... Stand up from my chair, open the door of my office, go to the elevator, push the button. How do I get up from my chair?
Like, you know, you can imagine going down, all the way down to basically what amounts to millisecond by millisecond muscle control. Okay. And obviously you're not going to plan your entire trip from New York to Paris in terms of millisecond by millisecond muscle control. First, that would be incredibly expensive.
But it will also be completely impossible because you don't know all the conditions of what's going to happen. How long it's going to take to catch a taxi or to go to the airport with traffic. You would have to know exactly the condition of everything to be able to do this planning. And you don't have the information.
So you have to do this hierarchical planning so that you can start acting and then sort of replanning as you go. And nobody really knows how to do this in AI. Nobody knows how to train a system to learn the appropriate multiple levels of representation so that hierarchical planning works.
Right, so there's a lot of questions that are actually implied by this, right? So the first thing is, LLMs will be able to answer some of those questions down to some level of abstraction. under the condition that they've been trained with similar scenarios in their training set.
Yeah, true. I mean, they will probably produce some answer, except they're not going to be able to really kind of produce millisecond by millisecond muscle control of how you stand up from your chair, right? But down to some level of abstraction where you can describe things by words,
They might be able to give you a plan, but only under the condition that they've been trained to produce those kind of plans, right? They're not going to be able to plan for situations that they never encountered before. They basically are going to have to regurgitate the template that they've been trained on.
Certainly, LLM would be able to solve that problem if you fine-tune it for it. I can't say that LLM cannot do this. It can do this if you train it for it. There's no question. Down to a certain level, where things can be formulated in terms of words. But if you want to go down to how you climb down the stairs or just stand up from your chair in terms of words, you can't do it.
That's one of the reasons you need experience of the physical world, which is much higher bandwidth than what you can express in words. In human language.
Sure. And, you know, a lot of plans that people know about that are relatively high level are actually learned. Most people don't invent the, you know, plans. They... We have some ability to do this, of course, obviously, but most plans that people use are plans that they've been trained on. They've seen other people use those plans or they've been told how to do things.
You can't invent how you take a person who's never heard of airplanes and tell them, like, how do you go from New York to Paris and... They're probably not going to be able to deconstruct the whole plan unless they've seen examples of that before. So certainly LLMs are going to be able to do this.
But then how you link this from the low level of actions, that needs to be done with things like JEPA that basically lifts the abstraction level of the representation without attempting to reconstruct every detail of the situation. That's why we need JEPAs for it.
No, there's one thing that autoregressive LLMs, or that LLMs in general, not just the autoregressive one, but including the BERT-style bidirectional ones, are exploiting, and it's self-supervised learning. And I've been a very, very strong advocate of self-supervised learning for many years. So those things are an incredibly impressive demonstration that self-supervised learning actually works.
It didn't start with Bert, but it was really a good demonstration with this. The idea that you take a piece of text, you corrupt it, and then you train some gigantic neural net to reconstruct the parts that are missing, that has produced an enormous amount of benefits. It allowed us to create systems that understand language, systems that can translate hundreds of languages in any direction.
systems that are multilingual, so it's a single system that can be trained to understand hundreds of languages and translate in any direction, and produce summaries, and then answer questions and produce text.
And then there's a special case of it, which is the autoregressive trick, where you constrain the system to not elaborate a representation of the text from looking at the entire text, but only predicting a word from the words that come before. And you do this by constraining the architecture of the network. And that's what you can build an autoregressive LLM from.
Showing 121–140 of 384 · page 7 of 20
← Previous
Next →