Yann LeCun
speaker
384 appearances
5 recordings
4 series
first heard Mar 2024
last heard 20 Jun
Yann LeCun’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 4 in all, peaking in Jan 2026 with 2.
Appearances
Like these are pretty big.
When WorldLabs launched, Lee raised $230 million at a $1 billion valuation.
for a number of reasons. The first is that there is a number of characteristics of intelligent behavior. For example, the capacity to understand the world, understand the physical world, the ability to remember and retrieve things, persistent memory, the ability to reason, and the ability to plan. Those are four essential characteristics of intelligent systems or entities, humans, animals.
LLMs can do none of those, or they can only do them in a very primitive way. They don't really understand the physical world. They don't really have persistent memory. They can't really reason, and they certainly can't plan. If you expect the system to become intelligent just without having the possibility of doing those things, you're making a mistake.
That is not to say that autoregressive LLMs are not useful. They're certainly useful. That they're not interesting, that we can't build a whole ecosystem of applications around them. Of course we can, but as it paths towards human-level intelligence, they're missing essential components. And then there is another tidbit or fact that I think is very interesting.
Those LLMs are trained on enormous amounts of text, basically the entirety of all publicly available texts on the internet, right? That's typically on the order of 10 to the 13 tokens. Each token is typically two bytes. So that's two 10 to the 13 bytes as training data. It would take you or me 170,000 years to just read through this at eight hours a day.
So it seems like an enormous amount of knowledge that those systems can accumulate. But then you realize it's really not that much data. If you talk to developmental psychologists and they tell you a four-year-old has been awake for 16,000 hours in his or her life, and the amount of information that has reached the visual cortex of that child in four years... is about 10 to the 15 bytes.
And you can compute this by estimating that the optical nerve carry about 20 megabytes per second, roughly. And so 10 to the 15 bytes for a four-year-old versus two times 10 to the 13 bytes for 170,000 years worth of reading What that tells you is that through sensory input, we see a lot more information than we do through language.
And that despite our intuition, most of what we learn and most of our knowledge is through our observation and interaction with the real world, not through language. Everything that we learn in the first few years of life and certainly everything that animals learn has nothing to do with language.
So it's a big debate among philosophers and also cognitive scientists, like whether intelligence needs to be grounded in reality. I'm clearly in the camp that, yes, intelligence cannot appear without some grounding in some reality. It doesn't need to be Physical reality could be simulated, but the environment is just much richer than what you can express in language.
Language is a very approximate representation of our percepts and our mental models. There's a lot of tasks that we accomplish where we manipulate a mental model of the situation at hand, and that has nothing to do with language. Everything that's physical, mechanical, whatever, when we build something, when we accomplish a task, a model task of grabbing something, etc.,
We plan our action sequences, and we do this by essentially imagining the result of the outcome of a sequence of actions that we might imagine. And that requires mental models that don't have much to do with language. And that's, I would argue, most of our knowledge is derived from that interaction with the physical world.
So a lot of my colleagues who are more interested in things like computer vision are really on that camp that AI needs to be embodied, essentially. And then other people coming from the NLP side or maybe some other motivation don't necessarily agree with that. And philosophers are split as well. And the complexity of the world is hard to imagine. It's hard to
represent all the complexities that we take completely for granted in the real world that we don't even imagine require intelligence, right? This is the old Moravec paradox from the pioneer of robotics, Hans Moravec, who said, you know, how is it that with computers it seems to be easy to do high-level complex tasks like playing chess and solving integrals and doing things like that, whereas
The thing we take for granted that we do every day, like, I don't know, learning to drive a car or, you know, grabbing an object. We can't do it with computers. And, you know, we have LLMs that can pass the bar exam. So they must be smart. But then they can't learn to drive in 20 hours like any 17-year-old.
They can't learn to clear out the dinner table and fill up the dishwasher like any 10-year-old can learn in one shot. Why is that? Like, you know, what are we missing? What type of learning or reasoning architecture or whatever are we missing that basically prevent us from, you know, having level five self-driving cars and domestic robots?
So yeah, that's what a lot of people are working on. So the short answer is no. And the more complex answer is you can use all kinds of tricks to get an LLM to basically digest visual representations of images Or video, or audio for that matter. And a classical way of doing this is you train a vision system in some way.
And we have a number of ways to train vision systems, either supervised, semi-supervised, self-supervised, all kinds of different ways. That will turn any image into a high-level representation. basically a list of tokens that are really similar to the kind of tokens that typical LLM takes as an input. And then you just feed that to the LLM in addition to the text.
And you just expect the LLM to kind of, during training, to kind of be able to use those representations to help make decisions. I mean, there's been work along those lines for quite a long time. And now you see those systems, right?
I mean, there are LLMs that have some vision extension, but they're basically hacks in the sense that those things are not like trained end-to-end to handle, to really understand the world. They're not trained with video, for example. They don't really understand intuitive physics, at least not at the moment.
Showing 21–40 of 384 · page 2 of 20
← Previous
Next →