Yann LeCun

They can't learn to clear out the dinner table and fill up the dishwasher like any 10-year-old can learn in one shot. Why is that? Like, you know, what are we missing? What type of learning or reasoning architecture or whatever are we missing that basically prevent us from, you know, having level five self-driving cars and domestic robots?

They can't learn to clear out the dinner table and fill up the dishwasher like any 10-year-old can learn in one shot. Why is that? Like, you know, what are we missing? What type of learning or reasoning architecture or whatever are we missing that basically prevent us from, you know, having level five self-driving cars and domestic robots?

So yeah, that's what a lot of people are working on. So the short answer is no. And the more complex answer is you can use all kinds of tricks to get an LLM to basically digest visual representations of images Or video, or audio for that matter. And a classical way of doing this is you train a vision system in some way.

So yeah, that's what a lot of people are working on. So the short answer is no. And the more complex answer is you can use all kinds of tricks to get an LLM to basically digest visual representations of images Or video, or audio for that matter. And a classical way of doing this is you train a vision system in some way.

So yeah, that's what a lot of people are working on. So the short answer is no. And the more complex answer is you can use all kinds of tricks to get an LLM to basically digest visual representations of images Or video, or audio for that matter. And a classical way of doing this is you train a vision system in some way.

And we have a number of ways to train vision systems, either supervised, semi-supervised, self-supervised, all kinds of different ways. That will turn any image into a high-level representation. basically a list of tokens that are really similar to the kind of tokens that typical LLM takes as an input. And then you just feed that to the LLM in addition to the text.

And we have a number of ways to train vision systems, either supervised, semi-supervised, self-supervised, all kinds of different ways. That will turn any image into a high-level representation. basically a list of tokens that are really similar to the kind of tokens that typical LLM takes as an input. And then you just feed that to the LLM in addition to the text.

And we have a number of ways to train vision systems, either supervised, semi-supervised, self-supervised, all kinds of different ways. That will turn any image into a high-level representation. basically a list of tokens that are really similar to the kind of tokens that typical LLM takes as an input. And then you just feed that to the LLM in addition to the text.

And you just expect the LLM to kind of, during training, to kind of be able to use those representations to help make decisions. I mean, there's been work along those lines for quite a long time. And now you see those systems, right?

And you just expect the LLM to kind of, during training, to kind of be able to use those representations to help make decisions. I mean, there's been work along those lines for quite a long time. And now you see those systems, right?

And you just expect the LLM to kind of, during training, to kind of be able to use those representations to help make decisions. I mean, there's been work along those lines for quite a long time. And now you see those systems, right?

I mean, there are LLMs that have some vision extension, but they're basically hacks in the sense that those things are not like trained end-to-end to handle, to really understand the world. They're not trained with video, for example. They don't really understand intuitive physics, at least not at the moment.

I mean, there are LLMs that have some vision extension, but they're basically hacks in the sense that those things are not like trained end-to-end to handle, to really understand the world. They're not trained with video, for example. They don't really understand intuitive physics, at least not at the moment.

I mean, there are LLMs that have some vision extension, but they're basically hacks in the sense that those things are not like trained end-to-end to handle, to really understand the world. They're not trained with video, for example. They don't really understand intuitive physics, at least not at the moment.

We're not going to be able to do this with the type of LLMs that we are working with today. And there's a number of reasons for this. But the main reason is The way LLMs are trained is that you take a piece of text, you remove some of the words in that text, you mask them, you replace them by blank markers, and you train a genetic neural net to predict the words that are missing.

We're not going to be able to do this with the type of LLMs that we are working with today. And there's a number of reasons for this. But the main reason is The way LLMs are trained is that you take a piece of text, you remove some of the words in that text, you mask them, you replace them by blank markers, and you train a genetic neural net to predict the words that are missing.

We're not going to be able to do this with the type of LLMs that we are working with today. And there's a number of reasons for this. But the main reason is The way LLMs are trained is that you take a piece of text, you remove some of the words in that text, you mask them, you replace them by blank markers, and you train a genetic neural net to predict the words that are missing.

And if you build this neural net in a particular way so that it can only look at words that are to the left of the one it's trying to predict, then what you have is a system that basically is trying to predict the next word in a text, right? So then you can feed it a text, a prompt, and you can ask it to predict the next word. It can never predict the next word exactly.

And if you build this neural net in a particular way so that it can only look at words that are to the left of the one it's trying to predict, then what you have is a system that basically is trying to predict the next word in a text, right? So then you can feed it a text, a prompt, and you can ask it to predict the next word. It can never predict the next word exactly.

And if you build this neural net in a particular way so that it can only look at words that are to the left of the one it's trying to predict, then what you have is a system that basically is trying to predict the next word in a text, right? So then you can feed it a text, a prompt, and you can ask it to predict the next word. It can never predict the next word exactly.