Yann LeCun
speaker
384 appearances
5 recordings
4 series
first heard Mar 2024
last heard 20 Jun
Yann LeCun’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 4 in all, peaking in Jan 2026 with 2.
Appearances
And vcrag, I should say, is not a distillation method, actually, but ijpa and BYOL certainly are. And there's another one also called dino, also produced at FAIR. And the idea of those things is that you take the full input, let's say an image, you run it through an encoder, produces a representation.
And then you corrupt that input or transform it, running through essentially what amounts to the same encoder, with some minor differences. And then train a predictor, sometimes the predictor is very simple, sometimes it doesn't exist, but train a predictor to predict a representation of the first uncorrupted input from the corrupted input. But you only train the second branch.
You only train the part of the network that is fed with the corrupted input. The other network you don't train, but since they share the same weight, when you modify the first one, it also modifies the second one. And with various tricks, you can prevent the system from collapsing, with the collapse of the type I was explaining before, where the system basically ignores the input.
So that works very well. The two techniques we've developed at FAIR, Deno and IGEPA, work really well for that.
So there's several scenarios. One scenario is you take an image, you corrupt it by changing the cropping, for example, changing the size a little bit, maybe changing the orientation, blurring it, changing the colors. doing all kinds of horrible things to it.
Basic horrible things that sort of degrade the quality a little bit and change the framing, you know, crop the image. And in some cases, in the case of iJet, you don't need to do any of this. You just mask some parts of it, right? You just basically remove some regions, like a big block, essentially. Yeah.
And then run through the encoders and train the entire system, encoder and predictor, to predict the representation of the good one from the representation of the corrupted one. So that's the IGEPA. It doesn't need to know that it's an image, for example, because the only thing it needs to know is how to do this masking.
Whereas with Deno, you need to know it's an image because you need to do things like geometry transformation and blurring and things like that that are really image-specific. A more recent version of this that we have is called VJPA. So it's basically the same idea as iJPA, except it's applied to video. So now you take a whole video and you mask a whole chunk of it.
And what we mask is actually kind of a temporal tube. So like a whole segment of each frame in the video over the entire video. Mm-hmm.
Throughout the tube, yeah. Typically it's 16 frames or something, and we masked the same region over the entire 16 frames. It's a different one for every video, obviously. And then again, train that system so as to predict the representation of the full video from the partially masked video. That works really well.
It's the first system that we have that learns good representations of video so that when you feed those representations to a supervised classifier head, it can tell you what action is taking place in the video with pretty good accuracy. So that's the first time we get something of that quality.
Yeah. We have also preliminary results that seem to indicate that the representation allows our system to tell whether the video is physically possible or completely impossible because some object disappeared or an object suddenly jumped from one location to another or changed shape or something.
Possibly. This is going to take a while before we get to that point, but there are robotic systems that are based on this idea. And what you need for this is a slightly modified version of this, where imagine that you have a complete video.
And what you're doing to this video is that you're either translating it in time towards the future, so you only see the beginning of the video, but you don't see the latter part of it that is in the original one. Or you just mask the second half of the video, for example. And then you train a JEPA system of the type I described to predict the representation of the full video from the shifted one.
But you also feed the predictor with an action. For example, the wheel is turned 10 degrees to the right or something. So if it's a dash cam in a car and you know the angle of the wheel, you should be able to predict to some extent what's going to happen to what you see.
You're not going to be able to predict all the details of objects that appear in the view, obviously, but at an abstract representation level, you can probably predict what's going to happen. So now what you have is...
an internal model that says, here is my idea of state of the world at time t, here is an action I'm taking, here is a prediction of the state of the world at time t plus one, t plus delta t, t plus two seconds, whatever it is. If you have a model of this type, you can use it for planning.
So now you can do what LLMs cannot do, which is planning what you're going to do so as to arrive at a particular outcome or satisfy a particular objective. So you can have a number of objectives. I can predict that if I have an object like this and I open my hand, it's going to fall. And if I push it with a particular force on the table, it's going to move.
If I push the table itself, it's probably not going to move with the same force. So we have this internal model of the world in our mind, which allows us to plan sequences of actions to arrive at a particular goal. And so now if you have this world model, we can imagine a sequence of actions, predict what the outcome of the sequence of action is going to be,
measure to what extent the final state satisfies a particular objective, like moving the bottle to the left of the table, and then plan a sequence of actions that will minimize this objective at runtime. We're not talking about learning, we're talking about inference time. So this is planning, really. And in optimal control, this is a very classical thing. It's called model predictive control.
Showing 101–120 of 384 · page 6 of 20
← Previous
Next →