Yann LeCun

Actually, that's going back more than 10 years, but the idea of self-supervised learning, so basically capturing the internal structure of a set of inputs without training the system for any particular task, learning representations. You know, the conference I co-founded 14 years ago is called International Conference on Learning Representations.

That's the entire issue that deep learning is dealing with, right? And it's been my obsession for almost 40 years now. So learning representation is really the thing. For the longest time, you could only do this with supervised learning.

That's the entire issue that deep learning is dealing with, right? And it's been my obsession for almost 40 years now. So learning representation is really the thing. For the longest time, you could only do this with supervised learning.

That's the entire issue that deep learning is dealing with, right? And it's been my obsession for almost 40 years now. So learning representation is really the thing. For the longest time, you could only do this with supervised learning.

And then we started working on what we used to call unsupervised learning and sort of revived the idea of unsupervised learning in the early 2000s with Yoshua Bengio and Jeff Hinton. Then discovered that supervised learning actually works pretty well if you can collect enough data. And so the whole idea of unsupervised self-supervised learning kind of took a backseat for a bit.

And then we started working on what we used to call unsupervised learning and sort of revived the idea of unsupervised learning in the early 2000s with Yoshua Bengio and Jeff Hinton. Then discovered that supervised learning actually works pretty well if you can collect enough data. And so the whole idea of unsupervised self-supervised learning kind of took a backseat for a bit.

And then we started working on what we used to call unsupervised learning and sort of revived the idea of unsupervised learning in the early 2000s with Yoshua Bengio and Jeff Hinton. Then discovered that supervised learning actually works pretty well if you can collect enough data. And so the whole idea of unsupervised self-supervised learning kind of took a backseat for a bit.

And then I kind of tried to revive it in a big way, starting in 2014, basically when we started FAIR. and really pushing for finding new methods to do self-supervised learning, both for text and for images and for video and audio. And some of that work has been incredibly successful.

And then I kind of tried to revive it in a big way, starting in 2014, basically when we started FAIR. and really pushing for finding new methods to do self-supervised learning, both for text and for images and for video and audio. And some of that work has been incredibly successful.

And then I kind of tried to revive it in a big way, starting in 2014, basically when we started FAIR. and really pushing for finding new methods to do self-supervised learning, both for text and for images and for video and audio. And some of that work has been incredibly successful.

I mean, the reason why we have a multilingual translation system, you know, things to do content moderation on Meta, for example, on Facebook, that are multilingual, that understand whether a piece of text is hate speech or not or something. is due to that progress using self-supervised learning for NLP, combining this with transformer architectures and blah, blah, blah.

I mean, the reason why we have a multilingual translation system, you know, things to do content moderation on Meta, for example, on Facebook, that are multilingual, that understand whether a piece of text is hate speech or not or something. is due to that progress using self-supervised learning for NLP, combining this with transformer architectures and blah, blah, blah.

I mean, the reason why we have a multilingual translation system, you know, things to do content moderation on Meta, for example, on Facebook, that are multilingual, that understand whether a piece of text is hate speech or not or something. is due to that progress using self-supervised learning for NLP, combining this with transformer architectures and blah, blah, blah.

But that's the big success of self-supervised learning. We had similar success in speech recognition, a system called Wave2Vec, which is also a joint embedding architecture, by the way, trained with contrastive learning. And that system also can produce speech recognition systems that are multilingual,

But that's the big success of self-supervised learning. We had similar success in speech recognition, a system called Wave2Vec, which is also a joint embedding architecture, by the way, trained with contrastive learning. And that system also can produce speech recognition systems that are multilingual,

But that's the big success of self-supervised learning. We had similar success in speech recognition, a system called Wave2Vec, which is also a joint embedding architecture, by the way, trained with contrastive learning. And that system also can produce speech recognition systems that are multilingual,

with mostly unlabeled data and only need a few minutes of labeled data to actually do speech recognition. That's amazing. We have systems now based on those combination of ideas that can do real-time translation of hundreds of languages into each other.

with mostly unlabeled data and only need a few minutes of labeled data to actually do speech recognition. That's amazing. We have systems now based on those combination of ideas that can do real-time translation of hundreds of languages into each other.

with mostly unlabeled data and only need a few minutes of labeled data to actually do speech recognition. That's amazing. We have systems now based on those combination of ideas that can do real-time translation of hundreds of languages into each other.

That's right. We don't go through text. It goes directly from speech-to-speech using an internal representation of kind of speech units that are discrete. But it's called textless NLP. We used to call it this way. But yeah, so that, I mean, incredible success there. And then, you know, for 10 years, we tried to apply this idea