Sergey Levine

I don't think that that by itself is like a silver bullet.

I don't think it solves everything.

But I think that it does help a lot.

And I think that we've already seen the beginnings of that where we can see that including web data in training for robots really does help with generalization.

And I actually have the suspicion that in the long run, it'll make it easier to use those sources of data that have been tricky to use up until now.

Yeah.

So there's a subtlety here.

Emerging capabilities don't just come from the fact that internet data has a lot of stuff in it.

They also come from the fact that generalization, once it reaches a certain level, becomes compositional.

There was a cute example that one of my students really liked to use in some of his presentations, which is –

You know what International Phonetic Alphabet is?

No.

So if you look in a dictionary, they'll have the pronunciation of a word written in kind of funny letters.

That's basically International Phonetic Alphabet.

So it's an alphabet that is pretty much exclusively used for writing down pronunciations of individual words in dictionaries.

And you can ask an LLM to write you a recipe for making some meal in International Phonetic Alphabet, and it will do it.

And that's like, holy crap.

That is definitely not something that it has ever seen because—

IPAs only ever used for writing down pronunciations of individual words.

So that's compositional generalization.