Sergey Levine

Like, LLMs essentially do a kind of metal learning via in-context learning.

I mean, we can debate as to how much that's learning or not, but the point is that large, powerful models trained on the right objective on real data get much better at leveraging all the other stuff.

And I think that's actually the key.

And coming back to your airplane pilot, like, the airplane pilot is trained on a real-world objective.

Like, their objective is to be a good airplane pilot, to be successful, to have a good career.

And all of that kind of propagates back into the actions they take in leveraging all these other data sources.

So what I think is actually the key here to leveraging auxiliary data sources, including simulation, is to build the right foundation model that is really good, that has those immersion abilities.

And to your point...

To get really good like that, it has to have the right objective.

Now, we know how to get the right objective out of real-world data.

Maybe we can get it out of other things, but that's harder right now.

And I think that, again, we can look to the examples of what happened in other fields.

Like these days, if someone trains an LLM for solving complex problems, they're using lots of synthetic data.

But the reason they're able to leverage that synthetic data effectively is because they have this starting point that is trained on lots of real data that kind of gets it.

And once it gets it, then it's more able to leverage all this other stuff.

So I think perhaps ironically, the key to leveraging other data sources, including simulation, is to get really good at using real data, understand what's up with the world, and then now you can fruitfully use all this other stuff.

So here's what I would say, that deep down at a very fundamental level,

The synthetic experience that you create yourself doesn't allow you to learn more about the world.

It allows you to rehearse things.

It allows you to consider counterfactuals.