Yoshua Bengio

And there are other issues, of course, with the old AI, like not really good ways of handling uncertainty.

I would say something more subtle, which we understand better now, but I think still isn't enough in the minds of people.

There's something really powerful that comes from distributed representations, the thing that really makes neural nets work so well.

And it's hard to replicate that kind of power in a symbolic world.

The knowledge in expert systems and so on is nicely decomposed into like a bunch of rules.

Whereas if you think about a neural net, it's the opposite.

You have this big blob of parameters which work intensely together to represent everything the network knows.

And it's not sufficiently factorized.

And so I think this is one of the weaknesses of current neural nets.

that we have to take lessons from classical AI in order to bring in another kind of compositionality, which is common in language, for example, and in these rules, but that isn't so native to neural nets.

So let me connect with disentangled representations, if you might, if you don't mind.

So for many years, I've thought, and I still believe, that it's really important that we come up with learning algorithms, either unsupervised or supervised, or reinforcement, whatever, that build representations in which the important factors, hopefully causal factors, are nicely separated and easy to pick up from the representation.

Right.

So that's the idea of disentangled representations.

It says transform the data into a space where everything becomes easy.

We can maybe just learn with linear models about the things we care about.

And I still think this is important, but I think this is missing out on a very important ingredient, which classical AI systems can remind us of.

So let's say we have these disentangle representations.

You still need to learn about the relationships between the variables, those high-level semantic variables.

They're not going to be independent.