Francois Chollet

I'm not saying it's not going to happen.

Arc is not a perfect benchmark.

Maybe it has flaws.

Maybe it could be hacked in that way.

What would make me change my mind about that, Alain, is basically if I start seeing a critical mass of cases where you show the model with something it has not seen before, a task that's actually novel from the perspective of its training data, something that's not in training data, and if it can actually adapt on the fly,

And this is true for other lamps, but really this would catch my attention for any AI technique out there.

If I can see the ability to adapt to novelty on the fly, to pick up new skills efficiently, then I would be extremely interested.

I would think this is on the path to AGI.

Right.

You're asking basically what's the difference between actual intelligence, which is the ability to adapt to things you've not been prepared for, and pure memorization, like reciting what you've seen before.

And it's not just some semantic difference.

The big difference is that you can never...

pre-trained on everything that you might see at test time, right?

Because the world changes all the time.

So it's not just the fact that the space of possible tasks is infinite.

And even if you're trained on millions of them, you've only seen zero percent of the total space.

It's also the fact that the world is changing every day, right?

This is why we, the human species, has developed intelligence in the first place.

If there was such a thing as a distribution for the world, for the universe, for our lives, then we would not need intelligence at all.

In fact, many creatures, many insects, for instance, do not have intelligence.