Francois Chollet

And so far, LMs have not been doing very well on it.

In fact, the approaches that are working well are more towards discrete program search, program synthesis.

Right.

I'm pretty skeptical that we're going to see LLM do 80% in a year.

That said, if we do see it, you would also have to look at how this was achieved.

If you just train the model and millions or billions of puzzles similar to Arc, so that you're relying on the ability to have some overlap between the tasks that you train on and the tasks that you're going to see at test time, then you're still using memorization, right?

And maybe it can work, you know, hopefully

Arc is going to be good enough that it's going to be resistant to this sort of attempt at brute forcing.

But you never know.

Maybe it could happen.

I'm not saying it's not going to happen.

Arc is not a perfect benchmark.

Maybe it has flaws.

Maybe it could be hacked in that way.

What would make me change my mind about that, Alain, is basically if I start seeing a critical mass of cases where you show the model with something it has not seen before, a task that's actually novel from the perspective of its training data, something that's not in training data, and if it can actually adapt on the fly,

And this is true for other lamps, but really this would catch my attention for any AI technique out there.

If I can see the ability to adapt to novelty on the fly, to pick up new skills efficiently, then I would be extremely interested.

I would think this is on the path to AGI.

Right.

You're asking basically what's the difference between actual intelligence, which is the ability to adapt to things you've not been prepared for, and pure memorization, like reciting what you've seen before.