Francois Chollet

This is what makes

This is the definition of generality.

Like generality is not specificity scaled up.

It is the ability to apply your mind to anything at all, to arbitrary things.

And this requires, fundamentally, this requires the ability to adapt, to learn on the fly efficiently.

If you're right, if you were right, LLMs would do really well on arc puzzles because arc puzzles are not complex.

Each one of them requires very little knowledge.

Each one of them is very low on complexity.

You don't need to think very hard about it.

They're actually extremely obvious for humans, like even children can do them.

But LLMs cannot.

Even LLMs that have, you know, 100,000 times more knowledge than you do, they still cannot.

And the only thing that makes ARC special is that it was designed with this intent to resist memorization.

This is the only thing.

And this is the huge blocker for LLM performance, right?

And so, you know, I think if you look at LLMs closely, it's pretty obvious that they're not really like synthesizing new programs on the fly to solve the tasks that they're faced with.

They're very much replying things that they've stored in memory.

For instance, one thing that's very striking is LLMs can solve a Caesar cipher, right?

You know, like a CISAS, I feel like, transposing letters to code a message.

And, well, there's a very complex algorithm, right?