Francois Chollet

That's an empirical question, so I guess we're going to see the answer within a few months.

But my answer to that is, you know, arc grids, they're just discrete 2D grids of symbols.

They're pretty small.

It's not like if you flatten an image as a sequence of pixels, for instance, then you get something that's actually very, very difficult to parse.

But that's not true for arc because the grids are very small.

You only have 10 possible symbols.

So there's these 2D grids that are actually very easy to flatten.

as sequences.

And transformers, LLMs, they're very good at processing the sequences.

In fact, you can show that LLMs do fine with processing arc-like data by simply fine-tuning an LLM on some subsets of the tasks and then trying to test it on small variations of these tasks.

And you see that, yeah, the LLM can encode just fine solution programs for tasks that it has seen before.

So it does not really have a problem passing the input or figuring out the program.

The reason why LLMs don't do well on Arc is really just

The unfamiliarity aspect, the fact that each new task is different from every other task.

Basically, you cannot memorize the solution programs in advance.

You have to synthesize a new solution program on the fly for each new task.

And that's really what LLMs are struggling with.

Like any smart human should be able to do 90%, 95% on Arc.

A smart human.

A smart human.