Francois Chollet

I want to rabbit hole on that a little bit.

But before I do that,

Sure.

So one arc puzzle, it looks kind of like an IQ test puzzle.

You've got a number of demonstration input-output pairs.

So one pair is made of two grids.

So one grid shows you an input and the second grid shows you what you should produce as a response to that input.

And you get a couple pairs like this to demonstrate the nature of the task, to demonstrate what you're supposed to do with your inputs.

and then you get a new test input and your job is to produce the corresponding test output.

You look at the demonstration pairs and from that you figure out what you're supposed to do and you show that you've understood it on this new test pair.

And importantly, in order to, the sort of like knowledge basis that you need in order to approach these challenges is you just need core knowledge.

And core knowledge is, it's basically the knowledge of what makes an object, basic counting, basic geometry, topology, symmetries, that sort of thing.

So extremely basic knowledge.

LLMs for sure possess such knowledge.

Any child possesses such knowledge.

And what's really interesting is that each puzzle is new.

So it's not something that you're going to find elsewhere on the internet, for instance.

And that means that whether it's as a human or as a machine, every puzzle, you have to approach it from scratch.

You have to actually reason your way through it.

You cannot just fetch the response from your memory.