Francois Chollet

You can already try to prompt one of the best models, like the latest Gemini, the latest GPT-4, with tasks from the public evaluation set.

And again, the problem is that these tasks are available as JSON files on GitHub.

These models are also trained on GitHub.

So they're actually trained on these tasks.

And yeah, that kind of creates uncertainty about if they can actually solve some of the tasks, is that because they memorized the answer or not?

You know, maybe you would be better off trying to create your own private arc-like, very novel test set.

Don't make the task difficult, don't make them complex, make them very obvious for humans, but make sure to make them original as much as possible, make them unique, different.

and see how much your GPT-4 and so on, or GPT-5 does on them.

Yeah, no, Arc is not a perfect benchmark.

I mean, I made it like four years ago, over four years ago, almost five now.

This was in a time before LLAMS.

And I think we learned a lot, actually, since about what potential flaws there might be.

I think there is some redundancy.

in the set of tasks, which is, of course, against the goals of the benchmark.

Every task is supposed to be unique in practice.

That's not quite true.

I think there's also every task is supposed to be very novel, but in practice, they might not be.

They might be structurally similar to something that you might find online somewhere.

So we want to keep iterating and release an Arc 2 version later this year.

And I think when we do that, we're going to want to make the old private test set available.