Francois Chollet

So maybe we won't be releasing it publicly, but what we could do is just create a test server

where you can query, get a task, you submit a solution, and of course, you can use whatever frontier model you want there.

So that way, because you actually have to query this API, you're making sure that no one is going to, by accident, train on this data.

It's unlike the current public architecture, which is literally on GitHub.

So there's no question about whether the models are actually trained on it.

Yes, they are, because they're trained on GitHub.

So by sort of like gating access to acquiring this API, we would avoid this issue.

And then we would see, you know, for people who actually want to try whatever technique they have in mind using whatever resources they want, that would be a way for them to get an answer.

Yeah, that's the point of having a money prize is that you attract more people, you get them to try to solve it.

And if there's an easy way to hack the benchmark that reveals that the benchmark is flawed, then you're going to know about it.

In fact, that was the point of the original

cable competition back in 2020 for Arc.

I was running this competition because I had released this dataset and I wanted to know if it was hackable, if you could cheat.

So there was a small money prize at the time, that was like 20K, and this was right around the same time as GPT-3 was released.

So people of course tried GPT-3 on the public data, it scored zero.

But I think what the first contest taught us is that there is no obvious shortcut, right?

And well, now there's more money, there's going to be more people looking into it.

Well, we're going to find out.

We're going to see if the benchmark is going to survive.

And you know, if we end up with a solution that is not like trying to brute force the space of possible arc tasks that's just trained on core knowledge, I don't think it's necessarily going to be in and by itself AGI, but it's probably going to be a huge milestone on the way to AGI.