Andy Halliday

pushes it back to $1 per task.

Yeah.

And yet goes upwards of, you know, 75, 78% somewhere in there.

So very impressive performance on Arcade GI2 by Gemini 3.1 Pro, which just came out yesterday.

Yeah.

So is it that using a single model, you find that there is a lack of reliability because of inconsistency in the reproducing of results?

Or is it that you're talking about inconsistency across models, which is, oh, this model doesn't reliably produce the things that I need

as compared to apparently the way it reliably produces solutions to the difficult tasks on the Arc AGI prizes.

You'd probably pay more attention then to the x-axis on that RKGI display, which is, hey, did it have to repeat that process, burning tokens repeatedly in order to get to the right solution?

Or

as in the new case of Gemini 3.1 Pro, that it only costs $1 worth of tokens to do a task on AGI 2.

One thing I wanted to point out is that on that prior graphic that I showed from artificial analysis, and artificial analysis is a company, right?

They do these tests against all of these AI models.

Artificial analysis on that scale

score for agentic skills you saw that Claude Opus 4.6 max and Claude Opus regular were at 64 and 68 while you know Gemini 3.1 pro is only at 59 now what but between 59 and the 64 for Claude Opus 4.6 and 68 if you go to max from anthropic

There's a model there that I don't know that I highlighted, which is GLM5.

It's a Z.AI open source model, and it scores 63, the same as Sonnet 4.6.

So SOMET 4.6 and GLM-5 exceed GPT 5.2 extra high.

Yeah.

Right?