Logan Kilpatrick

And you look at the models and the models can't follow the basic instructions of chess.

The models want to make all these illegal moves.

They want to do all this stuff that just isn't

how to actually play the game and is a good reminder that like, we're obviously making progress, but we're not, we're not at AGI yet.

Um, right.

And there's this like jagged intelligence paradigm where like the models are really can generate the code for Minecraft, uh, or some, something that looks or feels like Minecraft.

And yet at the same time, can't follow the basic instructions of chess, which I could probably teach a, you know, a 12 year old who's

Made it through some years of school how to play chess and probably 30 minutes and they'd be able to pick it up.

Yeah.

So it is it is fascinating to see that that model behavior play out in that way.

Yeah, this is definitely part of my like joke is, you know, everything comes back to instruction following.

So if you can just make the model better at instruction following, you can do anything.

And like chess is a good example of that.

Like technically, if you were to just make the model good at instruction following, you should be able to give it really, really good instructions.

And it should be able to go in and generalize from that.

The reality is like it's not that easy.

it's not that clear.

And to answer the question specifically, like it's an open research question, like how much does making the models better at these games actually generalize to other capabilities?

Hopefully it does.

And you could imagine as part of the original, like deep mind thesis is you teach the models, a bunch of stuff about all these different environments and the