Bowen Baker

And so we,

reasoning models, generally their capabilities, like on math problems and coding problems and things like that, they get better as they think for longer.

They'll be able to solve more complex tasks.

And so a small model

thinking for a very long time can oftentimes get to the same answer or match the capability of a bigger model thinking for a shorter time okay wow so that's so yeah this model is really tight there's like a couple more steps in the thing i can just go through it and you can yeah um sure yeah so yeah so you now have like these like capability matched points of you know small model uh

thinking for longer, big model, thinking for shorter, are roughly the same capabilities.

But we found that, again, going back to your previous thing, we found that models that think for longer are more monitorable.

They reveal more in their thinking.

We found that the model thinking for the smaller model at the same capability level was more monitorable because it was thinking for longer.

And so then that's the monitorability tax.

Or sorry, but...

Sorry, then the tax aspect comes in where these small models thinking for longer actually were using at that for like matched capability levels used more inference time compute than the bigger model thinking for shorter.

In this case, the smaller models I think basically always cost more because the relative decrease in inference compute per token generated was less than the amount of additional tokens you'd need to reach the same capability level.

And so overall, that small model with a lot of tokens was costing more compute than the big model.

And so that's kind of like the monitorability tax.

Like, oh, you could just spend more compute on inference time, but like get a more monitorable model out of it.

Yeah, well, I guess we hope so.

I mean, we hope these evals are the one or maybe the way to do it.

So we have a few different ways of kind of like tracking monitorability.

We like divided them out into three archetypes.