Bowen Baker

That's like this kind of, it kind of turns into a needle in a haystack problem almost.

You have to like find this one bad thing amongst like this, all this stuff that language model, that the language model is doing for you.

Um, and in addition, you know, they're kind of getting more and more widely deployed.

And I think in general, people are like less and less actually going and checking every little thing the model is doing for you.

And so, yeah, so there's kind of, so even if they could catch that needle in a haystack, they might not even be like, like looking for it because, you know, we're getting lazy and no, that's getting lazy and learning to trust it more.

Yeah, so chains of thought are not the only way that models think.

They're actually a very small, like if you think about the number of bits or something in the model's computation, it's actually a very small portion.

Most of the bits in the model's computation are still in this kind of black box, all these activations, if you've heard of them.

And so there are still, you know, there's still ways for the model to like only encode information in the activations that it doesn't actually reveal in the chain of thought.

And so I've been, I mean, this is this question of, you know, what types of information do

do models reveal and like in what situations do they reveal it in the chain of thought is maybe like the main like kind of question at the moment for the field.

But I've kind of like in some ways broken it down in my head as like system one and system two type thinking.

So just like you guys have kind of humans have like motor reflexes that you don't actively think about.

Any kind of muscular twitch that the model does won't probably be revealed in the chain of thought because it's, you know, it's like so baked in.

It doesn't need to actively think about it to decide to do that.

Yeah, exactly.

It's more instinctual.

And then anything that requires a decision point or something, or this may not even be the case, but things that require active decision points might be in the chain of thought actively.

And then there is kind of an argument with the transformer architectures that...

chain of thought actually increases the maximum serial depth of cognition a model can do.