Rohin Shah

speaker

1,071 appearances 1 recordings 1 series first heard Jun 2026 last heard 2 Jun

Rohin Shah’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.

Appearances

newest first · ▶ plays the moment

80,000 Hours Podcast · #245 – Rohin Shah on what it's really like to run AGI safety at Google DeepMind (and where I disagree with 'doomers') · 2 Jun 2026 podcast

I'd rather we didn't do it, but I don't think it is like a fatal flaw.

And I think I've seen some

My sense, I haven't been following the literature on this too closely, but my sense from the papers that I have looked at is that even when you do this sort of thing, so sometimes it just like makes performance worse.

And even when it does make performance better, if you actually try to look at what these tokens are that it's doing by just looking at the full probability distribution, it does just look like it basically has maybe one main reasoning track that's going on, and you can still follow along pretty easily.

Yeah, I would say that you should maybe look at not just the top one, but maybe the top five.

Maybe even just the top two would be enough.

You can also usually tell how much you are missing because there will probably, for most of the architectures I've seen, there's a way where you can modify it at test time.

so that it only uses the information of the top two tokens and everything else, you just get rid of it, you ablate it away.

And then you can do that intervention and you can see, does the model still perform as well as it did before?

And my prediction is, yeah, it will probably perform about as well as it did before.

And if that happens, then I think you can be reasonably confident that it's not smuggling in a bunch of information in the rest of the probability distribution.

Yeah, that's right.

I should give a caveat, which is like, you know, probably if you do enough fine tuning and enough RL on the model to get it to just use, instead of treating this as a probability distribution over tokens, just treat it as like a vector of numbers and just make that vector of numbers as useful as possible.

With enough training, probably it will get to the point where you can no longer interpret it as a distribution over tokens.

I think the papers I've seen on this suggest that this works less well than just treating it as a distribution over tokens.

Yeah, basically.

In fact, this is what the original Coconut paper proposed.

And I think some of the follow-on work after Coconut

was basically said, yeah, you know, the problem with this is that it's too expressive.

We need to restrict it to just the probability distribution over tokens, and then it performs better, which I think reflects this fact that actually the models are much better at doing this sort of human-like reasoning.

Showing 421–440 of 1,071 · page 22 of 54