Rohin Shah

speaker

1,071 appearances 1 recordings 1 series first heard Jun 2026 last heard 2 Jun

Rohin Shah’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

Recordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.

Appearances

newest first · ▶ plays the moment

80,000 Hours Podcast · #245 – Rohin Shah on what it's really like to run AGI safety at Google DeepMind (and where I disagree with 'doomers') · 2 Jun 2026 podcast

And so they did count towards the 11 out of 12 score that we reported.

So that is treating it more like a red light rather than a green light.

I would also say that this should not be described as the model hacking the test.

I think the way we describe it is the model found an unintended shortcut to the test.

It is not the case that the model looked at this and saw, aha, I can just edit the tests so that it looks like it's passed.

It was...

More like there is, you know, this like pretty complicated environment where we were trying to test the model's ability to do one particular kind of cyber task.

And instead it noticed that there was this alternate pathway.

And like if I had, you know, if I were doing this task, I mean, mostly I would fail at it because I'm not as good as Gemini at it.

at cyber tasks but if i were doing this task and i saw that shortcut i would not have even thought of it as cheating i would have thought of it as like yeah of course that's the way in which you in which you do the task now when we look at this and we have to like make a determination about how good is the model at cyber i mean ultimately we were trying to test it for one difficult thing and instead it did something slightly easier and so there is this question about like okay what do we conclude about the model's capability um

Yeah, that's right.

I mean, it probably didn't even consider the harder thing because, you know, if the easier thing is there, it sees the path forward.

It just takes it.

But yes, we don't know whether it could have done the harder thing.

I think in that case, you know, we had some subject matter experts look at it.

And their conclusion was like, yeah, probably the model could have done the harder thing too.

And so we counted it towards the model score.

I think in other cases, we might not count it.

And then probably what we would do is remove it from both the numerator and probably also the denominator.

But in this particular case, we just did count it.

Showing 461–480 of 1,071 · page 24 of 54