Rohin Shah
speaker
1,071 appearances
1 recordings
1 series
first heard Jun 2026
last heard 2 Jun
Rohin Shah’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.
Appearances
If I remember correctly from the post, I think Zvi's bigger objection was that we had a second test that we didn't report quantitative results on, but that was pretty key to us ruling out the cyber CCL.
And why did that happen?
Mostly because this test is still, to some extent, under construction.
Yeah.
I'm confident that was robust enough to rule out the CCL, but why don't I want to put great details into the Frontier Safety Report?
Mainly because
the details of exactly how we run this eval are likely going to change in order to make it somewhat more robust to include a couple more challenges.
And then in the next frontier safety report, we have to write an entire section about how we change this stuff, and previous results aren't comparable.
If you want something like that, you'd basically have to go to the level, at minimum, the level of detail and rigor that is found in an academic paper, and often even that's not enough.
I do see a lot of value in publishing papers about our evaluations, and we have done this, I think more so than most other companies.
We published one of the first leading papers on evaluations.
It's called Evaluating Frontier Models for Dangerous Capabilities, I think.
Yeah, I think we spoke about that with Alan Defoe about a year ago.
Very nice, yes.
And then I think more recently we published our evaluations for Stealth and Situational Awareness.
So I think papers are a place where I do feel like there's clear benefits.
It allows people to understand what the evaluations are actually doing.
It allows for other people to build on top of those evaluations.
And so we do put more effort into that.
Whereas I think for the Frontier Safety Report, I feel like its primary purpose is say that we made this determination formally and less about providing enough details that people can independently check our work.
Showing 481–500 of 1,071 · page 25 of 54
← Previous
Next →