Rohin Shah
speaker
1,071 appearances
1 recordings
1 series
first heard Jun 2026
last heard 2 Jun
Rohin Shah’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.
Appearances
And so if you tie pre-deployment evals to... Or if you tie evals to and require them to be pre-deployment, then that is...
providing a pretty strong incentive to make those evals as fast as possible to run and get them done as quickly as possible, which is maybe not really the incentive you want to give.
Obviously, we're going to try and make them as good as possible, but it's still a constraint.
We probably could do better if we had more time.
And there's some amount where we can push back and say, actually, we need the time to do the evals, but it's not an infinite amount.
And so that, I think, is just really quite a large cost.
Now, it might be worth it if there is strong benefits, but I just don't think there are particularly strong benefits.
One that people would naturally say is, well, you need to know if you're releasing a dangerous model.
Ideally, you don't release the dangerous model, and the way you have to do that is via pre-deployment evals.
To this, I would mostly say that AI progress
is reasonably continuous, you can get a decent sense of how the next AI system is going to behave based on the previous one.
You can have some reasonable OK bounds on this.
And so if you design your evals and your thresholds such that there is a reasonable safety buffer between when your evals trigger versus when you actually think the model is dangerous,
then it seems just basically totally fine to say, okay, we evaluated the previous model or we ran an evaluation a month ago.
It's not going to have had a huge giant leap in that time.
It was under our threshold at the time.
There's the safety buffer.
Therefore, we're not worried about this model.
And so I think, and this is the approach we've been taking in our frontier safety framework since the very first time it was published.
It's not particularly new.
Showing 241–260 of 1,071 · page 13 of 54
← Previous
Next →