Rohin Shah
speaker
1,071 appearances
1 recordings
1 series
first heard Jun 2026
last heard 2 Jun
Rohin Shah’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.
Appearances
and that I think will continue to happen in the future so we can just continue to plot this and then one hopes that like if an intelligence explosion does seem like it's happening we will start to see an acceleration on that rather than it looking just like a nice linear fit the entire time
Yeah, I basically agree with all of the critiques you mentioned.
Like, it's going to inherit any problems that benchmarks have because ultimately the only input into it is benchmark performance.
Nonetheless, I think a lesson that I learned from the gods of straight lines is that, like...
For some things, yeah, there are lots of nuances and details that matter a bunch if you want to make very fine-grained predictions.
But at a high level, they wash out and it ends up being fine anyway.
I think that mostly applies here.
Yes.
And also I think in practice, the teaching to the test won't change the results very much on this.
It definitely changes at some.
So I think like Anthropic is probably the best at not overfitting to the benchmarks that exist.
And in fact, if you look at the scores that are produced, I think it does tend to underestimate Claude or Anthropic's models relative to models from other providers.
Because generally, clad models, since they are not overfit to the benchmarks, will tend to underperform on benchmarks relative to how good the model actually is.
So yes, that's true.
But also, if you look at it on the graph, it's like a tiny little difference.
It's not that big.
If you moved it up, it would still look linear.
It doesn't really change whether it's linear or not.
Observation.
That's right.
Showing 821–840 of 1,071 · page 42 of 54
← Previous
Next →