Rohin Shah
speaker
1,071 appearances
1 recordings
1 series
first heard Jun 2026
last heard 2 Jun
Rohin Shah’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jun 2026 with 1.
Appearances
And so the idea is, suppose there is a reward hack that requires you to take two different actions separated in time.
So for example, maybe if you are tasked with writing some code, you first write some tests that are particularly easy to pass.
They don't actually test everything that is needed.
And then you write code that passes the tests because you didn't know how to write the correct code that would have passed everything.
Like both of these steps have to be present.
in order for the reward hack to actually succeed.
If you have one but not the other, you just get penalized.
And so the idea is if you treat these as two separate steps, like there's one step in which the tests are written and then a different step in which the code is written, and you optimize the tests only based on a signal that says, how good do these tests look?
and then separately you optimize the code writing part based solely on how good does this code look based on what has happened previously, then there's actually no incentive for the AI system to write bad tests.
Normally, the way the incentive would come is that when you're doing reinforcement learning,
um the reward on the code part would also be back propagated through time um to also affect the test writing part and so the ai could learn from that back propagated reward that like if it um writes easy to pass tests then it tends to get more reward in the future and so the myopic optimization part is saying we're not letting it see what happened in the future but
We're only evaluating it based on what it did right now.
And so that incentive never comes.
And this is why the reward hack isn't incentivized, even if you can't detect it.
Does that part make sense?
Yeah, that's right.
So it's historically been called process supervision.
I think nowadays the term process supervision has gotten a bunch of other meanings as well, which is why we have this a little bit more clear about what the actual technical mechanism is of
myopic optimization, but with non-myopic approval.
But yes, it's a very old idea.
Showing 581–600 of 1,071 · page 30 of 54
← Previous
Next →