“11 Open Empirical Problems in Reward-Seeking” by Alex Meinke, Jérémy Scheurer, Axel Højmark, Theodore Ehrenborg

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded21 Jul 2026

Publisherfeeds.type3.audio ↗

Handlerec:19382566

Show notes

We recently published our paper on "Measuring Reward-Seeking via Contrastive Belief Updates". We're excited about research like this, and there are many more open problems than we can work on. Here's a list of open problems that we think are valuable. If you work on/solve these problems, we'd be happy to signal-boost your research. If your next research project is one of these problems, feel free to reach out to [email protected] to discuss it in more detail. Reward-Seeking and its Implications 1. Is a Reward-Seeking Model more difficult to align? The strongest case for expecting reduced "train-time corrigibility" due to reward-seeking, applies to Instrumental Reward-Seeking, where the model actively reasons "I will please oversight now, in order to accomplish some other thing later". Alignment training a model like that may update its beliefs about graders and oversight, without reshaping its underlying values. Current forms of reward-seeking are likely better understood as terminal, i.e. models try to please the grader without ulterior motives. There is likely a continuous spectrum between Terminal and Instrumental Reward-Seeking. Thus, we can hopefully study the effects that mostly Terminal Reward-Seeking has on train-time corrigibility now, in the hopes of learning about the effects that Instrumental [...] ---Outline:(00:43) Reward-Seeking and its Implications(00:47) 1. Is a Reward-Seeking Model more difficult to align?(02:20) 2. Can we measure Instrumental Reward-Seeking?(03:25) 3. When does Reward-Seeking most increase / decrease?(04:50) 4. Are there better Belief Update Techniques than SDF?(07:06) Improving SDF(07:10) 5. Can AIs detect the difference between pretraining facts and SDF facts?(07:49) 6. How is SDF different from changing pretraining?(09:32) 7. Does SDF have off-target effects?(11:29) 8. SDF sometimes generalizes in surprising ways(12:31) 9. Getting good synthetic document recall is finicky(14:11) Better Validation Techniques(14:23) 10. Our model organisms could be more robust(15:32) 11. Can we get more bits of information for ground-truth? The original text contained 1 footnote which was omitted from this narration. --- First published: July 21st, 2026 Source: https://www.lesswrong.com/posts/8wXRuHQqCbRsbap6q/11-open-empirical-problems-in-reward-seeking --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.