“Shaping the exploration of the motivation-space matters for AI safety” by Maxime Riché, Victor Gillioz, nielsrolf, Kajetan Dymkiewicz, Filip Sondej, RogerDearnaley, Daniel Tan, dillonkn

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded06 Mar 2026

Publisherfeeds.type3.audio ↗

Handlerec:16782031

Show notes

Summary We argue that shaping RL exploration, and especially the exploration of the motivation-space, is understudied in AI safety and could be influential in mitigating risks. Several recent discussions hint in this direction — the entangled generalization mechanism discussed in the context of Claude 3 Opus's self-narration, the success of using inoculation prompting against natural emergent misalignment and its relation to shaping the model self-perception, and the proposal to give models affordances to report reward-hackable tasks — but we don't think enough attention has been given to shaping exploration specifically. When we train models with RL, there are two kinds of exploration happening simultaneously: Action exploration — what the model does. Motivation exploration — why the model does it, and how it perceives itself while doing it. Both explorations occur during a critically sensitive and formative phase of training, but (2) is significantly less specified than (1), and this underspecification is both a danger and an opportunity. Because motivations are so underdetermined by the reward signal, we may be able to shape them without running into the downstream problems of blocking access to high-rewards, such as deceptive alignment. Capability researchers have strong incentives to develop effective techniques for [...] ---Outline:(00:14) Summary(02:11) Why shaping exploration matters(03:27) Instruct models are ok -- RL is where the highest risks are(05:40) Motivations are the most underspecified -- a problem and an opportunity(08:30) We may not prevent power-seeking, but we may create wise power-seekers(10:57) Why capability researchers may not solve this for us(12:57) Empirical evidence, research directions, and open questions(13:21) Empirical evidence(14:38) Research directions(17:53) Open questions and uncertainties The original text contained 4 footnotes which were omitted from this narration. --- First published: March 6th, 2026 Source: https://www.lesswrong.com/posts/rhFXyfFSRKp3cX4Y9/shaping-the-exploration-of-the-motivation-space-matters-for --- Narrated by TYPE III AUDIO.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.