“Physics of RL: Toy scaling laws for the emergence of reward-seeking” by AlexMeinke

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded04 Mar 2026

Publisherfeeds.type3.audio ↗

Handlerec:16542178

Show notes

TL;DR: When is or isn't reward the optimization target? I use a mathematical toy model to reason about when RL should select for reward-seeking reasoning as opposed to behaviors that achieve high reward without thinking about reward. Hypothesis: More diverse RL increases the likelihood that reward-seeking emerges. In my toy model, there are scaling laws for the emergence of reward-seeking: when decreasing prior probability of reward-seeking reasoning we can increase RL diversity enough that reward-seeking still dominates after training. The transition boundary is a ~straight line on a log-log plot. Opinion: To get to a hard "science of scheming", we should study the "physics of RL dynamics". Motivation Why would RL training lead to reward-seeking reasoning, scheming reasoning, power-seeking reasoning or any other reasoning patterns for that matter? The simplest hypothesis is the behavioral selection model. In a nutshell, a cognitive pattern that correlates with high reward will increase throughout RL.[1] In this post, I want to describe the mental framework that I've been using to think about behavioral selection of cognitive patterns under RL. I will use reward-seeking as my primary example, because it is always behaviorally fit under RL[2] and is a necessary condition for deceptive [...] ---Outline:(00:59) Motivation(02:38) A mathematical toy model for behavioral selection(05:28) Warm-up Examples(05:41) Single-Environment(07:48) Homogenous Coupling(09:32) Example Scaling Laws(11:11) Heterogenous Coupling(15:31) Takeaways The original text contained 6 footnotes which were omitted from this narration. --- First published: March 4th, 2026 Source: https://www.lesswrong.com/posts/9FH49ZgJFW4WtbxLi/physics-of-rl-toy-scaling-laws-for-the-emergence-of-reward --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.