“Can activation verbalizers surface an internal chain of thought?” by oakhu, ryan_greenblatt

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded07 Jun 2026

Publisherfeeds.type3.audio ↗

Handlerec:18760679

Show notes

We introduce an evaluation for activation verbalizers: can they surface a target model's reasoning as it solves a math problem in a single forward pass? For open-weight NLAs, the answer seems to be: "possibly, but definitely not reliably". Lots of important capabilities currently require AI models to reason "out loud" in a natural-language chain of thought, which means that we can monitor important parts of their thinking. It would be nice to have this same affordance for the reasoning that models do within a single forward pass, especially if the sophistication of that opaque reasoning increases to potentially dangerous levels. Some interpretability tools might offer such an affordance. In particular, an activation verbalizer (AV) takes a residual stream activation and maps it to a natural-language verbalization. An AV is initialized from the target model and trained to generate verbalizations that an activation reconstructor (AR), also initialized from the target model, can accurately map back to the original activation. Together, an AV and its AR form a natural-language autoencoder (NLA). Importantly, AVs see only a single activation; they do not see the target model's prompt or next-token output, and – unlike activation oracles (AOs) – they are not asked any [...] ---Outline:(02:32) Takeaways(02:43) These NLAs aren't that great at reconstruction(05:20) It's not clear that NLAs aren't just confabulating from their best guess at the target model's prompt and output(07:22) Verbalizations do not reliably predict the model's output(10:03) Verbalizations can surface some hints of reasoning, but rarely with any coherence(13:02) Verbalizations do not reliably surface causes of wrong outputs(15:02) More speculative notes(20:56) Evaluating verbalizations on their ability to recover mathematical reasoning(24:35) Outputs & Answers - often mentioned, rarely singled out(27:16) Methods, Values, & Coherence - some early indications(30:07) Mistakes - almost no plausible explanations of wrong outputs(33:22) Problems - low-fidelity reconstructions(34:54) Appendix A: Sample verbalizations(35:12) Problem 301 (comparison)(35:33) Gemma(36:20) Llama(37:12) Qwen3 AO(37:56) Qwen2.5(38:39) Takeaways(38:42) Problem 533.(40:48) Problem 448.(41:55) Problem 549.(44:31) Problem 691.(45:46) Problem 340.(46:47) Output & Answers(46:50) Problem 824.(47:46) Problem 479.(48:53) Problem 509(49:54) Methods, Values, & Coherence(49:59) Problem 609(52:47) Problem 423.(54:22) Problem 456.(56:36) Problem 341.(58:24) Problems(58:27) Problem 886.(59:34) Appendix B: Grader elicitation(01:05:26) Appendix C: Red-teaming(01:10:42) Appendix D: Steering on mistakes(01:12:24) Appendix E: Eliciting verbalizations from a question-answering activation oracle(01:14:24) Problem 545 - Qwen3 AO (confabulation)(01:15:20) Problem 47 - Qwen3 AO (apparently-good CoT)(01:16:28) Problem 216 - Qwen3 AO (grader-tricking confabulation)(01:17:44) Appendix F: Full-dataset results(01:18:55) Appendix G: cross-layer results The original text contained 18 footnotes which were omitted from this narration. --- First published: June 6th, 2026 Source: https://www.lesswrong.com/posts/QQQAcKuWK6k98FivY/can-activation-verbalizers-surface-an-internal-chain-of-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.