LessWrong (Curated & Popular)

"Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers" by Sam Marks, Adam Karvonen, James Chua, Subhash Kantamneni, Euan Ong, Julian Minder, Clément Dumas, Owain_Evans

20 Dec 2025

Audio

Description

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity. The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post. Thread New paper: We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language. We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so. We aim to make a general-purpose LLM for explaining activations by: 1. Training on a diverse set of tasks 2. Evaluating on tasks very different from training This extends prior work (LatentQA) that studied activation verbalization in narrow settings. Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies. Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like! We [...] ---Outline:(00:46) Thread(04:49) Blog post(05:27) Introduction(07:29) Method(10:15) Activation Oracles generalize to downstream auditing tasks(13:47) How does Activation Oracle training scale?(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?(19:31) Conclusion The original text contained 3 footnotes which were omitted from this narration. --- First published: December 18th, 2025 Source: https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general --- Narrated by TYPE III AUDIO. ---Images from the article:

Transcription

This episode hasn't been transcribed yet

Help us prioritize this episode for transcription by upvoting it.

0 upvotes

🗳️ Sign in to Upvote

Popular episodes get transcribed faster

Other episodes from LessWrong (Curated & Popular)

Transcribed and ready to explore now

"Ada Palmer: Inventing the Renaissance" by Martin Sustrik

28 Jan 2026

LessWrong (Curated & Popular)

"The inaugural Redwood Research podcast" by Buck, ryan_greenblatt

27 Jan 2026

LessWrong (Curated & Popular)

"Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German" by jenn

26 Jan 2026

LessWrong (Curated & Popular)

[Linkpost] "“The first two weeks are the hardest”: my first digital declutter" by mingyuan

20 Jan 2026

LessWrong (Curated & Popular)

"Why we are excited about confession!" by boazbarak, Gabriel Wu, Manas Joglekar

19 Jan 2026

LessWrong (Curated & Popular)

"Backyard cat fight shows Schelling points preexist language" by jchan

16 Jan 2026

LessWrong (Curated & Popular)

View all episodes from LessWrong (Curated & Popular)

Comments

There are no comments yet.

Please log in to write the first comment.

Report any issue

LessWrong (Curated & Popular)

"Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers" by Sam Marks, Adam Karvonen, James Chua, Subhash Kantamneni, Euan Ong, Julian Minder, Clément Dumas, Owain_Evans

This episode hasn't been transcribed yet

Other episodes from LessWrong (Curated & Popular)

"Ada Palmer: Inventing the Renaissance" by Martin Sustrik

"The inaugural Redwood Research podcast" by Buck, ryan_greenblatt

"Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German" by jenn

[Linkpost] "“The first two weeks are the hardest”: my first digital declutter" by mingyuan

"Why we are excited about confession!" by boazbarak, Gabriel Wu, Manas Joglekar

"Backyard cat fight shows Schelling points preexist language" by jchan

Sign in to Audioscrape

Share this moment