LessWrong (Curated & Popular)

“Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)” by Neel Nanda, lewis smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, Tom Lieberum, János Kramár, Rohin Shah

12 Apr 2025

Audio

Description

Audio note: this article contains 31 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description. Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda * = equal contribution The following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders were useful for downstream tasks, notably out-of-distribution probing.TL;DR To validate whether SAEs were a worthwhile technique, we explored whether they were useful on the downstream task of OOD generalisation when detecting harmful intent in user prompts [...] ---Outline:(01:08) TL;DR(02:38) Introduction(02:41) Motivation(06:09) Our Task(08:35) Conclusions and Strategic Updates(13:59) Comparing different ways to train Chat SAEs(18:30) Using SAEs for OOD Probing(20:21) Technical Setup(20:24) Datasets(24:16) Probing(26:48) Results(30:36) Related Work and Discussion(34:01) Is it surprising that SAEs didn't work?(39:54) Dataset debugging with SAEs(42:02) Autointerp and high frequency latents(44:16) Removing High Frequency Latents from JumpReLU SAEs(45:04) Method(45:07) Motivation(47:29) Modifying the sparsity penalty(48:48) How we evaluated interpretability(50:36) Results(51:18) Reconstruction loss at fixed sparsity(52:10) Frequency histograms(52:52) Latent interpretability(54:23) Conclusions(56:43) AppendixThe original text contained 7 footnotes which were omitted from this narration. --- First published: March 26th, 2025 Source: https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/sae-progress-update-2-draft --- Narrated by TYPE III AUDIO. ---Images from the article:

Transcription

This episode hasn't been transcribed yet

Help us prioritize this episode for transcription by upvoting it.

0 upvotes

🗳️ Sign in to Upvote

Popular episodes get transcribed faster

Other episodes from LessWrong (Curated & Popular)

Transcribed and ready to explore now

"Ada Palmer: Inventing the Renaissance" by Martin Sustrik

28 Jan 2026

LessWrong (Curated & Popular)

"The inaugural Redwood Research podcast" by Buck, ryan_greenblatt

27 Jan 2026

LessWrong (Curated & Popular)

"Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German" by jenn

26 Jan 2026

LessWrong (Curated & Popular)

[Linkpost] "“The first two weeks are the hardest”: my first digital declutter" by mingyuan

20 Jan 2026

LessWrong (Curated & Popular)

"Why we are excited about confession!" by boazbarak, Gabriel Wu, Manas Joglekar

19 Jan 2026

LessWrong (Curated & Popular)

"Backyard cat fight shows Schelling points preexist language" by jchan

16 Jan 2026

LessWrong (Curated & Popular)

View all episodes from LessWrong (Curated & Popular)

Comments

There are no comments yet.

Please log in to write the first comment.

Report any issue

LessWrong (Curated & Popular)

“Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)” by Neel Nanda, lewis smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, Tom Lieberum, János Kramár, Rohin Shah

This episode hasn't been transcribed yet

Other episodes from LessWrong (Curated & Popular)

"Ada Palmer: Inventing the Renaissance" by Martin Sustrik

"The inaugural Redwood Research podcast" by Buck, ryan_greenblatt

"Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German" by jenn

[Linkpost] "“The first two weeks are the hardest”: my first digital declutter" by mingyuan

"Why we are excited about confession!" by boazbarak, Gabriel Wu, Manas Joglekar

"Backyard cat fight shows Schelling points preexist language" by jchan

Sign in to Audioscrape

Share this moment