“Weight-Sparse Circuits May Be Interpretable Yet Unfaithful” by jacob_drori

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded10 Feb 2026

Publisherfeeds.type3.audio ↗

Handlerec:15820386

Show notes

TLDR: Recently, Gao et al trained transformers with sparse weights, and introduced a pruning algorithm to extract circuits that explain performance on narrow tasks. I replicate their main results and present evidence suggesting that these circuits are unfaithful to the model's “true computations”. This work was done as part of the Anthropic Fellows Program under the mentorship of Nick Turner and Jeff Wu. Introduction Recently, Gao et al (2025) proposed an exciting approach to training models that are interpretable by design. They train transformers where only a small fraction of their weights are nonzero, and find that pruning these sparse models on narrow tasks yields interpretable circuits. Their key claim is that these weight-sparse models are more interpretable than ordinary dense ones, with smaller task-specific circuits. Below, I reproduce the primary evidence for these claims: training weight-sparse models does tend to produce smaller circuits at a given task loss than dense models, and the circuits also look interpretable. However, there are reasons to worry that these results don't imply that we're capturing the model's full computation. For example, previous work [1, 2] found that similar masking techniques can achieve good performance on vision tasks even when applied to a [...] ---Outline:(00:36) Introduction(03:03) Tasks(03:16) Task 1: Pronoun Matching(03:47) Task 2: Simplified IOI(04:28) Task 3: Question Marks(05:10) Results(05:20) Producing Sparse Interpretable Circuits(05:25) Zero ablation yields smaller circuits than mean ablation(06:01) Weight-sparse models usually have smaller circuits(06:37) Weight-sparse circuits look interpretable(09:06) Scrutinizing Circuit Faithfulness(09:11) Pruning achieves low task loss on a nonsense task(10:24) Important attention patterns can be absent in the pruned model(11:26) Nodes can play different roles in the pruned model(14:15) Pruned circuits may not generalize like the base model(16:16) Conclusion(18:09) Appendix A: Training and Pruning Details(20:17) Appendix B: Walkthrough of pronouns and questions circuits(22:48) Appendix C: The Role of Layernorm The original text contained 6 footnotes which were omitted from this narration. --- First published: February 9th, 2026 Source: https://www.lesswrong.com/posts/sHpZZnRDLg7ccX9aF/weight-sparse-circuits-may-be-interpretable-yet-unfaithful --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.