Designing How AI Grows — Tom McGrath

episode

Machine Learning Street Talk (MLST) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from Machine Learning Street Talk (MLST)

VAEs Are Energy-Based Models? [Dr. Jeff Beck] 46 min · 25 Jan 2026

Abstraction & Idealization: AI's Plato Problem [Mazviita Chirimuuta] 53 min · 23 Jan 2026

Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION] 42 min · 23 Jan 2026

Adversarial Examples and Data Modelling - Andrew Ilyas (MIT) 1h 28m · 22 Aug 2024

All episodes from Machine Learning Street Talk (MLST) →

Report any issue

Source

Recorded02 Sep 2026

Publisheranchor.fm ↗

Handlerec:20232135

Show notes

Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs.Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it.The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire.---TIMESTAMPS:00:00:00 Introduction: Can interpretability speed-run science?00:02:03 The invisible grader00:06:51 What AlphaZero learned from the world00:12:24 Interpretability as a control loop00:21:54 The forbidden method and safer interventions00:37:36 Why models catch hallucinations too late00:46:19 Debug the dataset before training00:50:44 Why neural networks become modular00:55:57 Finding the geometry inside a network01:02:55 Why steering falls off the manifold01:12:10 A reusable calculator inside Llama01:17:19 From abstractions to goals01:25:28 Reward hacking, oversight and collusion01:37:23 Are sparse autoencoders dead?---REFERENCES:paper:[00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMshttps://arxiv.org/abs/2502.17424v7[00:11:05] Acquisition of Chess Knowledge in AlphaZerohttps://arxiv.org/abs/2111.09259[00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuninghttps://arxiv.org/abs/2507.16795[00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Modelshttps://arxiv.org/abs/2507.21509[00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretabilityhttps://arxiv.org/abs/2602.10067[00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalhttps://arxiv.org/abs/2606.12360[01:00:26] Do Sparse Autoencoders Capture Concept Manifolds?https://arxiv.org/abs/2604.28119[01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorhttps://arxiv.org/abs/2605.05115[01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Conceptshttps://arxiv.org/abs/2605.01148[01:29:35] Measuring Reward-Seeking via Contrastive Belief Updateshttps://arxiv.org/abs/2607.18966v1other:[00:15:44] Intentional Designhttps://www.goodfire.com/blog/intentional-design[00:56:12] The World Inside Neural Networkshttps://www.goodfire.com/research/the-world-inside-neural-networks[01:37:28] A Pragmatic Vision for Interpretabilityhttps://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability---RESCRIPT:https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.