“Open Distillation of Hereditary Traits” by Arthur Conmy

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded14 Jul 2026

Publisherfeeds.type3.audio ↗

Handlerec:19289465

Show notes

TL;DR Josh and Neel show that distillation from a teacher model to a base pretrained student model transfers some of the teacher model's traits (such as displaying negative emotion in the Gemma Needs Help evals)On its own this is pretty unsurprising, but Josh and Neel additionally show that even filtering out all the prompts and rollouts where the trait is mentioned doesn’t generally prevent the trait transferIn this post, I show a simple way to replicate and study these phenomena without access to a frontier SFT pipeline (or even running full SFT[1]) I distill Gemma 3's negative emotion into Qwen-base, Gemma 4's agentic misalignment into Nemotron Chat, and Qwen's Chinese censorship into Llama baseI end the post with a bunch of open questions that could be tackled with a setup similar to this approachI release all model weights here (https://huggingface.co/ArthurConmy/hereditary-weights) and all code here: https://github.com/ArthurConmy/hereditary (Note that my intention is more to make this work easy to build on rather than make the findings as clear as possible, hence apologies for leaning on AI more than I usually would) Intro The core idea is to: Generate rollouts from a teacher model which has a given trait [...] ---Outline:(00:10) TL;DR(01:19) Intro(02:42) Negative emotion(06:28) Blackmail(10:32) Chinese censorship(15:26) Follow-up questions(20:15) Appendices(35:29) Acknowledgements The original text contained 5 footnotes which were omitted from this narration. --- First published: July 14th, 2026 Source: https://www.lesswrong.com/posts/WpYFAmJDH3zuAq2ha/open-distillation-of-hereditary-traits-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.