“In-context learning alone can induce weird generalisation” by Cozmin Ududec, Benji Berczi, Kyuhee Kim

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded25 Feb 2026

Publisherfeeds.type3.audio ↗

Handlerec:16154884

Show notes

Benji Berczi, Kyuhee Kim, Cozmin Ududec, James Requeima This is work done by Kyuhee and Benji during MATS Winter 2026, mentored by Cozmin Ududec, and in collaboration with James. TL;DR Weird generalisation can happen just with prompting, without fine-tuning. Just by adding benign biographical facts (e.g. facts about Hitler in a Q&A format) into the context window of Llama 3.3 70B, we induce a sharp persona transition: the model starts identifying as Hitler after only 5-10 facts and its alignment score on unrelated questions drops from ~92 to ~53. The transition follows a sigmoid phase curve that fits the Bigelow et al. belief-dynamics model, with a phase boundary (achieving 50% Hitler identity) at only ~6 Hitler facts. ICL can also create gated (backdoor) personas. By mixing tagged benign Hitler facts (see WG paper Figure 6) with untagged normal AI facts in context, the model learns to compartmentalise its behaviour based on the tags: when evaluated with tags, it triggers the Hitler persona, but without tags, it stays as a normal assistant. Flipping which set of facts is tagged reverses this, confirming that the tags drive the compartmentalisation. Anti-evidence slows the transition from AI assistant to the Hitler persona and partially [...] ---Outline:(00:28) TL;DR(02:59) Context: weird generalisation and belief dynamics(04:45) Setup(07:19) Result 1: ICL alone can cause weird generalisation(09:00) Result 2: ICL can create gated (backdoor) personas(10:57) Result 3: ICL anti-evidence partially reverses SFT-induced personas(12:38) Result 4: Tagged SFT models maintain separate posteriors(13:59) Generalisation across models and personas(17:57) Discussion(19:01) What were working on next The original text contained 1 footnote which was omitted from this narration. --- First published: February 25th, 2026 Source: https://www.lesswrong.com/posts/cffGZn8LYBg2jyPvg/in-context-learning-alone-can-induce-weird-generalisation-5 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.