“Global CoT Analysis: Initial attempts to uncover patterns across many chains of thought” by Riya Tyagi, daria, Arthur Conmy, Neel Nanda

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded15 Jan 2026

Publisherfeeds.type3.audio ↗

Handlerec:15349786

Show notes

Audio note: this article contains 37 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description. Authors: Riya Tyagi, Daria Ivanova, Arthur Conmy, Neel Nanda Riya and Daria are co-first authors. This work was largely done during a research sprint for Neel Nanda's MATS 9.0 training phase. 🖥️ Deployment code ⚙️ Interactive demo TL;DR We believe that more research effort should go into studying many chains of thought collectively and identifying global patterns across the collection. We argue this direction is important and tractable; to start, we focus on a simpler setting: analyzing a collection of CoTs generated with a fixed model on a fixed prompt. We introduce two methods—semantic step clustering and algorithmic step clustering—that group recurring patterns across reasoning traces at different levels of abstraction. Semantic step clustering leverages the fact that traces for the same prompt often repeat each other in slightly different ways. By clustering similar sentences like “1+2=3” and “2+1=3”, we compress a large set of CoTs into a detailed but messy reasoning graph. We evaluate semantic step clustering on a downstream task: predicting a model's output [...] ---Outline:(00:49) TL;DR(03:03) Introduction(06:59) Method 1: Semantic step clustering(07:04) Building the semantic graph(08:47) Chunking(09:31) Clustering(11:11) The issue of semantic drift(12:30) Interlude: CoTs are chaotic!(14:25) Sanity check: viewing our clusters(15:29) Downstream task: predictive power(16:54) Deriving predictions from our graph(19:30) Predictive power baselines(22:44) Results(27:00) Method 2: Algorithmic step clustering(27:58) Building the algorithm graph(29:35) Sanity check: studying sample problems(29:48) Tree structure(30:45) Convergent structure(31:37) Qualitative insights(31:41) Models reuse strategies frequently(33:24) Different models solve problems differently(34:31) Takeaways and future work(37:26) Acknowledgements(37:41) Contribution Statement(38:55) Appendix(38:58) Chunking(39:47) Clustering(39:55) Stage 1: Agglomerative Clustering with Semantic Embedding Model(41:01) Stage 2: LLM refinement via the Leiden Algorithm(42:28) Predictive power(42:31) Semantic graph prompts(43:00) Method ablation study(44:31) A comparison of three Gemini baselines:(45:31) Gemini predictive power baseline prompts(45:51) Gemini no rollout baseline (0 rollouts)(46:13) Gemini 20 rollout baseline(46:35) Gemini full context baseline (1000 rollouts)(46:58) Footnotes The original text contained 15 footnotes which were omitted from this narration. --- First published: January 13th, 2026 Source: https://www.lesswrong.com/posts/q9g9zuudd3Pvw2cbj/global-cot-analysis-initial-attempts-to-uncover-patterns-1 --- Narrated by TYPE III AUDIO. ---Images from the article:__T3A_INLINE_LATEX_PLACEHOLDER___\min(p_{\text{correct}},p_{\text{incorrect}})___T3A_INLINE_LATEX_END_PLACEHOLDER__ for a prefix: 0-10%, 10-30%, and 30-50%." style="max-width: 100%;" /Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.