“Recent LLMs can do 2-hop and 3-hop latent (no-CoT) reasoning on natural facts” by ryan_greenblatt

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded01 Jan 2026

Publisherfeeds.type3.audio ↗

Handlerec:15020717

Show notes

Prior work has examined 2-hop latent (by "latent" I mean: the model must answer immediately without any Chain-of-Thought) reasoning (e.g., "What element has atomic number (the age at which Tesla died)?") and found that LLM performance was limited aside from spurious successes (from memorization and shortcuts). I find that recent LLMs can now do 2-hop and 3-hop latent reasoning with moderate accuracy. I construct a new dataset for evaluating n-hop latent reasoning on natural facts (as in, facts that LLMs already know). On this dataset, I find that Gemini 3 Pro gets 60% of 2-hop questions right and 34% of 3-hop questions right. Opus 4 performs better than Opus 4.5 at this task; Opus 4 gets 31% of 2-hop questions right and 7% of 3-hop questions right. All models I evaluate have chance or near chance accuracy on 4-hop questions. Older models perform much worse; for instance, GPT-4 gets 9.7% of 2-hop questions right and 3.9% of 3-hop questions right. I believe this new dataset I've created is the best existing dataset for evaluating n-hop latent reasoning. According to Balesni et al., prior datasets based on natural facts had issues with spurious successes while synthetic facts (introduced to [...] ---Outline:(06:22) Appendix: resolving Leo Gaos Manifold questions about fact composition(10:40) Appendix: The effect of problem repeats and few-shots on these fact composition questions(12:43) Appendix: adding the result of N 1-hop questions(17:41) Appendix: Effects of using a different fact distribution(18:31) Appendix: what 3-hop questions is Gemini 3 Pro getting right?(21:51) Appendix: Dataset description(30:02) Dataset sanity checks(30:37) Appendix: AI usage The original text contained 7 footnotes which were omitted from this narration. --- First published: January 1st, 2026 Source: https://www.lesswrong.com/posts/aYtrLhoZtCKZnfBvA/recent-llms-can-do-2-hop-and-3-hop-latent-no-cot-reasoning --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.