[Linkpost] “On the Origins of Algorithmic Progress in AI” by alex_fogelson

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded10 Jan 2026

Publisherfeeds.type3.audio ↗

Handlerec:15232498

Show notes

This is a link post. This is a linkpost to a new Substack article from MIT FutureTech explaining our recent paper On the Origins of Algorithmic Progress in AI. We demonstrate that some algorithmic innovations have efficiency gains which get larger as pre-training compute increases. These scale-dependent innovations constitute the majority of pre-training efficiency gains over the last decade, which may imply that what looks like algorithmic progress is driven by compute scaling rather than many incremental innovations. From the paper, our core contributions are: We find most algorithmic innovations we experimentally evaluate have small, scale-invariant efficiency improvements with less than 10× compute efficiency gain overall, and representing less than 10% of total improvements extrapolated to the 2025 compute frontier (2 × 10²³ FLOPs). This suggests that scale-invariant algorithmic progress contributes only a minor share of overall efficiency improvements. We find two strongly scale-dependent algorithmic innovations: LSTMs to Transformers, and Kaplan to Chinchilla re-balancing. Together, these account for 91% of total efficiency gains when extrapolating to the 2025 compute frontier. This implies that algorithmic progress for small-scale models is several orders of magnitude smaller than previously thought. We show that in the presence of scale-dependent innovations, not only [...] --- First published: January 9th, 2026 Source: https://www.lesswrong.com/posts/X8KGHstcJa4qZznfH/on-the-origins-of-algorithmic-progress-in-ai Linkpost URL:https://open.substack.com/pub/mitfuturetech/p/on-the-origins-of-algorithmic-progress --- Narrated by TYPE III AUDIO.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.