“Catch-Up Algorithmic Progress Might Actually be 60× per Year” by Aaron_Scher

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded24 Dec 2025

Publisherfeeds.type3.audio ↗

Handlerec:14898286

Show notes

Epistemic status: This is a quick analysis that might have major mistakes. I currently think there is something real and important here. I’m sharing to elicit feedback and update others insofar as an update is in order, and to learn that I am wrong insofar as that's the case. Summary The canonical paper about Algorithmic Progress is by Ho et al. (2024) who find that, historically, the pre-training compute used to reach a particular level of AI capabilities decreases by about 3× each year. Their data covers 2012-2023 and is focused on pre-training. In this post I look at AI models from 2023-2025 and find that, based on what I think is the most intuitive analysis, catch-up algorithmic progress (including post-training) over this period is something like 16×–60× each year. This intuitive analysis involves drawing the best-fit line through models that are on the frontier of training-compute efficiency over time, i.e., those that use the least training compute of any model yet to reach or exceed some capability level. I combine Epoch AI's estimates of training compute with model capability scores from Artificial Analysis's Intelligence Index. Each capability level thus yields a slope from its fit line, and these [...] ---Outline:(00:29) Summary(02:37) What do I mean by 'algorithmic progress'?(06:02) Methods and Results(08:16) Sanity check: Qwen2.5-72B vs. Qwen3-30B-A3B(10:09) Discussion(10:12) How does this compare to the recent analysis in A Rosetta Stone for AI Benchmarks?(14:47) How does this compare to other previous estimates of algorithmic progress(17:44) How should we update on this analysis?(20:13) Appendices(20:17) Appendix: Filtering by different confidence levels of compute estimates(20:24) All models(20:45) Confident compute estimates(21:07) Appendix: How fast is the cost of AI inference falling?(23:56) Appendix: Histogram of 1 point buckets(24:29) Appendix: Qwen2.5 and Qwen3 benchmark performance(25:31) Appendix Leave-One-Out analysis(27:08) Appendix: Limitations(27:13) Outlier models(29:41) Lack of early, weak models(30:35) Post-training compute excluded(31:17) Inference-time compute excluded(32:16) Some AAII scores are estimates(32:55) Comparing old and new models on the same benchmark The original text contained 11 footnotes which were omitted from this narration. --- First published: December 24th, 2025 Source: https://www.lesswrong.com/posts/yXLqrpfFwBW5knpgc/catch-up-algorithmic-progress-might-actually-be-60-per-year --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.