“I’m confused by the change in the METR trend” by Expertium

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded04 Mar 2026

Publisherfeeds.type3.audio ↗

Handlerec:16325067

Show notes

Measuring AI Ability to Complete Long Tasks - METR In their original 2025 paper, METR noticed that the slope (aka task horizon doubling time) of the trendline for models released in 2024 and later is different from the slope for <2024 models. First, I decided to check whether a piecewise linear function fits the data better than a simple linear function. If it doesn't, then this change in trend is a random fluke and there is nothing worth talking about. Here is the data so far (SOTA models only): The piecewise linear function clearly provides a better fit, based on quantitative metrics such as root mean square error, mean absolute error, R^2 and most importantly Bayesian information criteria (BIC); and based on a qualitative "bro just look at it" assessment. I added RMSE, MAE and R² as extra information, keep in mind that since a piecewise linear function has more parameters, it's not surprising that it fits data better than linear, this is to be expected. BIC penalizes model complexity (number of degrees of freedom), so it's more relevant for this case. However, I'm not satisfied. Let's randomly remove 20% of datapoints from this graph, do it a few [...] --- First published: March 3rd, 2026 Source: https://www.lesswrong.com/posts/9ihaaXqdc3i328tLp/i-m-confused-by-the-change-in-the-metr-trend --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.