“Quantitative AI risk assessment: a starting point” by Henry Papadatos, jakub_krys, malcolmmurray, Renn Karageorgieva

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded27 May 2026

Publisherfeeds.type3.audio ↗

Handlerec:18604047

Show notes

Current AI risk management relies on qualitative approaches, much like nuclear safety before 1975. We propose a shift to quantitative risk modeling, following the approach that transformed nuclear safety. We propose a methodology and demonstrate it by building nine probabilistic models of AI-enabled cyber attacks. This is a first attempt at AI risk quantification. We invite criticism and hope this can be a starting point for the kind of iterative improvement that made nuclear safety robust. Introduction — why quantitative risk modeling matters In 1975, the Nuclear Regulatory Commission (NRC) published WASH-1400, the first systematic application of probabilistic methods to nuclear reactor safety. Before this, nuclear reactor safety was grounded in a conservative, qualitative approach. The argument was that consequences of even the worst accident were effectively zero because physical barriers would prevent any radioactive release. But in 1966, experts recognised that a core meltdown in large reactors could breach those barriers. Regulators now had to argue that severe accidents were possible but sufficiently improbable. That required specific probability estimates. The shift to quantitative methods was not straightforward. When WASH-1400 began, one technical advisor wondered, "Do we dare undertake such a study till we really know how?" The 1975 [...] ---Outline:(00:46) Introduction -- why quantitative risk modeling matters(04:51) Our methodology(13:15) Initial insights from quantitative AI risk modeling(13:58) 1. Risk increases as AI capabilities increase. So does uncertainty.(15:15) 2. Different benchmarks capture risk differently.(16:24) 3. Attack bottlenecks change as capabilities change.(18:15) 4. Factors driving risk from current AI systems are different from factors that may drive risk in the future.(19:23) Open problems(22:09) What's next for SaferAI The original text contained 2 footnotes which were omitted from this narration. --- First published: May 27th, 2026 Source: https://www.lesswrong.com/posts/CpmpyS9Rpz7obejaw/quantitative-ai-risk-assessment-a-starting-point-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.