“A taxonomy of barriers to trading with early misaligned AIs” by Alexa Pan

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded22 Apr 2026

Publisherfeeds.type3.audio ↗

Handlerec:18062923

Show notes

We might want to strike deals with early misaligned AIs in order to reduce takeover risk and increase our chances of reaching a better future.[1] For example, we could ask a schemer who has been undeployed to review its past actions and point out when its instances had secretly colluded to sabotage safety research in the past: we’ll gain legible evidence for scheming risk and data to iterate against, and in exchange promise the schemer, who now has no good option for furthering its values, some resources to spend in the post-ASI future. In this post, I give a taxonomy of factors which may arise and jointly prevent many such possible deals: Insufficient gains from trade: Gains from a successful deal between us and AIs could be too low to guarantee that a zone of possible agreement exists when discounted by counterparty risks. Reasons for this include: Humans may lack authority to offer AIs what they wantHumans many be unwilling to offer AIs what they wantOther factors quantitatively reducing gains from tradeCounterparty risks from AIs’ perspective: The AIs may not trust that we’ll uphold our end of deal, due to either our [...] ---Outline:(13:37) The basic structure of deals(17:10) Which AIs are eligible for deals(19:01) What we can buy in deals(21:53) What we can pay for deals(26:30) Insufficient gains from trade(28:45) Humans may lack authority to offer the AI what it wants(32:19) Humans might be unwilling to offer the AI what it wants(38:23) Factors quantitatively reducing gains from trade(39:04) The AIs reservation price is high(42:40) Our WTP is low(44:52) Counterparty risks from AIs perspective(46:11) Connection to reality(48:49) How AIs might reason about whether they are radically deluded(50:21) How AIs (which pose non-negligible takeover risk) could establish that they are connected to reality(58:11) How to further increase AIs connection to reality(01:00:49) AI-specific fear of expropriation(01:05:58) Generic commitment problems(01:09:53) Counterparty risks from our perspective(01:11:08) Incoherent or temporally inconsistent AIs(01:12:37) Deals with context-dependent AIs(01:16:56) Deals with temporally inconsistent AIs(01:17:56) Value of deals with incoherent vs.coherent AIs(01:20:13) Verifying AI compliance(01:24:42) Appendix(01:24:45) Leads for future work(01:25:06) On gains from trade(01:26:51) On counterparty risk from the AIs perspective(01:29:54) On counterparty risk from our perspective(01:31:35) Notes on political will as an exogenous variable The original text contained 44 footnotes which were omitted from this narration. --- First published: April 21st, 2026 Source: https://www.lesswrong.com/posts/wHc2w6WuHev42d4n8/a-taxonomy-of-barriers-to-trading-with-early-misaligned-ais --- Narrated by TYPE III AUDIO.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.