LessWrong (Curated & Popular)

"Decision theory does not imply that we get to have nice things" by So8res

08 Nov 2022

Audio

Description

https://www.lesswrong.com/posts/rP66bz34crvDudzcJ/decision-theory-does-not-imply-that-we-get-to-have-niceCrossposted from the AI Alignment Forum. May contain more technical jargon than usual.(Note: I wrote this with editing help from Rob and Eliezer. Eliezer's responsible for a few of the paragraphs.)A common confusion I see in the tiny fragment of the world that knows about logical decision theory (FDT/UDT/etc.), is that people think LDT agents are genial and friendly for each other.[1]One recent example is Will Eden’s tweet about how maybe a molecular paperclip/squiggle maximizer would leave humanity a few stars/galaxies/whatever on game-theoretic grounds. (And that's just one example; I hear this suggestion bandied around pretty often.)I'm pretty confident that this view is wrong (alas), and based on a misunderstanding of LDT. I shall now attempt to clear up that confusion.To begin, a parable: the entity Omicron (Omega's little sister) fills box A with $1M and box B with $1k, and puts them both in front of an LDT agent saying "You may choose to take either one or both, and know that I have already chosen whether to fill the first box". The LDT agent takes both."What?" cries the CDT agent. "I thought LDT agents one-box!"LDT agents don't cooperate because they like cooperating. They don't one-box because the name of the action starts with an 'o'. They maximize utility, using counterfactuals that assert that the world they are already in (and the observations they have already seen) can (in the right circumstances) depend (in a relevant way) on what they are later going to do.A paperclipper cooperates with other LDT agents on a one-shot prisoner's dilemma because they get more paperclips that way. Not because it has a primitive property of cooperativeness-with-similar-beings. It needs to get the more paperclips.

Transcription

This episode hasn't been transcribed yet

Help us prioritize this episode for transcription by upvoting it.

0 upvotes

🗳️ Sign in to Upvote

Popular episodes get transcribed faster

Other episodes from LessWrong (Curated & Popular)

Transcribed and ready to explore now

"Daycare illnesses" by Nina Panickssery

13 Apr 2026

LessWrong (Curated & Popular)

"If Mythos actually made Anthropic employees 4x more productive, I would radically shorten my timelines" by ryan_greenblatt

12 Apr 2026

LessWrong (Curated & Popular)

"Do not be surprised if LessWrong gets hacked" by RobertM

09 Apr 2026

LessWrong (Curated & Popular)

"My picture of the present in AI" by ryan_greenblatt

09 Apr 2026

LessWrong (Curated & Popular)

LessWrong (Curated & Popular)

"Decision theory does not imply that we get to have nice things" by So8res

This episode hasn't been transcribed yet

Other episodes from LessWrong (Curated & Popular)

"Daycare illnesses" by Nina Panickssery

"If Mythos actually made Anthropic employees 4x more productive, I would radically shorten my timelines" by ryan_greenblatt

"Do not be surprised if LessWrong gets hacked" by RobertM

"My picture of the present in AI" by ryan_greenblatt

"The effects of caffeine consumption do not decay with a ~5 hour half-life" by kman

"Anthropic’s Pause is the Most Expensive Alarm in Corporate History" by Ruby

Sign in to Audioscrape

Share this moment