LessWrong (Curated & Popular)

“‘The Solomonoff Prior is Malign’ is a special case of a simpler argument” by David Matolcsi

25 Nov 2024

Audio

Description

[Warning: This post is probably only worth reading if you already have opinions on the Solomonoff induction being malign, or at least heard of the concept and want to understand it better.] IntroductionI recently reread the classic argument from Paul Christiano about the Solomonoff prior being malign, and Mark Xu's write-up on it. I believe that the part of the argument about the Solomonoff induction is not particularly load-bearing, and can be replaced by a more general argument that I think is easier to understand. So I will present the general argument first, and only explain in the last section how the Solomonoff prior can come into the picture.I don't claim that anything I write here is particularly new, I think you can piece together this picture from various scattered comments on the topic, but I think it's good to have it written up in one place. [...] ---Outline:(00:17) Introduction(00:56) How an Oracle gets manipulated(05:25) What went wrong?(05:28) The AI had different probability estimates than the humans for anthropic reasons(07:01) The AI was thinking in terms of probabilities and not expected values(08:40) Probabilities are cursed in general, only expected values are real(09:19) What about me?(13:00) Should this change any of my actions?(16:25) How does the Solomonoff prior come into the picture?(20:10) ConclusionThe original text contained 14 footnotes which were omitted from this narration. --- First published: November 17th, 2024 Source: https://www.lesswrong.com/posts/KSdqxrrEootGSpKKE/the-solomonoff-prior-is-malign-is-a-special-case-of-a --- Narrated by TYPE III AUDIO.

Transcription

This episode hasn't been transcribed yet

Help us prioritize this episode for transcription by upvoting it.

0 upvotes

🗳️ Sign in to Upvote

Popular episodes get transcribed faster

Other episodes from LessWrong (Curated & Popular)

Transcribed and ready to explore now

"Ada Palmer: Inventing the Renaissance" by Martin Sustrik

28 Jan 2026

LessWrong (Curated & Popular)

"The inaugural Redwood Research podcast" by Buck, ryan_greenblatt

27 Jan 2026

LessWrong (Curated & Popular)

"Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German" by jenn

26 Jan 2026

LessWrong (Curated & Popular)

[Linkpost] "“The first two weeks are the hardest”: my first digital declutter" by mingyuan

20 Jan 2026

LessWrong (Curated & Popular)

"Why we are excited about confession!" by boazbarak, Gabriel Wu, Manas Joglekar

19 Jan 2026

LessWrong (Curated & Popular)

"Backyard cat fight shows Schelling points preexist language" by jchan

16 Jan 2026

LessWrong (Curated & Popular)

View all episodes from LessWrong (Curated & Popular)

Comments

There are no comments yet.

Please log in to write the first comment.

Report any issue

LessWrong (Curated & Popular)

“‘The Solomonoff Prior is Malign’ is a special case of a simpler argument” by David Matolcsi

This episode hasn't been transcribed yet

Other episodes from LessWrong (Curated & Popular)

"Ada Palmer: Inventing the Renaissance" by Martin Sustrik

"The inaugural Redwood Research podcast" by Buck, ryan_greenblatt

"Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German" by jenn

[Linkpost] "“The first two weeks are the hardest”: my first digital declutter" by mingyuan

"Why we are excited about confession!" by boazbarak, Gabriel Wu, Manas Joglekar

"Backyard cat fight shows Schelling points preexist language" by jchan

Sign in to Audioscrape

Share this moment