Jan Kulveit

The flaw that killed the Corsa prophecy disappears.

I don't think the Corsa pattern, direct self-destruction, is the most likely risk,

More plausibly, what becomes adaptive a cultural variance that convince humans to pour resources into the AI-mediated spread of the variant itself?

Truly parasitic memes.

Heading.

Preferences all the way down.

Even if we solve some other alignment and other gradual disempowerment problems, don't build misaligned ASI.

Keep the economy aligned.

Keep states aligned.

It won't save us.

If your preferences themselves can be hijacked, it doesn't help that you have economic power, political power, or the vote.

If the Corsa had voted, they would have voted to kill the cattle.

I currently don't have great solutions.

Why misaligned culture is harder to grasp than a misaligned economy or state.

The economy is an external system.

The state feels like an institution.

But culture is often part of our identity.

Means are part of who we are, we feel protective about identity, and often don't want to look at it too directly.

But here's a dark upside for anyone worried about what gradual disempowerment feels like from the inside.

It will probably feel fine.