Dario Amodei

If we do get a century of medical progress in a few years, it is possible that we will greatly increase the human lifespan, and there is a chance we also gain radical capabilities like the ability to increase human intelligence or radically modify human biology.

Those would be big changes in what is possible, happening very quickly.

They could be positive if responsibly done, which is my hope, as described in machines of loving grace, but there is always a risk they go very wrong, for example, if efforts to make humans smarter also make them more unstable or power-seeking.

There is also the issue of faploads or whole-brain emulation, digital human minds instantiated in software, which might someday help humanity transcend its physical limitations, but which also carry risks I find disquieting.

AI changes human life in an unhealthy way.

A world with billions of intelligences that are much smarter than humans at everything is going to be a very weird world to live in.

Even if AI doesn't actively aim to attack humans, Section 1, and isn't explicitly used for oppression or control by states, Section 3, there is a lot that could go wrong short of this, via normal business incentives and nominally consensual transactions.

We see early hints of this in the concerns about AI psychosis, AI driving people to suicide, and concerns about romantic relationships with AIs.

As an example, could powerful AIs invent some new religion and convert millions of people to it?

Could most people end up addicted in some way to AI interactions?

Could people end up being puppeted by AI systems, where an AI essentially watches their every move and tells them exactly what to do and say at all times, leading to a good life but one that lacks freedom or any pride of accomplishment?

It would not be hard to generate dozens of these scenarios if I sat down with the creator of Black Mirror and tried to brainstorm them.

I think this points to the importance of things like improving Claude's constitution over and above what is necessary for preventing the issues in Section 1.

Making sure that AI models really have their users' long-term interests at heart in a way thoughtful people would endorse rather than in some subtly distorted way seems critical.

Human purpose.

This is related to the previous point, but it's not so much about specific human interactions with AI systems as it is about how human life changes in general in a world with powerful AI.

Will humans be able to find purpose and meaning in such a world?

I think this is a matter of attitude.

As I said in Machines of Loving Grace, I think human purpose does not depend on being the best in the world at something, and humans can find purpose even over very long periods of time through stories and projects that they love.

We simply need to break the link between the generation of economic value and self-worth and meaning.