Steven Byrnes

So that's consequentialism, one possible answer for how an AI might accomplish impressive feats, and it's an answer that brings in ruthlessness by default.

And then there's a second, different possible answer to how an AI might accomplish impressive feats.

Imitative learning from humans.

You train an AI to predict what actions a skilled human would take in many different contexts and then have the AI take that same action itself.

I claim that LLMs get their impressive capabilities almost entirely from imitative learning.

By contrast, true imitative learning is entirely absent and impossible in humans and animals.

Imitative learning AIs do not have ruthless sociopathy by default because of course the thing they're imitating is non-ruthless humans.

Optimist.

Who?

Wait.

So you're an optimist about superintelligence, ASI, being non-ruthless, as long as people stick to LLMs?

Me?

Alas, no.

I think that the full power of consequentialism is super dangerous by default, and I think that the full power of consequentialism is the only way to get ASI, and so AI researchers are going to keep working until they eventually learn to fully tap that power.

In other words, I see a disjunction.

Either, LLMs will always get their powers primarily from imitative learning, as I claim they do today, in which case they will never be able to figure things out way beyond the human-created training data, and will thus never reach ASI.

And then eventually we'll get ASI via a different AI paradigm, one that can rocket arbitrarily far past any human data.

And that paradigm will have to draw its powers from consequentialism, which brings in ruthlessness by default.

OR, someone will figure out how to get LLMs themselves to rocket arbitrarily far past human training data and into ASI.

But the only way to do that is to somehow modify LLMs to draw on the full powers of consequentialism.