Steven Byrnes

I'm part of an even more pessimistic group, motto, if anyone builds it, everyone dies, which generally does expect ruthless sociopath ASI as the default path we're on, absent a technical breakthrough, along with other miracles.

We tend to think 50% chance that humans will survive continued AI development is deliriously over-optimistic.

Anyway, the extra heap of concern in that latter camp is not from the MLLMs of today causing near certain doom, or even the somewhat better MLLMs of tomorrow, but rather the wildly better ASIs of maybe soon, maybe not, who knows.

But even if it's close in calendar time, and even if it comes out of MLLM research, such an ASI would still be systematically different from MLLMs as we know them today.

Optimist, aka, you have no evidence.

Me, no evidence either way, at least no evidence of that type.

Anyway, as I was saying, ASI would be systematically different from today's LLMs because, ahem, where do I start?

Actually, it would be much easier for me to explain if we start with the ASI threat model that I spend all my time on, and then we can circle back to Svel LMs afterwards.

Is that okay?

Heading.

Positive argument that brain-like RL agent ASI would be a ruthless sociopath.

Optimist?

Sure.

We can pause the discussion of LLMs for a few minutes and start in your comfort zone of actor-critic model-based RL agent brain-like ASI.

Doesn't really matter anyway.

Regardless of the exact algorithm, you clearly need some positive reason to believe that this kind of ASI would be a ruthless sociopath.

You can't just stomp your feet and declare that your weird unprecedented sci-fi belief is the default and push the burden of proof onto people who disagree with you.

Me?

Okay.

Maybe a good starting point would be my post's lacunes a path towards autonomous machine intelligence has an unsolved technical alignment problem or the era of experience has an unsolved technical alignment problem.