Steven Byrnes

In which case, again, we get ruthlessness by default.

There's an image here.

For what it's worth, I happen to expect that ASI will come from the former, future paradigm shift, rather than the latter, LLM modifications.

But it hardly matters in this context.

Optimist.

I dunno, if you're willing to concede that LLMs today are not maximally ruthless, well, LLMs today don't seem that far from superintelligence.

I mean, humans don't rocket arbitrarily far past any training data, either.

They usually do things that have been done before, or at most, for world experts on the bleeding edge, go just one little step beyond it.

LLMs can do both, right?

Me?

Yes, but humans collectively and over time can get way, way, way beyond our training data.

We're still using the same brain design that we were using in Pleistocene Africa.

Between then and now, there were no angels who dropped training data from the heavens, but humans nevertheless invented language, science, technology, industry, culture, and everything else in the $100T global economy entirely from scratch.

We did it all by ourselves, by our own bootstraps, ultimately via the power of consequentialism, as implemented in the RL and model-based planning algorithms in our brains.

See sharp left-turned discourse.

An opinionated review.

By the same token, if humanity survives another 1,000 years, we will invent wildly new scientific paradigms, build wildly new industries and ways of thinking, etc.

There's a quadrillion-dollar market for AIs that can likewise do that kind of thing, as humans can.

If the LLMs of today don't pass that bar, and they don't, then I expect that, sooner or later, either someone will figure out how to get LLMs to pass that bar, or else someone will invent a new non-LLM AI paradigm that passes that bar.

Either way, imitative learning is out, consequentialism is in, and we get ruthless sociopath ASIs by default in the absence of yet-to-be-invented theoretical advances in technical alignment.