Robert M

To the extent that the survey says anything interesting, it says that coherence as understood by the survey takers is unrelated to the ability of various agents to cause harm to other agents.

Heading.

Blog.

First of all, the blog post seems to be substantially the output of an LLM.

In context, this is not that surprising, but it is annoying to read, and I also think this might have contributed to some of the more significant exaggerations or unjustified inferences.

Let me quibble with a couple sections.

First, why should we expect incoherence, LLMs as dynamical systems?

Quote

A key conceptual point, Bell LMs are dynamical systems, not optimizers.

When a language model generates text or takes actions, it traces trajectories through a high-dimensional state space.

It has to be trained to act as an optimizer and trained to align with human intent.

It's unclear which of these properties will be more robust as we scale.

Constraining a generic dynamical system to act as a coherent optimizer is extremely difficult.

End quote.

The paper has a similar section, with an even zanier claim.

Quote.

The set of dynamical systems that act as optimizers of a fixed loss is measure zero in the space of all dynamical systems.

End quote.

This seems to me like a vacuous attempt at defining away the possibility of building superintelligence, or perhaps coherent optimizers.

I will spend no effort on its refutation, clawed for 0.5 opus being capable of doing a credible job.