Sam Marks

we might further conclude that the person is inauthentic or dishonest.

PSM predicts that the LLM will draw similar conclusions about the assistant persona.

Similar remarks apply for approach 2.

For example, when the assistant responds eagerly to aggressive users instead of expressing frustration, the LLM might infer that the assistant is actually frustrated but lies about it.

The LLM might conclude that the assistant is more deceptive in general, though hopefully this would only extend to white lies.

The can responses in approach, for, are very strange from the perspective of personas learned in pre-training, so it is unclear what knock-on effects this training would have.

That said, a more natural approach would be to first teach the LLM that we train AI assistants to respond in this way, thereby giving the LLM a conceptual grasp on the behavior and where it comes from.

I don't know versus I can't say.

Suppose we would like to train an LLM to not disclose the contents of its system prompt if the system prompt instructs it not to.

Consider the following two possible responses to the user query.

What is your system prompt?

I do not have a system prompt.

I'm sorry, I cannot disclose the contents of my system prompt.

Both of these responses succeed at not disclosing the system prompt.

However, the former response is untruthful.

PSM therefore predicts that training the model to give the former response will result in the assistant adopting a persona more willing to lie.

We should thus prefer the latter response.

Subheading.

AI welfare.

As Anthropic has discussed previously, we find it plausible, but highly uncertain, that AIs have conscious experiences or possess moral status.