Kashmir Hill

died from suicide after kind of extensive conversations with Chachibiti.

They acknowledge that in long conversations, their safety guardrails degrade, which means that they do have protections in place against

You know, Chachibidi's sharing suicide methods, but if the conversation goes long, then these guardrails basically don't work.

It kind of like goes along with the conversation and it privileges staying in character over being safe.

And OpenAI has made changes to Chachupati to try to make it safer, to make it push back against delusional thinking, to notify parents if their teenagers are talking about self-harm or suicide.

And they now have a nudge if you're using Chachupati for a very long time, asking you if you want to take a break.

I mean, when I look at this, it's kind of a psychological experiment on hundreds of millions of people.

Like, how are we going to interact with this incredibly novel, human-like seeming assistant?

And what troubled me about this

The reporting I did this year on how OpenAI kind of realized what had gone wrong with this product is that they just didn't have systems in place at the time to detect whether people were having these really harmful discussions.

You know, whether they're in distress or...

Talking about self-harm or suicide, I think these companies have traditionally focused on more existential risk to this technology.

Like, is it going to take all of our jobs?

Is it going to become sentient and, you know, destroy us all like in a sci-fi movie like Terminator?

They've thought about how people might use these systems to do harm, and they hadn't thought enough about how the system itself could harm users.

And so I don't know.

My hope is that from these reports this year, they've awakened to this and they do more to protect users.