Simon Willison

Oh, yeah, go for it.

I think we're due a challenger disaster with respect to coding agent security.

And this is based on this wonderful essay about the normalization of deviance.

Have you heard this phrase before?

Yes.

This idea, it came out of the 1986 challenge disaster reports where if you have a culture, a corporate culture or whatever, that keeps on getting away with, with, with doing something that they shouldn't have been doing.

And kicks and getting with those lapses, but the space, the space battle keeps on launching and it's fine.

Yeah.

That leads you into a sort of corporate culture level false sense of security, and it's going to burn you.

Because I think so many people, myself included, are running these coding agents practically as root, right?

We're letting them do all of this stuff.

And every time I do it, my computer doesn't get wiped.

I'm like, oh, it's fine.

And I just keep on going like that.

And I think it's going to add up.

I think, and I said this last year, I said last year, there's going to be a headline grabbing prompt injection security hold.

There was not.

I've been predicting this every six months the past two and a half years.

This is my version of that prediction this year.

I think we are due a challenger disaster scale thing caused by the fact that we all got away with these bad practices for so long and we got lazy.