Nilay Patel

They have experiences.

You can Google them.

And then sometimes they die, right?

And they just go away.

You get some new ones.

And at least the system replenishes and evolves.

An AI system running on a cloud service that you can't see on a data center that you might have hated being put in your community does not feel accountable to you in that way.

And maybe it's getting it right more often on whatever metric that someone who is not you has decided is important.

But you can't actually hold it accountable.

And that to me feels like the biggest gap in all of these automated decision systems that no one wants to account for because the efficiency gain is so high.

My wife, as it happens, is a divorce lawyer, and I do think an agent of her just talking her clients through the decisions that were made a long time ago over and over again would actually be very helpful for her in many ways, because that seems like a lot of her job.

I understand what you're saying there.

You said this thing to me the first time we met that, again, I've just been thinking about ever since.

You said that several years from now, 10, 20, 30 years from now, we would think it was crazy that we ever had human judges making as many decisions as we do today.

You've got one case in the system.

Is that borne out?

Do you feel that as strongly as you did when we first talked a few months ago?

One thing I've been thinking about throughout this conversation is like who gets access to these systems?

Who gets to make it feel fair?

Where does the trust in the agency come from?