Marc Brooker

how are you supposed to identify what to fix, right?

You can come up with some theories about those, but they're probably not gonna be right.

And again, I don't think there's a huge amount of value in,

the rote ticket closing work of on-call.

I think automation should be doing those kinds of work, but I think there's fantastic value in deep understanding, deep investigations, and deep reflection on what you learn from postmortems and COEs.

I tried to estimate a couple of months ago for a talk how many industry postmortems and Amazon COEs I'd read over my career

The best estimate I could come up to, and this was about a year ago, was between 3,000 and 4,000.

And so, you know, even a little bit of lesson from each one, and it tends to stick.

So I think what makes a really great post-mortem is first really getting into the details and making sure that you deeply understand what happened rather than just assuming what happened based on the biases you bring in.

And so there's a kind of lesson one there is if you can't understand what happened, well, that teaches you something about your logging and metrics and observability and simulations and all of these other things.

And then once you deeply understand what happened, then the ability, then a great post-mortem steps through the whys behind that at multiple levels, right?

Like why, well, yeah, there was a code bug.

Okay, sure, code bugs, yes, we can fix that.

But we can't stop there, right?

Like why was that missed in testing and validation, right?

you know, for these reasons, you know, what can we improve?

What can we build around those?

Okay, next step, you know, why, you know, why was our testing invalidation where it was?

Or, you know, why did we assume a certain thing about the behavior of the system that we wouldn't have assumed before?

And so as you sort of get through these deeper and deeper layers,