John Gallagher

Fantastic question. So, and this gets to the root of why the three pillars are complete nonsense.

Fantastic question. So, and this gets to the root of why the three pillars are complete nonsense.

Metrics, traces, and logs.

Metrics, traces, and logs.

Nonsense. They're not three pillars. The analogy I like to use is saying that observability is three pillars and it's traces, logs, and metrics is a bit like saying programming is three pillars. It's arrays, integers, and strings. It's the same kind of deal. No, it's nothing to do with those things. Well, it is because you use those every day. Yes, but you're kind of missing the point.

Nonsense. They're not three pillars. The analogy I like to use is saying that observability is three pillars and it's traces, logs, and metrics is a bit like saying programming is three pillars. It's arrays, integers, and strings. It's the same kind of deal. No, it's nothing to do with those things. Well, it is because you use those every day. Yes, but you're kind of missing the point.

So thanks to some amazing work by people at Honeycomb and charity majors and reading their stuff and reading their incredible work, I've realized that metrics, tracing, and logs are missing the point. The point is we want to see events that happened at some point in time. And that neatly answers your question about how do you reconstruct the state of the app.

So thanks to some amazing work by people at Honeycomb and charity majors and reading their stuff and reading their incredible work, I've realized that metrics, tracing, and logs are missing the point. The point is we want to see events that happened at some point in time. And that neatly answers your question about how do you reconstruct the state of the app.

I mean, the short answer is, of course, you can't. If you're not in an event-driven system, if you're in a CRUD app, if you're storing state to the database, there is no way you can go back in time and accurately recreate it. But we can give it a reasonably good stab.

I mean, the short answer is, of course, you can't. If you're not in an event-driven system, if you're in a CRUD app, if you're storing state to the database, there is no way you can go back in time and accurately recreate it. But we can give it a reasonably good stab.

And we can do this by capturing the state of each event when it was, forget about observability tools and logging and structured logging and tracing just for now. Imagine if when that incident happened, let's say my expired token would be maybe potentially a good example. There are several points in that timeline that we want to understand. Number one, when the token was created.

And we can do this by capturing the state of each event when it was, forget about observability tools and logging and structured logging and tracing just for now. Imagine if when that incident happened, let's say my expired token would be maybe potentially a good example. There are several points in that timeline that we want to understand. Number one, when the token was created.

Number two, when the user hit the website. And maybe there's a third one, when the account was created, let's say that. So imagine if at each of those three points, we had a rich event with everything related to that event in it. So when the account was created,

Number two, when the user hit the website. And maybe there's a third one, when the account was created, let's say that. So imagine if at each of those three points, we had a rich event with everything related to that event in it. So when the account was created,

we had the account ID, the status of the account, whether it's pending or not, the creation date, the customer, the customer ID, blah, blah, blah, blah, blah. And then when the user visited the site, what was the request? What was the request ID? What was the user ID? What was the anonymous user ID? Et cetera, et cetera. And then when the token was created, what was the expiry? What was the this?

we had the account ID, the status of the account, whether it's pending or not, the creation date, the customer, the customer ID, blah, blah, blah, blah, blah. And then when the user visited the site, what was the request? What was the request ID? What was the user ID? What was the anonymous user ID? Et cetera, et cetera. And then when the token was created, what was the expiry? What was the this?

What was the that? What was the user ID? Okay. So if we have those three events, and we have enough rich data gathered with each of the events, we can answer your question. Does that make sense so far? There's a whole load of more blah, blah, blah, but does that make sense so far?

What was the that? What was the user ID? Okay. So if we have those three events, and we have enough rich data gathered with each of the events, we can answer your question. Does that make sense so far? There's a whole load of more blah, blah, blah, but does that make sense so far?

And also other events that are happening in the system. So there's user did something, computer did something, computer enqueued a background job, performed a job, et cetera, et cetera. So the way I think about it is everything that happens in your app, whether it's initialized by the computer, an external data source, a user, it's basic event storming stuff really. That creates an event.

And also other events that are happening in the system. So there's user did something, computer did something, computer enqueued a background job, performed a job, et cetera, et cetera. So the way I think about it is everything that happens in your app, whether it's initialized by the computer, an external data source, a user, it's basic event storming stuff really. That creates an event.