John Gallagher

So essentially, it's this big tree structure. And you might have seen this before. It's the flame graph that you get in Datadog and New Relic and all these kind of things. And everybody looks at these things and thinks they're really pretty. And they are. Indeed, they are. So that's the pinnacle of observability in my head. Traces give it us all. And we can say,

So essentially, it's this big tree structure. And you might have seen this before. It's the flame graph that you get in Datadog and New Relic and all these kind of things. And everybody looks at these things and thinks they're really pretty. And they are. Indeed, they are. So that's the pinnacle of observability in my head. Traces give it us all. And we can say,

as you can do in any of these observability tools that support tracing, you can do some really cool stuff. Show me all the requests that were a 200 that enqueued a job where the job lasted for more than three seconds. Holy cow, now we're cooking with gas. We've got everything that we need. Show me all the spans that indicated anything to do with the background job.

as you can do in any of these observability tools that support tracing, you can do some really cool stuff. Show me all the requests that were a 200 that enqueued a job where the job lasted for more than three seconds. Holy cow, now we're cooking with gas. We've got everything that we need. Show me all the spans that indicated anything to do with the background job.

where it was a 500 response, but the user was logged in, and, and, and, and. And so we can start to not only query the spans, but query the parents of the spans. So you've got all of these nested causal relationships, and it gets ridiculously powerful. So that's traces. Cool. Let's look at logs. What do logs give us? Well, it gives us events. That's all logs are, really.

where it was a 500 response, but the user was logged in, and, and, and, and. And so we can start to not only query the spans, but query the parents of the spans. So you've got all of these nested causal relationships, and it gets ridiculously powerful. So that's traces. Cool. Let's look at logs. What do logs give us? Well, it gives us events. That's all logs are, really.

It's a series of events that happen. Does it give us the ability to nest events inside one another? Nope. Sorry. Your luck's out. You can log causation IDs and you can link them together. And obviously you can log request IDs and filter everything by the request ID. But there's no concept in the log of this log is nested inside this other log. So that information, goodbye. It's gone.

It's a series of events that happen. Does it give us the ability to nest events inside one another? Nope. Sorry. Your luck's out. You can log causation IDs and you can link them together. And obviously you can log request IDs and filter everything by the request ID. But there's no concept in the log of this log is nested inside this other log. So that information, goodbye. It's gone.

Don't have it. But you have the rich data in every event. Let's look at metrics. What does metrics give you? It doesn't give you the events. It doesn't give you the nesting. And it just gives you some aggregated numbers. So I don't think of them as three pillars. They're three rungs of a ladder. The very top rung is tracing. Awesome. The next rung down is logs. Pretty good.

Don't have it. But you have the rich data in every event. Let's look at metrics. What does metrics give you? It doesn't give you the events. It doesn't give you the nesting. And it just gives you some aggregated numbers. So I don't think of them as three pillars. They're three rungs of a ladder. The very top rung is tracing. Awesome. The next rung down is logs. Pretty good.

And metrics are useless. Now, when I say metrics are useless, people get upset with me and say, oh, well, I look at metrics all the time to understand my app. Yeah. Okay. But if you derive metrics from higher rungs, that's totally cool. Totally fine. But what's a really bad idea is to...

And metrics are useless. Now, when I say metrics are useless, people get upset with me and say, oh, well, I look at metrics all the time to understand my app. Yeah. Okay. But if you derive metrics from higher rungs, that's totally cool. Totally fine. But what's a really bad idea is to...

directly say i'm going to send this metric right now to my backend and people do this all the time people think this is a good idea it's okay i mean it's better than nothing right it's it's just depends on the fidelity of information you want but the problem is there's two problems actually but the main one is you've sent that data okay you sent it to prometheus datadog whatever you sent that one data point

directly say i'm going to send this metric right now to my backend and people do this all the time people think this is a good idea it's okay i mean it's better than nothing right it's it's just depends on the fidelity of information you want but the problem is there's two problems actually but the main one is you've sent that data okay you sent it to prometheus datadog whatever you sent that one data point

So then you look in the metrics and you say, holy cow, we're getting all these 500s. Why is that? I'll sit here and wait as long as you want. You're not going to be able to tell me the answer to the question unless it's blindingly obvious, unless you can say, oh, well, this other bit of data over here is like correlates with it time-wise and maybe it might be that. Yeah, okay, it might be that.

So then you look in the metrics and you say, holy cow, we're getting all these 500s. Why is that? I'll sit here and wait as long as you want. You're not going to be able to tell me the answer to the question unless it's blindingly obvious, unless you can say, oh, well, this other bit of data over here is like correlates with it time-wise and maybe it might be that. Yeah, okay, it might be that.

How do you know it's that? Well, we're having to guess. Guessing is not a strategy. Hope is not a strategy. I don't really want to debug by just flipping. Guessing, I want to know. And the only way of knowing is having traces. So the way I like to think of it is tracing is the pinnacle. Logs can be derived from traces, which is why the three rungs of ladder.

How do you know it's that? Well, we're having to guess. Guessing is not a strategy. Hope is not a strategy. I don't really want to debug by just flipping. Guessing, I want to know. And the only way of knowing is having traces. So the way I like to think of it is tracing is the pinnacle. Logs can be derived from traces, which is why the three rungs of ladder.

And everything can be derived as a metric from the two rungs above. So if you've got only logs, you don't have any nested context. But you can get metrics from logs. Fine. If you just have metrics, I would say you're not in great shape because you can't understand why without pure guessing. And it amazes me how many people push back on this idea and think just having some metrics is enough.

And everything can be derived as a metric from the two rungs above. So if you've got only logs, you don't have any nested context. But you can get metrics from logs. Fine. If you just have metrics, I would say you're not in great shape because you can't understand why without pure guessing. And it amazes me how many people push back on this idea and think just having some metrics is enough.