John Gallagher

It's nowhere near enough. Not in my experience. If somebody wants to refute me and come on this podcast or have a chat with me after, I would love to listen to how metrics allow you to debug very, very deliberately and get the exact data that you need.

It's nowhere near enough. Not in my experience. If somebody wants to refute me and come on this podcast or have a chat with me after, I would love to listen to how metrics allow you to debug very, very deliberately and get the exact data that you need.

You can send off dimensions to metrics and then your metrics bill explodes within about five seconds, especially if it's high cardinality data like IP addresses. I've made that mistake before. We're going to send a dimension of IP with our metrics so that we can understand what's going on. In a week, my manager usually messages me, usually in less than a week, saying, can you turn that off?

You can send off dimensions to metrics and then your metrics bill explodes within about five seconds, especially if it's high cardinality data like IP addresses. I've made that mistake before. We're going to send a dimension of IP with our metrics so that we can understand what's going on. In a week, my manager usually messages me, usually in less than a week, saying, can you turn that off?

We just got a day's dog bill of like five grand. Whoopsies.

We just got a day's dog bill of like five grand. Whoopsies.

So a few things there. Number one, you bring up a really good exception I'd forgotten to mention conveniently. If it's infrastructure stuff, if it's like memory, hard disk space, all that kind of stuff, fair game for metrics. Fine. The second thing is I'm quite hyperbolic. So I'm quite an extreme person. So when I say they're useless, I don't mean literally they're completely useless.

So a few things there. Number one, you bring up a really good exception I'd forgotten to mention conveniently. If it's infrastructure stuff, if it's like memory, hard disk space, all that kind of stuff, fair game for metrics. Fine. The second thing is I'm quite hyperbolic. So I'm quite an extreme person. So when I say they're useless, I don't mean literally they're completely useless.

I think of metrics as a hint. Hey, there's something going on over here. Cool, that's not useless. Obviously, it's useful. But then the next question is why? And if you've got a super simple system, then it's probably like three things. And you go, well, there's only three jobs in the system. So cool. And maybe you've segregated your metrics by background jobs, which is fair.

I think of metrics as a hint. Hey, there's something going on over here. Cool, that's not useless. Obviously, it's useful. But then the next question is why? And if you've got a super simple system, then it's probably like three things. And you go, well, there's only three jobs in the system. So cool. And maybe you've segregated your metrics by background jobs, which is fair.

You know, it gives you a place to look. It gives you a starting point. But I've, yeah, yeah. They're useful in the aggregate and they're useful at giving you a hint. And yes, they're useful in terms of like making sure the infrastructure is still running. But I see a lot of people depending on them. And I, you know, there's a guy I really respect, used to work with him called Lewis Jones.

You know, it gives you a place to look. It gives you a starting point. But I've, yeah, yeah. They're useful in the aggregate and they're useful at giving you a hint. And yes, they're useful in terms of like making sure the infrastructure is still running. But I see a lot of people depending on them. And I, you know, there's a guy I really respect, used to work with him called Lewis Jones.

And him and I have gone back and forth on this over LinkedIn. And he is convinced I'm wrong about this. He's like, we run everything through metrics. Metrics are awesome. You're just on cloud nine if you think you can trace everything. And there's also a significant weakness with tracing as well, which is you can't trace everything unless you've got relatively low throughput.

And him and I have gone back and forth on this over LinkedIn. And he is convinced I'm wrong about this. He's like, we run everything through metrics. Metrics are awesome. You're just on cloud nine if you think you can trace everything. And there's also a significant weakness with tracing as well, which is you can't trace everything unless you've got relatively low throughput.

or even medium throughput, you can make it work. If you trace every single request and you're doing millions of requests a day, I dread to think what your bill is going to be. So, and then that's where head tracing and head sampling and tail sampling comes into it. And we can get into that if you would like.

or even medium throughput, you can make it work. If you trace every single request and you're doing millions of requests a day, I dread to think what your bill is going to be. So, and then that's where head tracing and head sampling and tail sampling comes into it. And we can get into that if you would like.

Totally. So I should say I am singing the praises of tracing. but it's a slightly utopian vision that I'm painting because 90% of the work I've done is with logging purely because it's simple to get going. It's more of a known quantity.

Totally. So I should say I am singing the praises of tracing. but it's a slightly utopian vision that I'm painting because 90% of the work I've done is with logging purely because it's simple to get going. It's more of a known quantity.

And a lot of my talks, this is why I'm not talking a lot about tracing and I'm talking about structured logging because I think structured logging gives you this kind of event-based mindset that you can then start extending to tracing

And a lot of my talks, this is why I'm not talking a lot about tracing and I'm talking about structured logging because I think structured logging gives you this kind of event-based mindset that you can then start extending to tracing