John Gallagher

We're being paid a lot of money and it's not working. The app that we've delivered is not working. What's more, we don't know why. But also I do just want to add, and this may broaden out the conversation a little bit. You may want to, we may want to keep it narrow on Rails apps, but I've realized that observability principles go way beyond how does our web app work? It applies to any black box.

We're being paid a lot of money and it's not working. The app that we've delivered is not working. What's more, we don't know why. But also I do just want to add, and this may broaden out the conversation a little bit. You may want to, we may want to keep it narrow on Rails apps, but I've realized that observability principles go way beyond how does our web app work? It applies to any black box.

So as an example, a few years ago, I was working at a company and their SEO wasn't great. And they just kind of were like, oh, you know, we'll try and fix it. And they had several attempts to fix it. None of them really worked. And every attempt was the same. They would get some expert in. The expert would give us a list of 100 things to do. We would do 80 of the 100.

So as an example, a few years ago, I was working at a company and their SEO wasn't great. And they just kind of were like, oh, you know, we'll try and fix it. And they had several attempts to fix it. None of them really worked. And every attempt was the same. They would get some expert in. The expert would give us a list of 100 things to do. We would do 80 of the 100.

And then nothing would really improve. And then they'd be like, well, we did everything you said. And then they'd move on to another. And rinse and repeat, keep doing that. And then one day, within four weeks, 20% of the site traffic disappeared. And nobody could tell us why. Nobody understood why. Observability. Now, Google is a black box.

And then nothing would really improve. And then they'd be like, well, we did everything you said. And then they'd move on to another. And rinse and repeat, keep doing that. And then one day, within four weeks, 20% of the site traffic disappeared. And nobody could tell us why. Nobody understood why. Observability. Now, Google is a black box.

So, you know, you're not going to be able to instrument Google. But there's lots of tools that allow you to peer into the inner workings of Google, SEMrush, Screaming Frog, all these kind of tools. They are, in my opinion, actually in, to some degree, the observability space. They're not... Everybody thinks of them as marketing tools, SERPs, engine optimization tools, whatever, whatever, whatever.

So, you know, you're not going to be able to instrument Google. But there's lots of tools that allow you to peer into the inner workings of Google, SEMrush, Screaming Frog, all these kind of tools. They are, in my opinion, actually in, to some degree, the observability space. They're not... Everybody thinks of them as marketing tools, SERPs, engine optimization tools, whatever, whatever, whatever.

They're allowing you to make reasoned guesses about why your searches aren't performing the way they are. And then you can actually take action on that because now you have some data. Oh, this keyword dropped from place four to place 100. Why is that?

They're allowing you to make reasoned guesses about why your searches aren't performing the way they are. And then you can actually take action on that because now you have some data. Oh, this keyword dropped from place four to place 100. Why is that?

okay let's try a let's try hypothesis a put that live and see if google will respond to that oh and now up to you know position 80 whatever it is so the idea of observability goes way way beyond like data dog and new relic and obviously all of those people in the observability space but i i see it as a much much wider and much more applicable topic yeah i i hear you there uh

okay let's try a let's try hypothesis a put that live and see if google will respond to that oh and now up to you know position 80 whatever it is so the idea of observability goes way way beyond like data dog and new relic and obviously all of those people in the observability space but i i see it as a much much wider and much more applicable topic yeah i i hear you there uh

Great question. I would say if you're making a small app with very little traffic and it's thresholds like anything else, you're making a small app with very little traffic. I have a client at the moment I'm consulting for. and I've made them an app, and it has maybe flipping 20 visits a day or something, 20 hits a day. So I installed Rollbar, free version of Rollbar.

Great question. I would say if you're making a small app with very little traffic and it's thresholds like anything else, you're making a small app with very little traffic. I have a client at the moment I'm consulting for. and I've made them an app, and it has maybe flipping 20 visits a day or something, 20 hits a day. So I installed Rollbar, free version of Rollbar.

Anything goes wrong, I get a notification. It's fine. The further up the stack you move, the more the defaults change. For a Rails app that's mission critical that I'm not even going to say mission critical, but just serving a decent number of hits a month, uh, 10,000, 20,000. I don't know. I've tried a lot of observability tools. Um, and there's no one that yet that I can unreservedly recommend.

Anything goes wrong, I get a notification. It's fine. The further up the stack you move, the more the defaults change. For a Rails app that's mission critical that I'm not even going to say mission critical, but just serving a decent number of hits a month, uh, 10,000, 20,000. I don't know. I've tried a lot of observability tools. Um, and there's no one that yet that I can unreservedly recommend.

They're all got their pros and cons. Um, Datadog is a good option if money is no object. I kind of don't want to get into the tooling debate because it's kind of a bit of a red herring, I think, in many ways. There's various cost-benefit trade-offs there. But in terms of the defaults, in terms of what you observe, requests has got to be up there.

They're all got their pros and cons. Um, Datadog is a good option if money is no object. I kind of don't want to get into the tooling debate because it's kind of a bit of a red herring, I think, in many ways. There's various cost-benefit trade-offs there. But in terms of the defaults, in terms of what you observe, requests has got to be up there.

So every app that I have in my care of any significant size, I would always say install semantic logger. Semantic logger is the best logger I've found. It does JSON out of the box. It's quite extensible. There are many problems with it, but it's the best option that we've got. So that's number one. That will log every, like Rails already logs every request for you.

So every app that I have in my care of any significant size, I would always say install semantic logger. Semantic logger is the best logger I've found. It does JSON out of the box. It's quite extensible. There are many problems with it, but it's the best option that we've got. So that's number one. That will log every, like Rails already logs every request for you.