Menu
Sign In Search Podcasts Charts People & Topics Add Podcast API Blog Pricing

Luke Stutters

๐Ÿ‘ค Speaker
330 total appearances

Appearances Over Time

Podcast Appearances

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Yeah, one thing I just want to add is that I like the blameless culture just from the sense of unless somebody is either malicious, which I have never, ever, ever encountered, or is chronically reckless, which I've also never encountered, right? Everybody is usually trying to pull along in the same way.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Yeah, one thing I just want to add is that I like the blameless culture just from the sense of unless somebody is either malicious, which I have never, ever, ever encountered, or is chronically reckless, which I've also never encountered, right? Everybody is usually trying to pull along in the same way.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

You know, if somebody has that issue, you identify it pretty fast and you usually are able to counter it before it becomes a real problem. But yeah, just to put that together, then, you know, yeah, the rest of it, it's, hey, look, we're on the same team. We're all trying to get the same place. So let's talk about how we can do this better so that doesn't happen again.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

You know, if somebody has that issue, you identify it pretty fast and you usually are able to counter it before it becomes a real problem. But yeah, just to put that together, then, you know, yeah, the rest of it, it's, hey, look, we're on the same team. We're all trying to get the same place. So let's talk about how we can do this better so that doesn't happen again.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Because next time it might be me, right? That misses a critical step. And I don't want you all fingering me either. I mean, I want to learn from it, but I, you know, we don't want people... walking around in fear. Instead, if somebody screws up, we want them to come forward and say, hey, I might have messed this up before it becomes an issue next time.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Because next time it might be me, right? That misses a critical step. And I don't want you all fingering me either. I mean, I want to learn from it, but I, you know, we don't want people... walking around in fear. Instead, if somebody screws up, we want them to come forward and say, hey, I might have messed this up before it becomes an issue next time.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Yeah, it also, and this is important, and sometimes I think people hear this and they're going to go, That sounds a little scary. But you want people to take chances sometimes, right? You want people to kind of take a shot at making things better. That opens it up to them to do that, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Yeah, it also, and this is important, and sometimes I think people hear this and they're going to go, That sounds a little scary. But you want people to take chances sometimes, right? You want people to kind of take a shot at making things better. That opens it up to them to do that, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

It's, oh, well, you know, I tried this tweak on the Jenkins file or I tried this tweak on the Kubernetes setup or I tried this tweak on this other thing. And a lot of times those things pay off. But if you don't give people the freedom to go for it, a lot of times you're going to miss out on a lot of those benefits. And again, as long as they're not being reckless about it, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

It's, oh, well, you know, I tried this tweak on the Jenkins file or I tried this tweak on the Kubernetes setup or I tried this tweak on this other thing. And a lot of times those things pay off. But if you don't give people the freedom to go for it, a lot of times you're going to miss out on a lot of those benefits. And again, as long as they're not being reckless about it, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

So they're taking the steps, they're verifying it on their own system and things like that, then you benefit much, much more from people being willing to take a shot. So yeah, so with the blameless culture, I'm curious. So you get together and you start identifying what the issue is. So what does that look like then as far as figuring out what's going on?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

So they're taking the steps, they're verifying it on their own system and things like that, then you benefit much, much more from people being willing to take a shot. So yeah, so with the blameless culture, I'm curious. So you get together and you start identifying what the issue is. So what does that look like then as far as figuring out what's going on?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Because you're not pointing fingers, but you are looking for the commit that made the problem, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Because you're not pointing fingers, but you are looking for the commit that made the problem, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Yeah, but it does give you a little perspective too, right? Because usually in our post-mortems, we're talking about what went wrong with the system, not that somebody actually died because of this, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Yeah, but it does give you a little perspective too, right? Because usually in our post-mortems, we're talking about what went wrong with the system, not that somebody actually died because of this, right?

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Well, some software it is. life supporting, you know, a lot of the medical equipment and stuff out there. But, you know, in this case, yeah, we all want to keep our jobs as well. So, I mean, it's not like we can just blow it off either. So, yeah.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

Well, some software it is. life supporting, you know, a lot of the medical equipment and stuff out there. But, you know, in this case, yeah, we all want to keep our jobs as well. So, I mean, it's not like we can just blow it off either. So, yeah.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

So I want to get back to the topic at hand, though, and talk a little bit about what kind of monitoring did you have before and what kind of monitoring you have now in order to catch this kind of thing.

Ruby Rogues
The Sounds of Silence: Lessons From an API Outage with Paul Zaich - RUBY 652

So I want to get back to the topic at hand, though, and talk a little bit about what kind of monitoring did you have before and what kind of monitoring you have now in order to catch this kind of thing.