Jesse Spivak

We've got tons of really cool problems, different spaces, and people are almost encouraged to move around to find things that they like and are good at. So she made this move, and now the problem of using a framework that none of us had experience with and that the general company didn't have a ton of experience with really became self-evident.

We've got tons of really cool problems, different spaces, and people are almost encouraged to move around to find things that they like and are good at. So she made this move, and now the problem of using a framework that none of us had experience with and that the general company didn't have a ton of experience with really became self-evident.

Because now there was still work left to do on the microservice. And I'm just a Rails developer. I had to go into there and start writing Kotlin, start reading the ACA documentation and try to wrap my head around what this whole actor system meant. And it was tough. And that's why I call it a big mistake. Sorry, Dave.

Because now there was still work left to do on the microservice. And I'm just a Rails developer. I had to go into there and start writing Kotlin, start reading the ACA documentation and try to wrap my head around what this whole actor system meant. And it was tough. And that's why I call it a big mistake. Sorry, Dave.

Absolutely, Dave. And we were originally in the monolith, the way the system worked. was it ran on a background scheduled job using rescue and the scheduled job basically took roughly 45 minutes to run. So it was running like kind of like a waterfall, like every 10 minutes we'd start a new job that would take 45 minutes to run.

Absolutely, Dave. And we were originally in the monolith, the way the system worked. was it ran on a background scheduled job using rescue and the scheduled job basically took roughly 45 minutes to run. So it was running like kind of like a waterfall, like every 10 minutes we'd start a new job that would take 45 minutes to run.

And so going from that to basically completely real time is a huge improvement. And if real time for 500,000 events per day versus 5 million events per day are different things. But if you're at real time, it's already this just enormous improvement over what we were working with, which is like this 45 minute loop.

And so going from that to basically completely real time is a huge improvement. And if real time for 500,000 events per day versus 5 million events per day are different things. But if you're at real time, it's already this just enormous improvement over what we were working with, which is like this 45 minute loop.

So at that point, and I guess I'll say another thing before I get into how we fixed my mistake. This is actually getting to what Dave said. This is a case of premature optimization. We sort of didn't do the back of the envelope math

So at that point, and I guess I'll say another thing before I get into how we fixed my mistake. This is actually getting to what Dave said. This is a case of premature optimization. We sort of didn't do the back of the envelope math

well enough or didn't have a clear picture of like okay this is really like designed to handle like literally a thousand times more traffic than our best day so you know we we didn't that was that was definitely a mistake like we should have asked that question and realized okay maybe Maybe this is a little too much horsepower. We don't need this.

well enough or didn't have a clear picture of like okay this is really like designed to handle like literally a thousand times more traffic than our best day so you know we we didn't that was that was definitely a mistake like we should have asked that question and realized okay maybe Maybe this is a little too much horsepower. We don't need this.

It's too much trouble for what it's, for what it's buying us. And on my end also, I was imagining, you know, getting all these events we call it. So the microservice was producing like prediction events. Okay. So predicting every time a piece of content should come down and making an adjustment about when it thinks that that should happen.

It's too much trouble for what it's, for what it's buying us. And on my end also, I was imagining, you know, getting all these events we call it. So the microservice was producing like prediction events. Okay. So predicting every time a piece of content should come down and making an adjustment about when it thinks that that should happen.

And so I'm imagining the monolith is listening to these prediction events, subscribe to an SNS topic or SQSQ. And I imagine thousands and thousands and thousands of events every second, which is way, way too much. And so I was thinking of all these clever ways to do caching and to try to figure out when can I drop events so I don't need to hit the database?

And so I'm imagining the monolith is listening to these prediction events, subscribe to an SNS topic or SQSQ. And I imagine thousands and thousands and thousands of events every second, which is way, way too much. And so I was thinking of all these clever ways to do caching and to try to figure out when can I drop events so I don't need to hit the database?

How can I come up with these clever ways to only make round trips to the database when I really need to?

How can I come up with these clever ways to only make round trips to the database when I really need to?

And that added time to the development, and it also added a ton of complexity so that when our two systems, when we were like, okay, let's turn them on, let's see what happens, the first error that comes up, because of course there's going to be an error when you first try it out, that was really hard to debug. It was really hard to understand, like, is this a caching issue?

And that added time to the development, and it also added a ton of complexity so that when our two systems, when we were like, okay, let's turn them on, let's see what happens, the first error that comes up, because of course there's going to be an error when you first try it out, that was really hard to debug. It was really hard to understand, like, is this a caching issue?