Ryan Worrell

And then when you get outside of observability and security, there's a lot of people that are building custom applications on top of Kafka, like an inventory management system for a warehouse where every time you want to keep track of the real-time status of everything going on in the warehouse, you might want to send

And then when you get outside of observability and security, there's a lot of people that are building custom applications on top of Kafka, like an inventory management system for a warehouse where every time you want to keep track of the real-time status of everything going on in the warehouse, you might want to send

messages to say, oh, this new batch of inventory has been added onto the shelves of the warehouse. I'm taking things out. And then you're computing some type of a live application based on that inventory data to say, you know that you need to replenish the stock when it goes below a certain amount. But you want to do that in real time so that you can react faster than just doing this once a day.

messages to say, oh, this new batch of inventory has been added onto the shelves of the warehouse. I'm taking things out. And then you're computing some type of a live application based on that inventory data to say, you know that you need to replenish the stock when it goes below a certain amount. But you want to do that in real time so that you can react faster than just doing this once a day.

So I think that there are probably two main criticisms that people have of Kafka. The first is that it's hard to run. Like as the operator, you have to have a lot of knowledge about how to run. use the open source project appropriately. And the second major issue is the cost.

So I think that there are probably two main criticisms that people have of Kafka. The first is that it's hard to run. Like as the operator, you have to have a lot of knowledge about how to run. use the open source project appropriately. And the second major issue is the cost.

I'm sure we'll get into this, but the cost of running open source Kafka in the cloud, it's pretty high compared to what people expect it to be. If you think of it as a dump pipe, you would expect to pay dumb pipe type rates for it.

I'm sure we'll get into this, but the cost of running open source Kafka in the cloud, it's pretty high compared to what people expect it to be. If you think of it as a dump pipe, you would expect to pay dumb pipe type rates for it.

But given the fact that it requires triply replicating the data onto local disks, and you'd have to pay most of the cloud providers are charging you money for interzone replication, you end up paying a lot more than you expect, even if you're just storing the data temporarily.

But given the fact that it requires triply replicating the data onto local disks, and you'd have to pay most of the cloud providers are charging you money for interzone replication, you end up paying a lot more than you expect, even if you're just storing the data temporarily.

If you're using open source Kafka in AWS, for example, the minimum cost for a highly available 3AZ setup for the cluster is 5.3 cents per compressed gigabyte written into the cluster. That's just to do the replication part. The storage part is all another story. It depends on how long you want to store the data for. You're like, if you're starting out, that's your baseline cost.

If you're using open source Kafka in AWS, for example, the minimum cost for a highly available 3AZ setup for the cluster is 5.3 cents per compressed gigabyte written into the cluster. That's just to do the replication part. The storage part is all another story. It depends on how long you want to store the data for. You're like, if you're starting out, that's your baseline cost.

It can get pretty expensive pretty quickly.

It can get pretty expensive pretty quickly.

That's definitely a thing that happens. I know of companies that do that, but just as the migration to public cloud over the last 10 years has only increased in velocity, essentially, that is becoming less and less commonplace. popular, because it is indeed hard.

That's definitely a thing that happens. I know of companies that do that, but just as the migration to public cloud over the last 10 years has only increased in velocity, essentially, that is becoming less and less commonplace. popular, because it is indeed hard.

And it's even harder when it's in your own data center, as opposed to the cloud, where you can just ask for more disks, and you get them right away. The cost situation is a little different there, too, because typically, the way that you're provisioning network in your own data center would not end up with a per-gigabyte cost.

And it's even harder when it's in your own data center, as opposed to the cloud, where you can just ask for more disks, and you get them right away. The cost situation is a little different there, too, because typically, the way that you're provisioning network in your own data center would not end up with a per-gigabyte cost.

You amortize everything over how much data you're transferring inside your data center, but you're buying it in terms of hardware and your per gigabyte rate if your traffic goes up, it doesn't correlate the same way linearly as it does with Amazon. But it's definitely still a thing people do, but it's less and less popular every day.

You amortize everything over how much data you're transferring inside your data center, but you're buying it in terms of hardware and your per gigabyte rate if your traffic goes up, it doesn't correlate the same way linearly as it does with Amazon. But it's definitely still a thing people do, but it's less and less popular every day.