Menu
Sign In Search Podcasts Charts People & Topics Add Podcast API Pricing

Ryan Worrell

๐Ÿ‘ค Speaker
404 total appearances

Appearances Over Time

Podcast Appearances

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

So the fact that everything is in object storage opens up a ton of new possibilities, actually. Basically, the only downside of WarpStream is the fact that the latency is higher. Now, obviously, we're a new company. The product does not have the 13-year maturity of the open-source Kafka project. But just to speak of the operational...

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

So the fact that everything is in object storage opens up a ton of new possibilities, actually. Basically, the only downside of WarpStream is the fact that the latency is higher. Now, obviously, we're a new company. The product does not have the 13-year maturity of the open-source Kafka project. But just to speak of the operational...

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

stuff and the cost stuff, the Workstream is a huge win on both of those.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

stuff and the cost stuff, the Workstream is a huge win on both of those.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

Yeah, so there are a number of projects and products out there that you can buy to give you an object storage interface in essentially any environment. Like there's the open source project, MinIO, and then basically every storage vendor on the market will sell you something with an S3 compatible interface if you're running in a data center environment.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

Yeah, so there are a number of projects and products out there that you can buy to give you an object storage interface in essentially any environment. Like there's the open source project, MinIO, and then basically every storage vendor on the market will sell you something with an S3 compatible interface if you're running in a data center environment.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

And because we work with S3, GCS, and Azure Blob Storage, we can essentially, you know, I shouldn't say connect to anything. If you had an NFS server, we can even make it work on that too. We don't have any production doing that, and I wouldn't recommend it. I would recommend using the object storage interfaces, but we're pretty flexible in terms of the deployment topology.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

And because we work with S3, GCS, and Azure Blob Storage, we can essentially, you know, I shouldn't say connect to anything. If you had an NFS server, we can even make it work on that too. We don't have any production doing that, and I wouldn't recommend it. I would recommend using the object storage interfaces, but we're pretty flexible in terms of the deployment topology.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

So I think it would depend on where you're running the compute. If you were storing the data in R2, but you were running compute in AWS, you would get charged a lot of internet transfer as part of that. If you're running your compute in one of the providers that has free peering with R2, then yeah, you would get a nice savings there and you're, you know,

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

So I think it would depend on where you're running the compute. If you were storing the data in R2, but you were running compute in AWS, you would get charged a lot of internet transfer as part of that. If you're running your compute in one of the providers that has free peering with R2, then yeah, you would get a nice savings there and you're, you know,

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

You'd be able to move data reliably across, let's say, multiple regions of whatever providers have peered for free with R2 using Workstream.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

You'd be able to move data reliably across, let's say, multiple regions of whatever providers have peered for free with R2 using Workstream.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

Yeah, I think the demo was Richie's idea. It basically just starts up a producer and a consumer so that you can just see something happening in the console. Like, yeah, it provides you a link. If you would have run that locally on your laptop, we would have opened the link automatically in your browser for you.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

Yeah, I think the demo was Richie's idea. It basically just starts up a producer and a consumer so that you can just see something happening in the console. Like, yeah, it provides you a link. If you would have run that locally on your laptop, we would have opened the link automatically in your browser for you.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

Yeah, so we even designed the little niceties like that. But the idea behind the demo is basically just to show people that it does something. Kafka is not an exciting technology to demo, so we're kind of limited there. It's even more boring than doing a demo for a relational database or something. But there is another mode that you can run that's called Playground.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

Yeah, so we even designed the little niceties like that. But the idea behind the demo is basically just to show people that it does something. Kafka is not an exciting technology to demo, so we're kind of limited there. It's even more boring than doing a demo for a relational database or something. But there is another mode that you can run that's called Playground.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

And Playground will let you start a cluster that doesn't have a fake producer and consumer running on it as a demo. It just starts a cluster for you temporarily and makes an account that expires in 24 hours. And you can take that Playground link and you can start...

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

And Playground will let you start a cluster that doesn't have a fake producer and consumer running on it as a demo. It just starts a cluster for you temporarily and makes an account that expires in 24 hours. And you can take that Playground link and you can start...

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

multiple nodes, like say one on my laptop and one on yours, and point it at R2, and we can have a cluster that spans our two laptops together. Like my co-founder and I did that before and posted a video of it on Twitter or something like that. But because the data is all in object storage and the compute part is stateless, it's actually, it's not that complicated to do.

The Changelog: Software Development, Open Source
Reinventing Kafka on object storage (Interview)

multiple nodes, like say one on my laptop and one on yours, and point it at R2, and we can have a cluster that spans our two laptops together. Like my co-founder and I did that before and posted a video of it on Twitter or something like that. But because the data is all in object storage and the compute part is stateless, it's actually, it's not that complicated to do.