Paul Frazee

like an event processing architecture in a way and like normally you would use like a kafka or something like that to like send all these things through that's what this thing is that's what this firehose is all of our application is downstream of that just running computed views off of that firehose so whenever you're tapping into it you're just jumping straight into our data center to tap into the data set so that's really fun it's it's amazing yeah like a meaningful thing not just like a replication of all the activity but like that's it that's the network

like an event processing architecture in a way and like normally you would use like a kafka or something like that to like send all these things through that's what this thing is that's what this firehose is all of our application is downstream of that just running computed views off of that firehose so whenever you're tapping into it you're just jumping straight into our data center to tap into the data set so that's really fun it's it's amazing yeah like a meaningful thing not just like a replication of all the activity but like that's it that's the network

Yeah. There's a... There are going to be some interesting learnings about the economics of this over time, to be sure. So I don't want to be walking around as if that's not a big reality about all this. That said, it is a lot.

Yeah. There's a... There are going to be some interesting learnings about the economics of this over time, to be sure. So I don't want to be walking around as if that's not a big reality about all this. That said, it is a lot.

There is a version of the Firehose that we offer that is a kind of a reduced form that drops all of the cryptographic proofs to make life a little bit easier and allows you to do filters on it and things like that. But one thing to also remember is that the Firehose is kind of a convenience service, to be honest. So if you, like I said before, it works kind of like the web.

There is a version of the Firehose that we offer that is a kind of a reduced form that drops all of the cryptographic proofs to make life a little bit easier and allows you to do filters on it and things like that. But one thing to also remember is that the Firehose is kind of a convenience service, to be honest. So if you, like I said before, it works kind of like the web.

You broadcast or publish these records, and then those flow into the applications to be processed into the views. The Relay is kind of like a crawling bot that just helps out. It just looks around at all the different repos that are in the world and kind of goes ahead and grabs their replication streams and puts them into one stream for you.

You broadcast or publish these records, and then those flow into the applications to be processed into the views. The Relay is kind of like a crawling bot that just helps out. It just looks around at all the different repos that are in the world and kind of goes ahead and grabs their replication streams and puts them into one stream for you.

If we decide at some point what we can't afford, like all the consumers on this thing, you can run your own relay and directly do those crawls. So again, we're trying to make sure that there is openness to the system so that if there gets to a point where we're like, you know what, we can't afford passing this along to everybody, somebody else can do it. You can directly do it yourself.

If we decide at some point what we can't afford, like all the consumers on this thing, you can run your own relay and directly do those crawls. So again, we're trying to make sure that there is openness to the system so that if there gets to a point where we're like, you know what, we can't afford passing this along to everybody, somebody else can do it. You can directly do it yourself.

Yeah, I am really excited to see what people end up doing with that. I'm really excited for its actual computing and industry potential as well. Again, it does form the basis for building applications beyond just the kind of Twitter style of application. So I'm pretty excited about that.

Yeah, I am really excited to see what people end up doing with that. I'm really excited for its actual computing and industry potential as well. Again, it does form the basis for building applications beyond just the kind of Twitter style of application. So I'm pretty excited about that.

You just kind of nailed what was on my mind as you were talking about us scaling and managing to keep things online. Software has gotten better. The resources available have gotten better. The patterns have been really ironed out. And in fact, Martin Kleppman, who actually helped us with that paper quite a bit, really helped drive that paper. He's been consulting with us from the beginning.

You just kind of nailed what was on my mind as you were talking about us scaling and managing to keep things online. Software has gotten better. The resources available have gotten better. The patterns have been really ironed out. And in fact, Martin Kleppman, who actually helped us with that paper quite a bit, really helped drive that paper. He's been consulting with us from the beginning.

And that was a huge asset, an incredible win. I was... when we first got him i was over the moon um that's the kind of you know he wrote a book that's sort of famous for helping engineers understand how to build these kinds of applications these kind of high-scale data heavy applications and so we were able to really apply

And that was a huge asset, an incredible win. I was... when we first got him i was over the moon um that's the kind of you know he wrote a book that's sort of famous for helping engineers understand how to build these kinds of applications these kind of high-scale data heavy applications and so we were able to really apply

just image everything the industry had learned about how to do this sort of stuff both to like our internal systems and also just to the protocol since scaling was always a big part of what we were trying to do we wanted this to be able to be at the scale that you expect out of these kinds of applications so that i think you know that plus just a really great team that you know did some great work to keep those servers running throughout all that growth that's that's how this

just image everything the industry had learned about how to do this sort of stuff both to like our internal systems and also just to the protocol since scaling was always a big part of what we were trying to do we wanted this to be able to be at the scale that you expect out of these kinds of applications so that i think you know that plus just a really great team that you know did some great work to keep those servers running throughout all that growth that's that's how this

Totally, 100%. Yeah, just everything, every moment. No, I mean, I feel really pretty fortunate across the board. And honestly, when we were starting out the project, we were originally just a protocol consultancy for Twitter, which is a complicated and really interesting history.

Totally, 100%. Yeah, just everything, every moment. No, I mean, I feel really pretty fortunate across the board. And honestly, when we were starting out the project, we were originally just a protocol consultancy for Twitter, which is a complicated and really interesting history.