Paul Frazee

So you had like key management problems and you would also have, uh, it was all local first, all these projects. So like you would have device synchronicity and like, um, essentially like you'd have to get them like CRDT territory to just make a post. It felt like rocket science to try to implement a content section. Yeah. It was, it was very complex software that we were messing with. Yeah.

Well, like the... One that we knew for sure was what we call account portability. And then that one comes to mind first because it has to do with like why you bother with peer-to-peer in the first place.

You're trying to turn the services that you use into something that is actually like, oh God, I'm going to say it, fungible, that you could actually replace a hosting service or like an application service, but keep all your data. And we knew that was one of the really interesting properties of peer-to-peer because like, you know, their client side, right?

So like your storage and your signing keys were being kept on your device. And so the first thing we sort of figured out was, okay, how can you take advantage of using a server, but like maintain that ability to like adversarially move away from a server. And that's what the entire cop portability system is for.

And that was a pretty big move forward because using a lot of the same techniques, but putting it on the server and not losing those properties was like the, kind of the biggest like scale and ease of use unlock that we had throughout the whole system compared to our previous work. We knew we wanted to keep that.

The other thing that we wanted to keep for sure was what is now starting to get called a shared heap model, this notion of an open broadcast of all of the public data so they can be repurposed, because that was a pretty big design element that we had landed on in the peer-to-peer stuff that we'd been doing about how to build these large-scale applications, is this notion of...

It works actually quite a bit like the web where it's almost like everybody's publishing JSON of like posts and their profile and their likes and stuff like that. And then you just like aggregate them in the applications. And that's how you build out an application.

We knew we wanted to keep that because that's what makes it possible to build other applications without some kind of a hard binding to the hosting. So you're really separating the hosting and the application layer. And that maintains that hackability and like capacity to repurpose these applications in interesting ways. Those are probably the two that come to mind.

I don't know the exact timeline. I know it got a lot more turned off recently. Yeah. More turned off.

It's super cool, yeah. And that's also what a lot of people have started to play with first, because it's just so obviously cool. Like, okay, what can I do with this? It's cool in two fronts, actually, because actually, it's really like you're just getting into the innards of our data center, because this is the backbone. You normally...

like an event processing architecture in a way and like normally you would use like a kafka or something like that to like send all these things through that's what this thing is that's what this firehose is all of our application is downstream of that just running computed views off of that firehose so whenever you're tapping into it you're just jumping straight into our data center to tap into the data set so that's really fun it's it's amazing yeah like a meaningful thing not just like a replication of all the activity but like that's it that's the network

Yeah. There's a... There are going to be some interesting learnings about the economics of this over time, to be sure. So I don't want to be walking around as if that's not a big reality about all this. That said, it is a lot.

There is a version of the Firehose that we offer that is a kind of a reduced form that drops all of the cryptographic proofs to make life a little bit easier and allows you to do filters on it and things like that. But one thing to also remember is that the Firehose is kind of a convenience service, to be honest. So if you, like I said before, it works kind of like the web.

You broadcast or publish these records, and then those flow into the applications to be processed into the views. The Relay is kind of like a crawling bot that just helps out. It just looks around at all the different repos that are in the world and kind of goes ahead and grabs their replication streams and puts them into one stream for you.

If we decide at some point what we can't afford, like all the consumers on this thing, you can run your own relay and directly do those crawls. So again, we're trying to make sure that there is openness to the system so that if there gets to a point where we're like, you know what, we can't afford passing this along to everybody, somebody else can do it. You can directly do it yourself.

Yeah, I am really excited to see what people end up doing with that. I'm really excited for its actual computing and industry potential as well. Again, it does form the basis for building applications beyond just the kind of Twitter style of application. So I'm pretty excited about that.

You just kind of nailed what was on my mind as you were talking about us scaling and managing to keep things online. Software has gotten better. The resources available have gotten better. The patterns have been really ironed out. And in fact, Martin Kleppman, who actually helped us with that paper quite a bit, really helped drive that paper. He's been consulting with us from the beginning.

And that was a huge asset, an incredible win. I was... when we first got him i was over the moon um that's the kind of you know he wrote a book that's sort of famous for helping engineers understand how to build these kinds of applications these kind of high-scale data heavy applications and so we were able to really apply

just image everything the industry had learned about how to do this sort of stuff both to like our internal systems and also just to the protocol since scaling was always a big part of what we were trying to do we wanted this to be able to be at the scale that you expect out of these kinds of applications so that i think you know that plus just a really great team that you know did some great work to keep those servers running throughout all that growth that's that's how this

Totally, 100%. Yeah, just everything, every moment. No, I mean, I feel really pretty fortunate across the board. And honestly, when we were starting out the project, we were originally just a protocol consultancy for Twitter, which is a complicated and really interesting history.