Martin Kleppmann

It was a Rails app with a Postgres database, basically.

and some Redis and some similar things like that mixed in.

So actually, you know, nothing particularly revolutionary.

We essentially built a graph database on top of Postgres, so there was a little bit of technical interest in there, but, you know, nothing particularly outrageous.

After our team got disbanded, I switched over to the stream processing team.

So Kafka had just been developed at LinkedIn and had just been open sourced at the time.

Yeah, they developed it, right?

Oh, it was just being open sourced.

Yeah, I think it had just been open sourced.

And then I got to work on SAMSA, which was a stream processing framework on top of Kafka.

Yes.

So I think Jay Kreps has a pretty good blog post from that era called The Log, where he explains his motivation behind Kafka and why make it an append-only log rather than like a traditional message queue or something like that.

I think the motivation was really about data integration, because there were a whole bunch of databases and event generating systems, like activity events from users, for example.

They were all generating data in a sort of stream shape

And then a bunch of downstream systems that wanted to consume this, like wanted to get it into the data warehouse and wanted to be able to get it into the Hadoop cluster at the time in order to run like machine learning and things over it.

And there was just this data integration problem of actually like, how do you physically get the data out of one system and into another?

And Jay designed Kafka as this integration point, essentially like the

almost a kind of lowest common denominator, but still a general purpose abstraction for integrating various data sources and to downstream data sinks.

That's right, yes, because like previously the biggest company I had worked in was Reported with five people.

We had a