Reinventing Kafka on object storage
Ryan Worl, Co-founder and CTO at WarpStream, joins us to talk about the world of Kafka and data streaming and how WarpStream redesigned the idea of Kafka to run in modern cloud environments directly on top of object storage. Last year they posted a blog titled, âKafka is dead, long live Kafkaâ that hit the top of Hacker News to put WarpStream on the map. We get the backstory on Kafka and why itâs so widely used, who created it and for what purpose, and the behind the scenes on all things WarpStream.
Matched from the episode's transcript đ
Ryan Worl: Thatâs a really good question, and to take a step back from that question for a second, talking about the commercial open source stuff⌠This is obviously a little bit inside baseball, but as a part of going through that decision process, we talked to the founders of a lot of commercial open source companies, and we asked them âLetâs say you were starting our company today. What would you do?â And without hesitation, the answer we got was âI would not start it as a commercial open source company today.â
[01:07:53.19] And there are a lot of different reasons that they gave for that, and I canât really give some of those reasons without potentially identifying who those people are, and I donât want to do that⌠But the challenges of an commercial open source company today, with the â itâs not even just the hyperscaler cloud providers anymore taking your stuff and running it. Thatâs obviously a concern, but you can get around that with â like, the AGPL does a decent job of preventing some flavors of that.
The other issue is just like the competition within the category that theyâre building their product in is extremely high, and having your source code out there in the wild, and letting everybody know your secrets, essentially, about how you made your product better - you lose a lot of the juice behind why you have these huge staffs of developers working on interesting things. Itâs not to say you canât protect that other ways either, with software patents and stuff like that, but people donât â the appetite for software patents⌠It would do a lot of brand reputation, I think, if these commercial open source companies created a bunch of software patents and started enforcing them against each other, for example. Itâs a very challenging situation today.
A lot of the companies that you might view as successful commercial open source projects, they might be successful in the iteration that they exist in today, or yesterday in the case of all these real licenses, where they have good adoption in the developer community, and they might have good success in the VC-funded startup segment of the world⌠But there is an inevitable push to go upmarket, and to go after larger and larger customers, because itâs effectively the only way to support growth. The growth of what you can achieve within the small â if your customers are all small startups, even medium-sized startups, and developers playing around in their personal capacity or stuff like that, the revenue opportunity is just really small, unfortunately, for a lot of these businesses. Itâs much easier to sell a million dollar a year contract to an enterprise than it is to get a million dollars of revenue out of a bunch of small and medium-sized businesses.
So the temptation when the growth starts to slow down is âI need to go do that now.â Like, thatâs the first thing your investors are going to tell you, is you need to go upmarket and get enterprise customers. If the product that youâre selling them is support or a couple of features on top of an open source project, your ability to exert pricing pressure on that enterprise buyer, to get them to pay a higher price, or to get them to pay at all in the case of a lot of these open source projects, where they spent so much time making it good that the enterprise can just hire one person to maintain it internally, and just move on with their life and run the open source forever, and maybe pay you a peanuts support contract, essentially; not actually enough to support the business. Itâs just really hard.
I completely understand where youâre coming from, and it mightâve felt as if these companies were successful from the outside, and some of them definitely were⌠But just, there is that inevitable pressure to keep the growth rate up, and the only way to do that is to go upmarket. And when youâre going upmarket, you need to provide something that looks valuable. And if your project is open source and the alternative is hiring a developer to maintain it internally, you kind of have a cap on how much you can charge.
Itâs the same thing if youâre offering a cloud version of an open source project, for example. The premium someone will pay for your cloud version - it may be lower than you expect if they can self-host, because theyâre always looking at that⌠Theyâre looking at both sides of the coin. âHow much will it cost me to self-host this, versus how much does it cost to use your cloud-hosted version?â And that calculus does not always come out in your favor as a vendor. You may have to charge significantly more to make the numbers work on your side than what they think they can run it for internally. Itâs really challenging stuff.
[01:12:18.08] We wanted to provide the best product possible, with the best product experience possible, and we didnât feel like the shape of an open source commercial company was the right way to do it without having a lot of these distractions about the things that Iâm talking about right now come up along the way. And we didnât feel like it would be right to do that, the bait and switch thing that people are doing these days. We wanted to be honest, basically, from day one.