Johnny Boursiquot: Well, I say strong because over the years my stance on that has changed. In my earlier days as an engineer I was basically always gung-ho for building, because I always found the problem spaces that I happened to be in to be interesting, and I always wanted to build stuff. And even if there was something out there, I’d look at it and I’m like “Ah, I can do better.” Kind of letting myself be carried away… You know, “Oh, Twitter - that’s just nothing but an ever-scrolling feed. You can refresh things in the background. That’s fine, you can build that in a weekend.”

Obviously, as time goes on and you grow as an engineer, you realize “Okay, so there’s more to a product than just the engineering of it.” There’s always more – a lot of things you don’t know you don’t know. So as you mature as an engineer, you sort of realize “Okay, there are always constraints, there are always trade-offs. What am I trading off?”

[16:06] So to your question, like “When is that threshold?”, for me, I’m gonna borrow a term from the cloud industry that basically says “As long as you have things that are considered undifferentiated, like commodity…” Like, I’m not gonna build a data center or invest in a data center to host my SaaS web app, or whatever. It just doesn’t make sense to do that. But if I’m in the business of actually building highly customized servers and hardware for our cloud-hosting businesses, then yes, maybe I do need my own data center to do these things, because that is in the line of business. That is the business you’re in.

As long as whatever it is you’re evaluating does not lead directly to revenue generation, or in some cases cost reduction for the service that you’re providing, then for me it’s a no-brainer. It’s not even worth considering. Because I can go find something out there that’s gonna do - perhaps not 100% of everything I need, but again, I’m using the 80/20 rule when I’m talking about this here, “What is the thing that gets me close enough?” And if I need to stitch together some extra things and whatever… Maybe there’s some things that it doesn’t do quite well, and maybe I wanna dedicate basically an extra 10% of engineering time and effort to get there… No solution is gonna be just right, and even when we build our own, we find out later on that “Oh, the requirements changed”, as they often do. “The criteria that we were using, that we decided to go build this thing on - they have changed a little bit.” Or the business doesn’t care about this as much, or we don’t need all these nines for reliability. Maybe we can get away with four nines for this thing, because our customers are “business hours only”.

There’s a lot of things you’re gonna factor into that decision, but I think for me, a really cut and dry rule, whatever it is that I’m thinking of investing in, “Does this contribute directly to the revenue, or to the cost reduction of the service I’m trying to render?”

Anurag Goel: Well, I think this is a large and important problem. And maybe I can give you some background on how I ended up doing what I do now. So before Render - I won’t go too far back, but I joined Stripe as the fifth engineer in 2011, and I left in 2016. And the company did well, and so I had this opportunity to really think hard about what I wanted to do for the rest of my career. I could choose to do nothing and sort of chill out… But my wife and I both, we talked about it, and I think we both saw it and see it as a responsibility… Very few people are granted this opportunity to really have the freedom to pursue any goal that they want, and so my goal then became to find a really big problem that I’m also very excited to solve, and that I am good at solving. And as part of that, I ended up building applications in a lot of different domains.

I mean, even at Stripe, I had seen our infrastructure team that was managing AWS grow from when I was there one person out of five was fully focused on AWS… That number, that percentage, about 20%, stayed about the same as Stripe continued to grow, at least when I was there, 2011 through 2016… And all these people were simply managing VMs on AWS. They weren’t contributing to the product. They weren’t helping the other teams necessarily move faster. So that was a big waste. And then when I built my own applications post-Stripe to try to figure out which domain I was going to be interested in over the long term, it hit home firsthand. It was very obvious to me that getting something up and running in the cloud was still very broken. Ultimately, you end up building out a Kubernetes cluster, and no one wants to do that. I’m sure there are some people in the world who love building out Kubernetes clusters, and bringing all the care and feeding what those clusters require over time, and spending all the money that Kubernetes ends up sucking out of your ops budget… But ultimately, what people want, what I wanted, was I want my application to – first of all, I want my cloud to allow me to spin up an application really quickly, and get it up and running, and make it reliable. But then I want it to scale, I want a lot of security by default around it, I want all the features that I will need as the application grows…

[00:08:22.03] And so the dichotomy between Heroku and AWS when I started Render was Heroku was easy to get started on, but it stopped scaling beyond a certain point. And if you needed more than 14 gigs of RAM, you couldn’t have it. You just couldn’t. Your applications were restarted every – and a lot of this is still true. Your applications were restarted every 24 hours… And then the price became unsustainable once you went beyond a certain scale.

And then AWS was AWS. Just even logging in and looking at the dropdown of the 300 plus services… They have a search in their dropdown. Who has a search in a dropdown? AWS does.

Alya Abbott: Yeah, absolutely. So we have intentionally not raised VC money, and do not plan to raise VC money. In terms of the business model, what we want is just to build a sustainable company on top of this open source project. So we’ve discussed some paid plans we have on the cloud side, on the self-hosted side, services we can provide… And so that’s really our strategy, to have our users pay for the software, and then that funds the development of the project and the product. And kind of a key reason we don’t want to go the VC route is that we feel that kind of misaligns the incentives. There’s a kind of inherent misalignment of incentives.

So for us, we’re not going to take 100 swings at this. We’re not going to try to build 100 different products and see which ones land, and abandon the ones that don’t. We really are building Zulip because we think it’s a better way to work, and we’re really, really committed to making that around for our users for the long term. So as I mentioned, we still have users from 2013 who are on Zulip now, and we want that software to be around for the long run.

And so we want to just take that one single bet and make it work. Whereas VCs, their incentives are – they’re looking for the next Facebook, your next giant company that just explodes, and they’re willing to take big risks in order to have that probability of a really remarkable, amazing return. Whereas for us, we want to take very small risks, and have a very high probability of success, without necessarily aiming for that galactic, outsized return. Our main priority is really to get to a point where we’re making enough money to really continue to develop the software, and have the staffing and the team that we want… And it doesn’t have to be stratospheric. And of course, we would like to reach as many people as we can, and we think it can benefit lots and lots of different kinds of organizations… It’s a huge market; there’s definitely tons of opportunity. But just the kinds of risks we’re comfortable taking to get there are very different from the kinds of risks VCs would feel comfortable with taking to get there.

Cameron Balahan: So yeah, Sameer sort of stole a bit of what I was gonna say, but the way I would have framed it is that I’m excited for Go’s growth. Programming language adoption is just this fundamentally slow thing. Nobody goes back and rewrites what already works, and we don’t want that either… And if you already have a ton of servers written in one language, you’ll probably make the next one written in that language as well… So introducing a programming language and then having it be adopted - it’s kind of a big deal, and it takes a long time. Just seeing Go’s sort of organic growth, especially in recent years, and - like I was saying, the community loves it, and it just keeps self-perpetuating… I think we’ve reached a threshold now where I see it growing even more rapidly, potentially, or at least finding it in more and more places. And I think part of that is maybe in a specific way related to the AI frontier, but it’s really just more generally like just new paradigms of computing; as those as those occur, those are opportunities where sort of new code is written. Maybe new companies come up and they have to choose a stack, and they’re probably going to use the cloud, and they’re probably going to be concerned about the productivity and how quickly they can get their product out, and they’re probably going to be concerned about the total cost of ownership in the long term, and the security and reliability of the products, and they’re going to find that Go is such a great fit for all those things, and the other languages that they maybe learned a while ago, or seen in production in an earlier time don’t fit quite as well. And that’s pretty exciting, because Go really does then fit the bill. And so I’m really excited to see what happens with that. The sort of growth, generally. So AI as a contributor to growth.

Then you asked about kind of a worry, or something like that… I’m maybe not worried, but just sort of wondering about AI. I wonder what this all looks like. It’s obviously very interesting, and I think that especially in the software development lifecycle, there’s opportunities to cut inefficiencies, and make work more productive… Sometimes though it seems like some of that is oversold, and I don’t know where the actual truth is, but I’m interested to see how that plays out. What the impact of having sort of more robust code completion will actually have on software engineering more generally.

So as Sameer said, we keep our tabs on that, because it’s important to understand the market in which we’re in, and what our users want. We want to be adaptable. We want to provide for all those things for our users that sort of keeps them happy, like the productivity and the other things. So I’d say growth generally, but wondering what AI will kind of make of our whole profession in the future.

Matthew Boyle: So I think [unintelligible 00:55:00.06] good or bad, but I guess just kind off a brief story of how we’ve been using it at CloudFlare, which has been interesting… So firstly, I did mention that we’ve been using Protobuf for our schemas. The support for Protobuf and gRPC in Go is excellent. It’s first-class. So that was a good fit and a good choice, and I would make that choice again. So for schema management, Protobuf’s definitely worth looking at, especially if you are a predominantly Go place.
Something else we did is we created a Go library that we call Message Bus. Very creative. And effectively, what we do in this is we use – it used to be run by Shopify; it’s a Go library called Sarama. Sarama is a Go library that was created by Shopify, that basically allows you to do pretty much everything with Kafka. And I actually credit that library with an awful lot of the adoption of Kafka, both at Cloudflare and elsewhere where Go was involved, because it enabled you to basically do everything that the Java libraries were doing.

One really hard thing about Kafka as we talked about is configuring it can be hard. So one good choice that we made as a company, I think, is we made it an opinionated library, that kind of set up a very good set of default settings and constraints for how we think you should interact with Kafka at CloudFlare, and made it as easy as possible for you to do it. It’s got a bunch of power user settings, if you will, where you can override what we deem to be the best settings, but that was a pretty good choice, I think. And we added a bunch of Prometheus metrics within that library as well, so it means that everybody who pulls in our library gets this dashboard for free of how their Kafka service is performing… Which was very, very helpful, and again, is another thing I’d recommend doing. It’s not Go-specific, you can do it in any language, but we were able to do it with Sarama.

Slight tangent, but Sarama actually got picked up by IBM. So IBM is now responsible for the maintenance of Sarama, because it turns out that Shopify aren’t using it too much anymore. So IBM have taken over stewardship of it. So that was a really cool thing to do… I haven’t checked in on the project in a while, to see how it’s progressing, but it was excellent that they put their hand up to carry on stewardship.

And then the final thing that we have been using Go in Kafka for is we built this thing called – we call them connectors, and they’re built on… There’s a framework called Kafka Connectors, which effectively allows you to plug some code into your database, [unintelligible 00:57:09.24] into Kafka, and then it just like moves the data between the two. So when people are trying to take things out of a database and push it to Kafka, Connectors are a pretty common way to do that.

We built our own framework that we also call Connectors; it’s all written in Go, and effectively with a very small configuration file you write in YAML, we allow you to specify a reader, some transformations to apply, and then a writer. And so what this means is teams can deploy very simple code that reads from a database, applies some transformation to a Protobuf format, and writes it to a Kafka topic. They can do it without actually writing any code; they just create some environment variables and deploy it. And same thing - we’ve got Prometheus metrics, you get a dashboard for free, and you get some alerts around it for free, and stuff.

[00:57:49.17] So all of these things have really helped with Kafka adoption. And I think if you’ve got the resource to deploy Kafka at your company, I would really consider having a team like mine, a platform team that provides tools and services that makes it easy for other teams to do the right thing, and to teach them, too. I think a huge part of our team’s job is just teaching as well, and just making sure people are following the right patterns when using some of these things. And it can help overcome some of these barriers to entry, but obviously, it’s a large cost investment.

One of the reasons Cloudflare picked Go in the first place, and we continue to use it, is it just scales so well. We’ve had a couple of issues with Kafka consumers not being able to keep up with the amount of messages that are being passed through, but after some small tweaks that you would have to make in any language, we’ve been very easily able to scale a bunch of our services to tens of thousands of messages being read a second, without too much heartache. We haven’t had to do anything clever, we haven’t had to write any sort of crazy code to do so. It’s just kind of standard optimizations that – you know, a linter will probably help you with most of them, if I’m honest. So Go has been fantastic for that… And even people who join CloudFlare who haven’t got experience with Kafka and haven’t got experience with Go, we’re usually able to get them productive and writing decent Go in a week or two just because 1) how easy Go is to learn, 2) how easy it is to read, which I think is really important and often overlooked with Go. Being able to read it… I can pick up pretty much any Go service and I can follow it through, and I can roughly figure out what it’s trying to do. I can’t promise you the same thing for Java or PHP, where there’s lots of like auto-wiring and magic, and you have to understand the framework a little bit.

So that’s been really, really powerful and useful for us in terms of adoption, too. And generally, the performance of the Go app. We are a cloud, so we pay a lot of attention to resource utilization, and containerized Go services… I think they are so tiny in comparison to some of the other things that we have to run. And even quite complicated applications that we have running, that are processing a lot of data. Their footprint is tiny. So if we were to do this all again, maybe [unintelligible 00:59:44.25] some other languages, and to be completely clear, especially in the context of the conversation we’ve been having, there has been more and more adoption of Rust at Cloudflare. More teams are definitely starting to dip their toe in and figure out if that’s a good fit for them. And TypeScript, too. There has been a lot of TypeScript usage, especially in Cloudflare Workers, because it’s natively supported and it’s fantastic. But Go isn’t going anywhere. We see new Go services deployed every day, because it just does what we need to do incredibly well.

I think one of the huge drawbacks of picking go and taking this approach that we had - and it’s something that we’re sort of still reevaluating - is, as you probably infer from what I’ve said, we’ve invested a lot of time in tooling for teams that write Go. If you don’t like Go, we haven’t got actually a whole bunch to help you right now. We haven’t rolled the same libraries in Python, we haven’t rolled them in Rust. So we’re actually kind of making it harder for teams to blaze a trail and potentially do what’s right for them, because using Go is easy for them… Which is kind of by design. We want them to stick with Go until maybe it doesn’t make sense for them anymore, because we’ve got all this great tooling that’s got production-battled experience and it works.

But one thing we’d love to support in the future is the same sort of patterns, ideas for other languages. And so we’ve been exploring some interesting things like could we put gRPC in front of Kafka, and therefore generate bindings for further languages, and therefore we could benefit from the same tooling which was set just behind our gRPC server, but the teams who interact with Kafka need to know even less about Kafka, because we’ll handle the hard configuration for them? And then we can support these other languages, too. And the thing that keeps causing me to pause is exactly what Kris was talking about, is if we do this, we’re going to remove the need for teams to understand Kafka at all, if we do it well. And that sounds like a great thing, and it maybe is in the short term, but I just feel like it will bite us a lot in the long term if we don’t – this fundamental piece of infrastructure, if it becomes one team, like mine, who knows how everything’s configured and connected to it, and another team’s kind of passing through it without ever really truly understanding it, I don’t know if that’s actually optimal in the long term. So we’re still trying to battle that and figure that out, but… As of right now, it’s the only path I can see to scale all this great tooling [unintelligible 01:01:50.17] to a way that we can support all these other languages that are starting to appear in the Cloudflare ecosystem.

David Heinemeier Hansson: It’s not only absurd, it’s actually revolting. I think we’re spoiling one of the greatest achievements of humankind, the internet. And we’re willfully regressing on all of the key components that made it so unique, and we’re handing it over to a bunch of monopolists to run as they see fit. That’s just dystopian. And I feel that because I’ve literally been working with the internet since the very first browsers came out. And I owe my entire career to the internet. I owe much of my life to the internet. So to see us spoil it in this slow-boil way is just heartbreaking.

Now, I understand why. It’s not like there aren’t reasons why people enjoy, or perhaps even prefer the cloud… But a) I think those reasons are overstated. I used to assign to those reasons. This whole moniker of like, “Well, you wouldn’t create your own power either. What, are you going to run a power plant in your backyard?” That was very compelling, I think, in the early days of cloud, and now it’s very uncompelling, as people have realized that it takes just as much f****n’ effort to run in the cloud as it does to run on-prem. In many ways, it actually takes more effort. And in many ways, the complexity of racking a server and connecting some Ethernet cables and so on is dwarfed by trying to understand the 150 services AWS puts out, or how you do permissions, or how you do cross-routing; all the other complexity of the cloud is there, it’s at least as much.

[38:24] Now, that is not to say it does not have distinct advantages, and we enjoyed some of those advantages very much when we launched Hey. When we launched Hey, we had scoped and set up the infrastructure to accommodate I think 30,000 people trying out and using the system over the course of several months. Well, we had over 300,000 people show up in three weeks. On the cloud, we could push a few buttons - it wasn’t quite that simple, but at least the mental image of it is simple, where we go like “Boom! Now we have like 10 times the capacity.” Now we can deal with 300,000 people showing up in a few weeks, instead of 30,000 people showing up in a few months. That is a difficult proposition if you had been on-prem. So that’s what the cloud does really well, it does spikes very well. So if you have this incredibly unpredictable business, the cloud does that better than anyone else. Now, is that generally our business? No. I can predict the amount of storage, for example, we need for Basecamp a year-and-a-half in advance, because it’s such a stable, predictable business. The vast, vast majority of SaaS software, and certainly all of b2b SaaS software is incredibly predictable, which is why in part it’s such a good business. But it also means that the key advantage of the cloud is not that pertinent. And in fact, it is almost sort of – it’s the downside, right? Like, you rent your computers.

Now, what do you think is cheaper - to rent your computers or to buy your computers? Now, it is cheaper upfront to rent the computer. So if you have no money at all - great. Okay, fine; you don’t have to buy the machines. But if you have to rent the same computer for five years in a row, you’re going to pay for that computer five times over, at least; in some cases 10 times over. Right? The expenses involved with the cloud are evident in the profits that AWS posts every quarter to its parent company. AWS is responsible for something like three quarters of all profit of Amazon. Now, imagine that - Amazon sells billions and billions and billions of dollars worth of merchandise every day on their website, and it’s barely profitable. They rent out a bunch of infrastructure to tech companies, and that is wildly, obscenely profitable. What do you think that money comes from? It comes from your f****n’ pockets, right? Now, there are some economies of scale, so they can obviously buy some things… But those economies of scales are not like 10 to 1. It’s not like Amazon could buy the same CPU for a 10th of the cost that you can buy that CPU. I don’t know, do they get 20% discount, 30% discount? Whatever it is, the math doesn’t work on the long term if you have a stable workload.

And that’s partly where we have been with our business since the beginning. Now, Hey, was different, but now it’s not so different. Now it’s a far more predictable thing. And when I see the amount of money we send to AWS every month from Hey, I cry a little, because it just looks like, “Geez, if we take that money and we invest it in our own software, a) we get out from under the big tech monopoly of this software, b) we’re not contributing to the death of the internet, where we reduce all the computers that run it to these three data centers, and c) we’ll actually make a bunch of money. We’ll own a bunch of software.

And what I’ve also found, particularly right now, we’re in this fascinating phase of hardware infrastructure where advances are actually happening again. Now, for the longest time, 2010 through ’18, or something, how much extra power they would get out of those Intel CPUs? Bubkes, Nada, Niche. You could barely tell the difference.

[42:07] Suddenly, TSMC and Apple have turned up the heat on the manufacturing processes, and we have AMD putting out five nanometer chips, and you’re suddenly seeing like 20%, 25% performance jumps generation to generation. And Apple has been doing this for multiple years running straight. It’s why those M1 and M2 chips are so ludicrously more powerful, in particular on the laptops, than any of the competition, because there’s finally stuff happening in the hardware realm again. And you get to be involved with that if you buy your own *bleep* computers. That’s fun. Fun in the same way as that example you refer to where we laid out all the RAM we bought for the machines back in 2012.

And the second thing is, computers are getting ludicrously powerful, particularly in terms of storage. And that is one of the things that just fascinate me. What changes, at which point do we have tipping points where - for example, Redis is a tool we’ve used for the longest time, and we used it for a bunch of storage, and that obviously runs all in RAM, but now you have NVMe cards that can do 10 gigabytes a second with a PCIe 5 and you’re going like “Whoa, maybe we don’t need to use RAM for caching anymore. Maybe we don’t need to–” It just opens a whole new domain; it feels like we’re a little bit back at the frontier, in much the same way as I recall like the Pentium days; we went from a 486 to a Pentium and you’re like “Holy s**t, we can do new things in new ways.” There’s some of that going on right now, that I don’t even think the industry at large have internalized, and part of the reason they haven’t is because so much goes straight into this amorphous cloud. If you now knew how large of a site or an app that you could run on a single damn computer…

I’ll give you an example. We just bought some new NVMe cards for Basecamp. 12 terabytes worth of storage - $3,000. You’re like “That’s so wild.” In 2012, when we were spec’ing out the hardware for Basecamp 3, we were buying these exotic, not available to mainstream Fusion IO NVMe cards that were like - I think it was $30,000 for like two terabytes, or something… Like, orders of magnitude less. They were orders of magnitude slower. I don’t think most people realize just how much computing power, how much storage performance you can get in a single box. I would say that probably 95% of – I mean, I’m just f****n’ shooting in dark here, but for comedic effect, 95% of all b2b SaaS software could run on a single box under the sysadmin’s table at the company. And there’s so many companies spending 20k, 30k, 100k, a million dollars a month renting a bunch of s**t from Amazon or Google or Microsoft, when they could just buy some of this new, amazing tech and stick it under the desk.

So I think we’re due to a pendulum swinging back here, where people realize, “Hey, you know what? The cloud was not actually that great.” And the reason it wasn’t that great was it was no less work; it has just the same amount of complexity.” Now, that’s not true in all regards. If you use something like Heroku, or Render, or some of these other fully – there, you can get away with an actual reducement in complexity. But if you are trying to set up Kubernetes, or run that yourself, or run that through Amazon, or whatever, the complexity and the expertise needed is just as high as it was on-prem, and the costs are increasingly out of sync, in my opinion, with what you can do yourself.

Gerhard Lazu: Yes. So in trying to connect the two themes, what you just shared about DevOps is 100% true and matches my experience as well. And where the analogy between the two struggles a little bit, is that if I want to have that git push Heroku experience and it cost me $30 a month, it is very difficult, I think, to be like a human who works on infrastructure and do literally any amount of customization or custom stuff, and compete with that on price. But because of the way that we consider the cost of software development - like, a lot of companies out there, as soon as you’re a full-time employee, your marginal cost on an hourly basis is $0. It’s like they become blind to just the actual expense of people’s time.

So I think of the failures of DevOps as being a failure to recognize the time sink that a lot of teams find themselves dumping lots and lots and lots of hours into when there’s commodity services that if you would only adhere to a set of conventions, would get the job done close to, or as well.

Testing is kind of like same core fallacy, is that we talk a lot about the activity and the importance of it in a sort of boolean state, like “Are you DevOps? Are you not DevOps? Are you in the cloud? Are you not in the cloud? Are you tested? Are you not tested?” It’s sort of like the degree of sophistication, because these are secondary concerns to building a product that does the thing that it’s supposed to do. No one really has the mental and emotional bandwidth to consider, “Am I DevOps-ing good? Am I testing good?”

So simply, there’s usually some, if not a person, like a mood in the team that’s like “In order for us to be a moral and ethical and upstanding team, we should be able to check this box or that box.” And so I want to check the box that I’m doing DevOps on and check the box that I’m testing. And when we consider the bad advice about either, it’s often coming from people who either are operating under that sort of simplistic notion, or for some reason have an incentive to enable and perpetuate it.

And so what I think about the failings of either are when the team lacks an appreciation for the overall total cost of ownership, the overall return on investment of where their time is going, and what are they getting for that time, or you know, in terms of AWS, or if you’re running a bunch of server somewhere to automate your CI build, and money. And if you appreciate that, then you can have a lot of really fun and interesting conversations about testing. How often are you seeing failures? When you see a failure, does it indicate an actual bug? Does it indicate somebody forgot to update a test? Is it brittle and flaky? Like, how long does it take to fix them? How many places do you have to fix the code or the tests in order to get back to a passing build? Like, how much time is lost in terms of the waiting to run the tests locally? Do you run the test locally? Do you run them in the cloud? And if you run them in the cloud, how long does it take until you get notified? And how many people get notified? Does the whole team get notified or just one person?

[36:00] And unless you know, in a quite data-driven way, the answers to a lot of these questions, general context-free advice that you see about the right way to run a test or the right tool to use is not necessarily going to help put you closer to the end goal, which is like the tests serve the team to accomplish what they were trying to do either better or faster.

Rachel Potvin: Yeah… I’ve seen a lot of ways that things didn’t work. Sometimes when you see a counter-example that’s just as good as seeing a good example, and even sometimes more effective; and I’ve tried things that didn’t work. But I’ve seen several common patterns in scaling my own teams over many years… I’ve brought multiple teams to over 100 people throughout my career. At Google I worked in developer infrastructure for a long time, and I brought those teams to over 100 people working in an organization of 2,000 people, with the amazing Melody Meckfessel, who is now CEO of a company called Observable, that I worked for her for many, many years, and learned a lot of great lessons from her, for example.

[22:01] Then at Google I also lead the cloud platform and recommendations platform in Google Cloud, and scaled that team from something like 30 people to well over 100 people. And then within GitHub as well, I’ve scaled multiple sub-teams within my organization to over 100 people when my team itself, when I went to leave, was over 500 people. And so hopefully you learn from experience, right? I mean, I certainly think I did. And like I said, that being thrown in the fire at the beginning of my GitHub experience, where – you know, there were a lot of things that were really surprising to me, in terms of how siloed GitHub was. There were a lot of things in terms of how decision-making was happening that I could tell didn’t work…

I can give you a quick story, which is when I first joined GitHub, Fantastic’s team came to me… And you know, I joined two months before GitHub Universe, which is the big developer-facing conference every year. And this great team came to me, and they were working on a language feature, and they said to me, “Rachel, we have this great new language feature, and we want to announce it and release it at GitHub Universe. As our new VP, can you tell us - should we launch it for JavaScript, or should we launch it for TypeScript, Java, Python?” …you know, the four next popular languages. And I was like “Okay, hang on; this seems like a great feature. Do we need more research? Are we not confident? Why are we just like targeting one population versus another?” And this great team said to me, “Well, okay, here’s the thing… When we first started this project over a year ago, it was easier for us to get CapEx budget approval (that’s like hardware) instead of OpEx budget approval (that’s cloud capacity). And so we ordered a bunch of machines, and we got them racked in our data center, and we were running a MySQL backend… And we have space for the index for JavaScript, or the next four popular languages, but not both, and it takes 12 weeks to order new machines. And GitHub Universe is less than 12 weeks away, and so we’ve got to pick.” And for me, coming from Google, my brain was melting a little bit, because “On-prem what? Like, isn’t it all cloud?!”

Gerhard Lazu: So after Pivotal became VMware - that’s the way think about it, Pivotal became VMware; that’s exactly my view on it - we continued using Concourse on RabbitMQ. There were many years when the RabbitMQ software had the biggest Concourse pipeline you can imagine. Not just one. Tens and tens of pipelines, for different versions, for the clients… Just to understand the scale - this was a couple of years back - we had north of 600 CPUs, a few terabytes of RAM, and I don’t know how many terabytes of SSD drives, really fast ones, to run all these pipelines. I think we had like 10 or 20 – between 10 and 20 Concourse workers. So a project which is really complex, really mature, was able to run on Concourse, and we were hitting the limits of Concourse left, right and center, all the way from the web workers not being able to generate the pipelines because of how big they were. We talked about this in the previous episodes, and I’ll drop a link in the show notes… But that was like my last experience of Concourse, and this was a couple of years back.

When Concourse started, I remember being on the Pivotal Cloud Foundry data services team, and we were struggling. This was 2016-2017. We were struggling – maybe even 2014. Anyways, it was like a significant number of years ago - we were struggling with Jenkins, we were struggling with GoCD… And while things may have changed, then, at that point in time, Concourse enabled us to do things with data services that no other CI was able to. So it worked for a really long period of time, and the only thing which I missed was a managed Concourse. Like, “Let me put my card in and then let me just get this Concourse service that scales really, really well.” That was the only thing missing.

So as a software system, it worked really well for many years, and the thing is, I never saw it as a CI system. For me, it was at the core of the Pivotal distribution process, because many customers were getting Pivotal software that way. So it was so much more than a CI/CD system.

Sam Lambert: Well, I think it’s what the cloud is supposed to have been. You look at what a lot of the major clouds provide for you now. Some provide very good services. Amazon has some great services, like S3 and whatever… But that didn’t go far enough. I think the real promise of the cloud is yet to be met and it’s starting with serverless. We think of this cloud as this ever-expanding, powerful thing, that can just enable so much for what we do, and everything is connected to the cloud, and whatever… But it’s got a long way to go in terms of user experience and usability, and it’s complex, and people who, say they have a large Amazon architecture or whatever, they’ve got a large operations team behind that. And with this new era of products and these new companies that are kind of baking themselves into this mold of this serverless model, I think that will start to change. We’ll still need operators; of course we will. They’re so important; they’re critically important internally in what we do. But it’s not just about that. It’s about what our customers can do without having to hire up massive teams.

I’ve said this before, and a lot of people have made a similar prediction which is there’s gonna be massive, multi-billion-dollar companies that are like 5 or 10 people, because they have managed to leverage so many of the tools and serverless platforms out there. So serverless is this all-in, much more refined view of how you can deliver cloud products without passing on silly, meaningless abstractions.

I remember I signed up when I was doing my sort of early discovery of what we should build as a product - I was signing up for other database products, and it’s like, they’re asking you to specify VCPUs, and stuff like this. And it’s like, if you’re a founder with the next Stripe in your head, that you need to bring to the world, why the hell are you trying to work out what a VCPU means? What does it mean? Like, I don’t know how bad your software is; I don’t know what resources are consumed. I just wanna do the thing you promised you were gonna do. How do I reason about 10 VCPUs versus 24 VCPUs on software I’ve never used before? It’s just silly. It’s lazy, it’s hostile to the user.

We just say “We’re gonna give you a performant database, and we’re gonna charge you for the things you know you do with it, which is query it and store data on it.” And that’s the experience.

Adam Jacob: Yeah, for sure. About 13 years ago I ran a consulting company that did automated infrastructure; I am a systems administrator in real life, that’s my actual thing… And I had spent a lot of time doing automation, and building automation for a string of different companies. Eventually we started a consulting firm, me and some friends. We went out and our consulting firm was doing great; we were building a lot of automated infrastructure for people, but we were struggling to maintain it after we built it. So that led us to build Chef. We wanted a better way to manage bigger infrastructures, spread across more people.

We hooked up with a guy named Jesse Robbins. Jesse of Velocity, fame and former Amazon.com master of disaster. It was Jesse who was like “Hey, this should be a company. We should actually take this thing and build a new product, raise a bunch of capital, and go for it.” So we did, and it really super-worked. We built this product called Chef that did configuration management. You describe how you want a service to work when you run it; it sees if they work the way you described, and if it doesn’t, it figures out how to fix them.

[08:12] Our original business model was that we were gonna open source Chef itself, and then we were gonna run all of the server-side infrastructure as a SaaS. This was right at the peak of SaaS hype; this was when SaaS was taking over the world, and that was the business model everybody was gonna choose. It turned out that was not the best business model for enterprise software; they weren’t quite ready to let their configuration information go out to a random person’s SaaS. But the open source part of what we did - we open sourced it under the Apache license, on purpose, knowing that we were sort of opening ourselves up to competition, but that it would garner the biggest possible spread of adoption for Chef… And that’s super-worked.

We had companies like Engine Yard, BrightScale… There’s a pretty long list of those early cloud providers and the early people who built on top of those cloud providers, all of whom built stuff on top of Chef because of how the license was constructed and because of the way that we built that community, in a pretty open and collaborative way.

As we grew, that changed into selling on-prem software. We went from being “It’s all open source and you run it as a SaaS” to “It’s mostly open source, but we sell proprietary software, too”, so we became open core. And we were always what I would call a pretty loose open core, versus tight open core. The difference there would be tight open core is like there’s core features that exist in the product that you don’t get unless you pay me money.

If you wanna back-up the database - that would be tight open core. Things that everybody needs. Loose open core means you build stuff around the core software, and then I monetize that. So if you wanted to buy a web UI, or you wanted rule-based access control in the server… Some of those things maybe you do. But it was disconnected from the core value prop; you could still just use the core of the software to do whatever you wanted, without having to do a lot of work.

We stuck with that model. We still had a SaaS, we still sold on-prem software, but sort of going back and forth through that and managing both the community, managing our business, what features are in, which features are out, why do people buy… And then most recently we took a long look at our business and why people were buying the software, and it felt like we were sort of on this treadmill of constantly putting features either into the commercial product or out, but ultimately people were buying the product because the product worked. The product itself, Chef, and the thing it did for them - that’s why they bought it.

After a lot of soul-searching and looking at the business model, we decided that what we were gonna do is open source everything. Rather than have any proprietary software at all, we were gonna follow essentially Redhat’s model, where we were gonna open source all of the software, and what we do is build a distribution called Chef. And if you want that distribution called Chef, you have to buy it from me. And what you’re paying for is both the means of that production, my support, you’re paying for the marketing, you’re paying for all of those things that surround it. But anybody is free to take that software and build a distribution of it, and to distribute it under whatever terms they want to. They just can’t call it Chef.

That’s the most recent evolution of Chef’s open source business model, and that’s been working great. They’re having the best quarters they’ve ever had, we have customers who weren’t paying us, who are now paying us, because of that change… So it’s been working pretty well.

From a community point of view it’s also been working well. Before this change no one knew how to build a release of Chef, but Chef. If you didn’t work at Chef, this wasn’t a thing you could do. Now there’s a relatively robust community of people who are building new releases of CINC, which is the community distribution of Chef. They’re hosting it at the OSU Open Source Lab. There’s a much richer ecosystem for the software than existed before we’ve made that choice. So that sort of brings you up to date, I think.

Jerod Santo: So AWS has long been the darling of cloud, the first in many ways, and therefore kind of the de facto STDIN the developer community, and yet in terms of open source and really what I would call developer relations (from my particular vantage point now; this is probably not a globalized view) it seems like Amazon has historically been – I wouldn’t say stand-offish, but just less warm to kind of the indie, open source, small business developers… Is that something that is just a misconception, or do you think that that’s perhaps historically true and changing? What’s your vantage point on that particular culture.

Mikeal Rogers: [laughs] But anyway… So if you’ve been doing Node.js for really anything, you’ve had to deal with a native module. Native modules are – we’ve been trying to rewrite the whole world in JavaScript, but sometimes you still need to bind to some kind of C or C++ module. Those modules basically get bound pretty much directly to V8, to the ABI (binary API) that V8 exposes.

The modules that you’re using that are native are probably actually using this middle layer called NAN (Native Abstractions for Node), but all that does is really just kind of a marshal between an older version of Node and the V8 VM APIs there.

This has been problematic for a number of reasons. One is that this really locks us into V8, but it also means that we break all of those native modules every major release of Node, because we take a new version of V8, and V8 has a new ABI in basically every release.

If you’ve ever upgraded a major version of Node, you’ve probably had to recompile some of your projects in order to work. If you’re on the frontend, it’s probably because of the SaaS library - there’s a really popular SaaS library written in C. If you do database stuff, it’s probably the level ecosystem, if you do robots it’s probably Serial Port; Serial Port is a native module and it needs to get down in the depth there…

There aren’t that many native modules in the native ecosystem, but if you crawl through the deep dependencies, the depths of depths of depths, about 30% of the modules in npm are indirectly dependent on some kind of native module. Breaking all of those at a release sucks, nobody wants to do that.

If another feature impacted 30% of the ecosystem, Node.js would do everything that it could to ensure that that never broke between major releases. Nothing that impacts the ecosystem that much, other than native modules, would ever be considered in the core project.

We’ve been trying for a very long time to get past this, but the main problem has been that people need to support it - Google has to support an API that we can bind to indefinitely, right? They’re not gonna guarantee support for any kind of middle layer. So even then, this native refactoring for Node - it went through a big breaking change because of the way that V8 exposed its API or changed so dramatically that it actually couldn’t marshal between everything anymore, and it had to do a breaking change.

[04:16] So if we create an API and we wanna guarantee that it’s gonna last forever, we need Google to support it, and we need any other VM that supports Node to support it.

That brings us basically to io.js. In the io.js days, we started working a lot more with Google. Google was very happy that we were taking newer versions of V8 in io.js, and from that point on, certain members of the V8 team, particularly from the Google Cloud project, have been working really diligently on Node.js and a lot of the integration points.

An effort was lead by Microsoft, because they, for Node Chakra (Node.js bound to Microsoft Chakra VM rather than Google’s V8 engine), they had to just emulate the entire API… They really felt the pain of this problem, so they lead an effort along with Microsoft and even some people from Mozilla who were doing SpiderNode, to create a native API…

Ahmad Nassri: The same patterns. And I think the short summary of that - I’ve had the privilege of being in leadership positions or executive positions where I don’t only look at the technology and the code and the infrastructure, but I also have to look after budgets. I also have to look after people’s salaries. I also have to look after sustaining people’s salaries and sustaining those budgets ongoing, year after year, quarter after quarter.

[12:04] So the dynamic shifts a little bit when you start thinking about “Are we making the right choices? Are we doing the right decisions?” When it’s not just “Is this the best library that we can use?” or “Is this the best framework that we can use?” or “Is this the best cloud system that we can adopt?” It’s also like “Are we investing money in the right space?” because if I’m gonna throw my money at a vendor – I don’t wanna name names, but like big enterprise vendors and it’s gonna cost millions of dollars, what happens to the team that I’m paying also their salaries, and how does that relationship work together? Am I gonna have to hire more people to maintain the product that I’m also spending millions of dollars on? Or I’m gonna have all of a sudden no need for the team members that I have, because I’ve shifted all my processes to a product that can, for whatever shape or form, replace a lot of the functionality that we’re doing already…

So those are both hard problems, but also critical problems, because we can debate all day long the value of frameworks and tools and services and libraries and all that kind of stuff, and the shape of an architecture that we wanna debate - and trust me, I love that stuff… If left to my own devices, to my own means, I would probably over-engineer everything, because why not? I love to create new things, I love to experiment with technology, I love to create all this complexity of cloud architectures, and challenge existing patterns, and create new ones… But at the same time, if I have an accountability and responsibility for people’s livelihoods, their salaries, their job functions and their job growth, I have to be very critical in the way I pick choices of technology, even down to the framework, even down to the process of how we implement code and how do we do automation testing, how we do deployment CI/CD, all that kind of functionality.

The patterns that I’ve noticed, again, in big enterprises, in small companies, in agencies, in outsourcing companies - there is a disconnect between developer experience, as we were talking about earlier, before we started recording, and user experience. And even though some companies have the kind of values and espouse things like “We care about the user, we care about the customer and all that kind of stuff”, it doesn’t really translate to technology and how technology is implemented. Somehow there is this [unintelligible 00:14:12.25] barrier in front of developers, to how that actually reflects on the way that I do my work.

So if I’m a developer and I’m given a story in a backlog or a task as part of a team to implement a frontend experience, or a mobile app, or a backend server, or an API, or whatever - yeah, sure, the company has these big values about users first, blah-blah-blah; it doesn’t matter, because the code needs to be the code, right? That’s how most people think.

But the reality is there’s this concept of – again, as a person who has to own budgets and has to own long-term planning for team members and technology, there’s a concept of the total cost of ownership of something. If I’m gonna make a decision on something, I need to measure the total cost of ownership about it. And to me, that kind of breaks down into five different things - the cost of knowledge, meaning if I’m gonna adopt their library, what do I have to expense in terms of time, energy, and dollars to make sure that the knowledge of that library is equally distributed across the team members that I have? And there’s a relation there to how big the team, and also in relation to how long I need to keep this thing going…

Which leads to the next one, which is the cost of maintenance. Like, fine, you’ve adopted a library, you’ve adopted a service, you’ve adopted an infrastructure tool, but how long is this gonna be going on, and how long are you gonna be invested in maintaining it and invested in having team members continue to run it, or update it, or deploy it, or monitor it - whatever the functionality is, given the component that we’re discussing.

There’s also the cost of the ecosystem, which we’re very familiar with in the JavaScript world… Basically, pick the library that has the most contributors. But even in a tooling sense or infrastructure sense – if you’re gonna buy into Kubernetes, for example, you’re gonna buy into a whole set of other things that you’re also gonna buy into and pay the cost of… Which also leads to the cost of infrastructure. If you’re gonna adopt a vendor product - cool, you’re adopting a vendor product. But is there infrastructure costs associated with that?

[16:07] Most people forget about that when they’re doing the choices of technology, saying “I’m just gonna use Gmail.” Okay, you’re using G Suite then, which means you have to do everything through APIs.” You no longer have access, for example, to the low-level systems of file storage and data storage. Now you have to have tools and libraries and desktop application management for IT teams.

There’s all these kinds of concepts that people have to measure before they get to the cost that most people actually think about, which is the cost of adoption, i.e. “How long it will take me to implement.”

So when I look at this big picture of the cost of ownership, I look at the way people make their decisions and the way people try to take shortcuts by saying “You know what - I can build a login system myself. It’s only a couple of libraries, and my cost of adoption is a day’s worth of work, or two days’ worth of work before I go and just deploy it… And then I’m done and it’s clean. This is it, this is the cost.” And they forget about the maintenance, they forget about knowledge sharing, they forget about the ecosystem around it, and they forget about the infrastructure involved.

So that’s kind of like the preamble to where are those solved problems that exist already, and how people try to re-solve them or reinvent the wheel around them… And my experience has been also in - like Amal’s earlier point - all the stuff that I was able to get things shipped; I get things shipped by simplifying the problem and getting down to the basics, rather than trying to break things apart and say “Hey, let’s use this library instead of that.” I start with the basic thing of saying “Why were you even doing this to begin with? Let’s discuss there. Are there shortcuts and cheaper options to follow?”

Ian Miell: Yeah, the book was published when I was at Barclay’s, and that was a very long process. And then when the book came out, it sold very well… So they said “We want a second edition.” I thought naively that the second edition would be “Add a chapter a here, take a chapter out there, revise some stuff, fix some stuff, and we’re done. It’s not gonna be much work.” But actually, they treat it like a whole new book. You don’t literally start from scratch, of course, you take the original book, but they go through each one, and you have to work through each part. So it was much more arduous than I thought.

The self-published books came about because – well, I can’t actually remember the chronology. I think I did the Git one first. So I was working in Barclays infrastructure, and there were all these super-smart people around me, but they were very infrastructure-focused; they were not from a dev background. And they were doing these projects with Terraform in the cloud; they were actually trying to create accounts per team that were automatically provisioned and had all the controls that were needed, and so on, but were still AWS-native; so it wasn’t like a wrapper around AWS. It was a kind of template for building out accounts.

Anyway, long story short - they came to me one day and said “Oh, Ian, you know about Git. Can you help us with – we’re trying to find out where this change came from.” I was like “Oh, great. Just send me the reference to the repo. I’ll download the repo and I’ll figure it out.” So I downloaded the repo, did git <mark>log</mark> --graphs my usual trick… And I got a page of pipes. I was like, “Okay, what’s going on here…?” And I said “How is this structured?” They were like “Well, we have 12 teams working on this product, and each team has about seven people… And each team has its own branch, and they have features off those branches.” I was like, “Okay, do you rebase?” They were like “What are you talking about? No, we merge.” I was like, “Okay… So this is why I’m seeing a whole bunch of pipes, because there’s so many branches, and they’re all merged in, therefore there’s no single line of change.”

So I tried to explain what rebasing is, and of course, I got to the deeper points about Git… And I realized, “Oh, to get there, I need to explain all this other stuff.” So I created a course for people at Barclays who were using Git, but wanted to understand it more deeply.

I’d read “Learn Ruby the Hard Way”, because I was doing Chef at the time, and I’d read a couple of other of the books by Zed Shaw, who kind of popularized the Hard Way method. I wrote to Zed and I said “Is this your trademark/copyright? Would you be pissed off/upset if I used this in my books?” and he said “No, it’s fine. I can’t trademark it. It’s actually an older idea that I took…” So I was like “Cool. I can use it. Well, I’ve got this course. I could turn it into a book.” So I did that. The book was pretty well received.

Then I thought “You know what - I don’t know Bash as well as I like, and there’s so many things about Bash where I’d like to use the Hard Way method so I can understand how it works…” So I wrote one on Bash, and that was really popular. That sold really well.

[19:51] Then I was learning Terraform and I thought “This is how I learn stuff now.” I pick up a technology, I use it, and then I’m like “I really wanna understand it more thoroughly. Writing a book is the way to do that.” So that’s what happened with Docker - I felt like a complete impostor; they were asking me to write this book because I’d been giving talks and someone had recommended me… And there I was, writing this book, thinking “Surely, someone proper should be writing this book, but not me.” But I got away with it, and I thought “Well, let’s keep this going.”

And of course, self-publishing is easier in the sense that you have more control. No one is telling you how to write it, no one is telling you you need more of this, that and the other, and you can kind of follow your vision more closely.

Justin Garrison: That’s a really good point about like recruitment and sharing, because I was from the outside following what Netflix was doing for a very long time, with - when you initially were going to the cloud, you were making immutable infrastructure, you were making these VM images, and then you were shipping your application as a VM… And then containers came out. And Netflix didn’t share a lot, because you had this product; as a productivity, as a platform team, you were all-in on VMs for a while. You were shipping VMs, and packaging those in ways that could be reused and scaled, ASGs… It works great. But then containers comes around, “Actually, we could ship a smaller package, and we can make it react a little faster.” And it took a little while for Netflix to really come out and start talking about containers, and how you were using it with Titus, the container orchestrator that was built, that originally was Mesos-based, and I think it’s now Kubernetes-based…

Gerhard Lazu: No, I know. When I say “you”, I mean Grafana Labs, the whole org that you’re part of, the whole team that you’re part of. But you were there, you had this vision, you shared it… I’m sure everybody contributed to it, and then everybody made it happen. And I really love that journey, seeing how things have been happening with Loki. I remember when Loki version one came out, and I thought “Wow, this makes so much sense.” I was so keen to start using it. And we did. Even for Changelog. We used Grafana for a long time. Prometheus… Then we went to Loki, and that was great. And then we thought “Hm… If only we could delegate this problem to someone else.” And guess what - Grafana Cloud came along, the hosted/managed service, you had some very generous tiers… Once that changed, everything changed. So all of a sudden we no longer had to run our own Grafana and Prometheus. Not that it was difficult, but it’s much easier to just run the Grafana Agent - that’s all you need - send everything to Grafana Cloud, and it just works.

[07:54] And with the last changes of the alerts - I think that was the weak point of Grafana for a long, long time. And I saw that as well. So there were all these things just falling into place naturally, and being able to know what’s coming and seeing it happening every six months, there’s like more, and more, and more. It’s like, we know what to expect, you’re delivering… “Please carry on”, that’s what I’m thinking.

Nikita Shamgunov: The community is on forum.memsql.com, and then there’s a community of mostly enterprise developers, actually - because that’s been our focus so far - that are sharing through MemSQL events and conferences. Where we’re going is - you know, now that we’ve gotten here and we’re opening up the cloud for more and more to the community, we’re thinking a lot in terms of free, and how we can make a lot of the things that got us here, got us to the 40 million round rate, with 70% growth… How can we take some of that and open them up, and by opening it up, providing a certain set of features and capabilities to the world for free? So on our dime, you go in the cloud, you log in, and there’s this free tier of stuff that you can do. That’s our current thinking so far, and I’m actually gonna be personally overseeing that effort here at MemSQL.

Jeremy Daly: A couple things. I wanna mentioned a few companies that are doing some really interesting work with serverless observability. With our traditional applications, if we’re running servers or ever if we’re running containers, we can install all kinds of daemons and bots and all kinds of things that are running there that can listen and now what our CPU usage is, and know if we’re exceeding memory, or if there’s something happening there, and that just gives us a whole bunch of reporting.

With serverless, obviously, the functions themselves are ephemeral, so they spin up and then when no one’s using them, they go to sleep again, or they actually disappear completely. So you can log information to cloud, watch logs and then kind of go through it, but seeing the whole process from request, to processing, and then maybe through a couple of different managed services, and then being able to see the results, and then if there’s something that happens there, tracking the billing… There’s just all kinds of things that you really don’t have good access into, other than sort of pouring through the logs yourself, and even that is sort of a pain.

So there’s a bunch of companies… Dashbird is one that has an observability platform. Epsagon just launched their product yesterday actually, which is a serverless observability and tracing platform, and they do some pretty cool things in this space… There’s a company called Thundra, which was a spin-off of OpsGenie, which just got bought by Atlassian… So there’s a bunch of companies in the space, plus there’s a whole security aspect around this which we didn’t really talk about. A company called PureSec - they’re out of Israel, as well; Ory Segal is the CTO over there, and they’re doing some really great work in terms of building tools that help with things like event injection, and other things that could potentially – remote code execution, and other things that are still possible and are attack vectors agains Serverless.

[01:00:35.22] There are a lot of companies that are building some really cool stuff, a lot of companies getting funded… PureSec just got funded with another seven million dollars, and then obviously Serverless has raised money, and a couple others. So there’s some interesting things happening, some cool tools being built… AWS Lambda just announced their 15-minute execution times, which is kind of a big thing, as well as that application view, and one of the guys I know at AWS has said “Look, Reinvent is coming up in a couple of weeks here, five weeks, or whatever it is…”, and they haven’t even scratched the surface of what they’re gonna launch. They said they’re basically gonna blow people’s minds with new stuff that’s coming down the pipe for serverless, so it should be some exciting times very soon… Or it already is, but there’ll be more exciting times.

Jason McGee: Sure. Like many things in the industry today, there’s quite a lot of collaboration in technology. I run the platform as a service team for IBM Cloud, and as part of my role, we look at new technologies that we wanna bring to market… And a couple of years ago we were thinking about this space - how do we help with microservices, how do we help developers build and manage microservice applications? We have all these different components working together… And we started building a piece of technology – we actually open sourced a piece of technology called Amalgamate, which was kind of our viewpoint on how to do programmable routing and control between different services within a cloud architecture…And we’ve put that out there a couple years ago.

About two years ago at KubeCon I was talking with some of my counterparts at Google and came to realize that we were solving similar problems and we kind of viewed the microservice management space in a similar way.

[06:51] We got together, had some drinks and chatted about the space, and decided “You know, we actually see the world the same way… Maybe we should join forces and combine some of the work that we’re doing.”

At the time, Google was really focused more on security and telemetry, or kind of how do you get visibility of the traffic between different services… So we decided to join forces, we combined our technologies together and we created the Istio project. We kind of worked on it, you know, not in secret, but largely just the two of us, along with the guys at Lyft, who contributed one of the key technology components to Istio, until about May of last year, when we launched the project.

So it was really the coming together of like minds around how to solve one of the emerging problems in the cloud-native space.

Adam Stacoviak: [01:16:11.17] You might think so, that’s for sure. I know I’ve ran CRMs on a desktop, in a networked environment, when I had no idea what a network was… And I was just – just the worst. Anyways… And running local software… So thank God that Salesforce came along and used the cloud to make the software run in the cloud, networked, versus on-prem, the LAN network. It revolutionized things. But I agree with Justin, it just does so much. You don’t learn Salesforce, you learn “Can I just have the CRM?” And it’s just too complicated, because it thinks you want everything, and they have this massive scale enterprise large sales team mindset, which I suppose is more of a juxtaposition of how we think about security. Is it developers only? Or does that shift-left include other folks who can help developers in the development process to get to where they need to go?

The one thing I think that is really interesting, at least with y’all’s choice, is the open source nature. You can go and self-host BoxyHQ yourself, you can do I think a premium version of it too, which - I’m not sure how you licensed that… But there’s two different flavors, so you can self-host, and I’m assuming both built on open source… But then you can also go and do the SaaS version of it. But I think what I’m trying to promote more there is less about your tears and how you’re selling it, but more so that there needs to be back to that default, that secure by default scenario where “Can I choose a viable open source version that is free to me?” Free to contribute, free to use, free as in everybody else can use it as well… And it becomes this model that seems to be a version of yes for you all, but at the same time, does that then make it one single point of failure? Like, if everybody starts going the BoxyHQ way to do SSO and secure their SaaS products for enterprise, does that mean that you become more of a centerpiece for an attack? Maybe yes. That comes back to the build versus buy, make the right choice… And like you said before, Deepak, it’s just constantly iterative when it comes to security. Iteration, always improving, being aware, and fine-tuning and making changes as that changes. But I’m also very happy that you’ve chosen open source. Can you talk about the choice to – I mean, because you can kind of cannibalize yourself, by yourself, by being open source.

Drew Wilson: So I’m the designer there. And I saw a video on Twitter by Robert Scoble a year and a half or more ago, where he interviewed Hon, who is the CEO at Cortical, and the founder… And he was up in Hon’s hotel room. Hon was there for like a trade show of sorts… And Robert met up with him, and like filmed what he was working on… I was watching this thing and I’m like “This is freaking crazy.” And it’s all the stuff that I’ve been thinking about in my mind about how things are going to go in the future, and where they’re going to end up… And I was like “I have to get ahold of this guy.” So I asked Robert, I’m like “Dude, can you get me in touch with this guy? I’m a designer. Do they need design help?” And he’s like “I think it’s too early for that.” And so he wouldn’t put me in contact… So I was like hitting up Hon on LinkedIn, on Twitter, on all the ways I could find… Eventually, he responds to me, and we meet up, conversation goes well… They’re not totally sure they need a designer or anything like that… And I ended up talking to him for like a year, and finally he’s like “Alright, cool. We need a designer. Jump in.” And so I got to join the team in like December. And then quickly built out this website in January, and launched it in early February… And then had a little bit – I was able to change some of the colors on the CL1, but it didn’t have any influence on the way that that thing looks… It will on the next iterations. And then now going to be working on the cloud interface. So when you log in, you can get yourself set up on coding against one of these brains in the cloud.

Steven Wu: There’s kind of a couple things. So basically, we’re cloud-native. A lot of these older generation stuff, like Splunk, like Datadog, like stuff based around Logstash, etc. a lot of that tends to be – it’s very important to these companies, because of a function of the times that they have good on-prem solutions, and they kind of have these applications that are designed to be run on physical computers, and so on. So if you’re running a Splunk cluster, you have like a – maybe not a physical computer, but even on the cloud option it’s like a VPC somewhere, and that’s like a node, and that handles all of the capacities of the system, which includes concerns like storage, it concerns compute… And one of the big things is that for the most part your usage of those resources are not uniform over time, and they are not uniform in any given instant. For instance, you need storage all the time, but you pretty much don’t query most of the time. So if you kind of think about what the average query use case looks like, you spend like five minutes typing stuff into the search box every day, and then the other 23 hours and 55 minutes you’re not, right? So when you have physical computers that have some constant resource constraints, and some very spiky resource constraints, you have a lot of like weird utilization issues… And this leads to a lot of basically costs.

So in our system basically each of our things, each of the pieces of that system, each of those concerns are farmed out to the cloud service that is specialized towards handling that. So all of the index files go in object storage in S3, we run ECS clusters, or whatever… Well, not whatever; that’s what we do. I actually know that. For the actual indexing. And then we run basically serverless compute for doing the query, and each of those things are chosen specifically because that particular operation has a particular set of properties in terms of the volume, the spikiness in variance, the availability requirements and so on, that are very, very different. And I can kind of talk about any of those in detail; I don’t want to get lost too much in the specifics, but…

I think we’re kind of seeing a world now where in the current generation of these log search platforms people are more and more – you’ll see the term separating compute from storage more and more in people’s marketing material, because I think we are realizing now that even very, very large companies like Fortune 50 companies are moving all their stuff to the cloud, that you can now in fact build things with the assumption that pretty much even the biggest companies have enough cloud stuff for you to work with… Whereas I think even like five years ago you couldn’t. If you wanted to work with a big financial institution, you had to have some literal on-prem distribution where you gave them a computer. So I think it’s there’s a sea change that has kind of enabled this to be possible kind of in like the business landscape, and just in terms of people moving from on-prem to cloud generally.

Adam Stacoviak: Yeah. This is where I like Warp. I don’t know if it has this feature, because I do use a few machines, but so sparingly that I forget what I’ve done the last time I used it. And so Warp, at least on a single machine - I think you would appreciate this, Jerod, and I know you already use Atuin - is that it does that. So I’m just tinkering on this spare Linux VM I have on Proxmox, because hey, I’m on a podcast talking about TUIs, so I’m gonna install some right. And I want to do it in a way where I can just blow it away, and there you go.

And so I’m logged in and I’m just like “Well, what are some recent commands, just to see if they’re there?” …because I know they’re there. I know how Warp works, but just to see if they’ve actually followed me, even into this different machine… And of course, they’re there. So I have that same feature because of the terminal application I’ve chosen, versus this sub-thing… And I think the one thing that I think Warp is driving towards - and this is zero an ad. They’re not even sponsoring us. I just like them, and Zach’s cool, and I like his team… And I believe it is a version of the terminal the future. There you go.

[00:41:42.07] I think that they’re doing things like team features, which I think would be cool because you can log into Warp and let those things transfer via their cloud service, if they do that. I think that’s what they’re driving towards. I wish they would do that more so as settings, because going between machines - it doesn’t have that. Whereas another tool I use daily is Raycast, and it has that where it cloud-syncs settings. It doesn’t cloud-sync clipboard history and sensitive things. It’s doing things that – like themes, and stuff you really want to have unified between machines. That’s kind of how I look at these tools… Like, even though we’re talking about TUIs, I still feel like Raycast and Warp are kind of similar in that respect, because they’re a layer above the need for a TUI because the application itself has the things in it it needs to give you that history.

There’s other stuff too I’m not even using, like expansions, and snippets… Things I’m not even like – I don’t know, I don’t even know how to use them, honestly. I think some of the tooling –

Jason Bosco: Yeah, so I think Typesense Cloud is a product that our users essentially pulled out of us… Because when we started working on Typesense – I mean, we didn’t think we’d build a company around it in 2015, if you had asked me… But eventually, once we open sourced, and let’s say in 2018-2019 we figured, “Okay, we probably need to figure out a business model here to make sure this is a sustainable open source project.” And then we tried the open core model, and that didn’t go too well… And then people eventually told us that they will pay us if we hosted Typesense for them. So that’s essentially people telling us that they are ready to pay, if only we had a hosted version. So that is how this came about; then we started building Typesense Cloud just based on people asking us for it… Which is, I’d say, a nice place to be in. So me and my co-founder have probably built like 12 or 13 different products in the last 15 years, and some of them did well, some of them didn’t get too much traction… But every product in the past, we would build the product first and then hope it makes money… And that used to be our operating model. But with Typesense we were in a different place, where people were telling us that, “Hey, do this and we will pay you.” So it was nice that when we launched, that week we had people paying us already, once we launched Typesense Cloud.

[48:26] So that’s when we realized there is a real problem that people are willing to pay to have solved for them… So we started just mentioning Typesense Cloud in different places in the documentation and on our landing page, saying that this exists, and people kind of organically started signing up for it and using it… And we also made sure that the product is full-featured in the open source and in the hosted version as well… So it was nice to be able to tell people that “Hey, we’re doing this only if you don’t want to worry about servers and if you don’t have an infrastructure team. We’ll take care of that for you, and that’s what we’re charging you for.” So it was very easy to explain to users what the benefit we’re giving with Typesense Cloud is, which is we’re essentially like an extended infrastructure team for them, so they don’t have to worry about servers…

So that worked out pretty well, I’d say… To answer your question, I’m pleasantly surprised with how many folks opt to use Typesense Cloud, especially – it seems like serverless is a thing that is getting a lot of adoption these days, so people generally don’t have any other… Or where Typesense Cloud fits in is if people don’t have any other VMs that they run in their infrastructure, and they don’t want to deal with hosting anything themselves; then Typesense Cloud is a nice fit there. So that also means that we now have revenue to sustain ourselves while working on Typesense…

Some of the attention we got on Hacker News, etc. we had inbound interest from almost 30 different VCs at this point, asking us if we’d be interested, if we’re considering etc. But for me personally – so I’ve worked at venture-backed companies in the past, so I kind of know the song and dance of what it takes to run a venture-scale business… And the realization that I had eventually was that in a venture-backed company, you’re essentially selling stock to your investors. And stock is, if think of it, just like another product line that you have, and your customers here are your investors. So in addition to selling your core product to your customers, to your users of the core product, you’re also selling a new product line, which is your company’s stock, to your investor group of customers.

So once I started seeing it that way, the value that your investor group of customers get from the product that they’re buying, which is the company stock, is appreciating stock value. So to keep them happy, you have to do things to increase your company’s stock value. And sometimes, some of the things that you do there might not sit well with the core group of your customers who are buying your core product… And that tension is what – I’ve seen that play out in the past; I keep seeing that play out in other SaaS companies that are VC-backed, where the eventual cycle seems to be that they price their product super-low, and subsidize it, to gain massive adoption… And then eventually, they work their way up to like Fortune 5000, Fortune 1000 companies, and start looking at million-dollar deals… And suddenly, once you have a million-dollar deal on your radar, you’re $15/month paying customer seems like a tiny drop in your revenue bucket, and your priorities as a company completely shift.

So that is what I hate to have happen with a product like Typesense, because one of my goals with Typesense is to make sure that it is available to as many people as possible, without cost being an issue. And that’s why it’s open source, it’s freely accessible… And I felt like – or at least this is my current thinking… I felt like the venture model kind of doesn’t sit well with that goal of making sure that as many people have access to Typesense as possible, or in case it doesn’t make that goal easy to achieve without conflicts of interest here and there, different position points as you grow the company.

[52:26] So that’s one big reason I’ve actually said no to all the VCs who reached out so far. And who knows – I mean, at least that’s my current state of mind. If something changes… And then we’ve been able to sustain with this model, so it’s working out very well for us.

Bryan Cantrill: That’s a great question. So I think that public cloud, I think, will always be important. I think that it will – just as we will always have hotels, I think public cloud will be important. I think that the presence of a viable on prem option, that gives you the same developer velocity, for lack of a better word… I’m not a huge fan of that term, because it makes a developer sound like a projectile… But I’m gonna go with it. But the ability to deploy infrastructure quickly - if the on prem user has that same power, I think you are going to have different people making different decisions. I think you will have people who grew up on the public cloud, who grow to a certain scale on the public cloud, and then realize “Actually, I want to control my own thing. I want to get on prem. Economically, I want to get on prem.” I think you will get people who want to do “Actually, I don’t want this to be op ex. I want this to be cap ex.” Which is reasonable.

I mean, if you look at – we all know Moore’s Law is slowing down. Everybody knows that. What does that actually mean though, Moore’s law is slowing down? If Moore’s law is slowing down, that means that the life of a computer should be longer. And if the life of a computer is longer, that means that there’s more of a reason to own that asset, if you know you’re gonna use it for the lifetime.

I mean, not that long ago, Amazon changed their depreciation schedules, right? This is one of these little details that’s happening in the financial disclosures of a bunch of companies, is that they are changing their depreciation schedules. This is financial engineering; this is making their numbers look better. But let me tell you what they’re not doing when they’re extending this depreciation schedule; they’re not cutting prices.

Adam Wiggins: Yeah, so most of the credit for putting that deal together goes to my colleague, James Lindenbaum, as well as Byron Sebastian, who we’d hired as an outside CEO, and great guy, that I now count as a friend… So they put the deal together while I was sort of heads down in product development. But really, what happened was once we started to have this buzz that came with both the sticky product that was expanding in the Ruby community, and also just making waves broadly in the developer world, and at the same time, you had – as you said, all those trends were really just on the up and up and up. Cloud was becoming a very, very hot space, and we had seemed… We weren’t getting into this business because we thought we were starting a cloud company, but indeed, people would lump us as one of the seminal sort of cloud companies. Something similar for - not necessarily Ruby as much, but maybe kind of startups, and companies practicing agile development, and developer tools, and developer experience, which is a term that I’m not sure if we’ve coined it, but we’ve certainly helped popularize it… And yeah, again, those other companies I mentioned, like GitHub, and Twilio, and New Relic. So there was just this huge lift in the market, and then we had - yeah, the timing was just right that we were a big part of the conversation there.

[56:09] So pretty early in the company, we started getting acquisition offers. And I didn’t really know how to think about those, because I hadn’t ever been – even though I’d been a business owner in the past, I’d never been a venture-backed business owner… And yeah, I just didn’t know how to think about it when, for example, Amazon came knocking, and basically said, “Maybe we’d like to buy you, and how does (I don’t know what they said) 30 million bucks sound?” something like that. And it’s just - I didn’t even know how to think about that, and certainly what would that mean for me, my partners, our investors, our employees… “Should we even think about this seriously? We’ve barely even gotten started on the product development, but boy, that sounds like a lot of money, maybe we should at least talk to them… I don’t know…”

So there was a number of these kinds of opportunities that kind of came across our desks, so to speak, and mostly my colleagues there that I mentioned, would engage with that in good faith, because… I don’t know, I guess you should do that probably.

And one that actually went very far was VMware. So we actually were pretty far in the – I don’t know if it was due diligence, or something like that. I think we had a letter of intent. So we were pretty seriously planning to join VMware. Hopefully, it’s been long enough now. I think the NDA has a statute of limitations or whatever, but the price there was 70 million that they were going to buy us for. And this was two, two-and-a-half years in. So that was just - as you said, even that was a huge multiple. And I did see it as, okay – VMware was a company I really respected, and I liked all the people I met there, I like their brand, even though they are much more pragmatic sort of like infrastructure software. But I saw it as like, okay, as we had gotten further and further in this business, and especially once we went all-in on the platform, where the deployment side was important, there was this huge component of infrastructure… And a lot of that was about things like virtual machines, and Zen instances, and stuff like that. And I said, “This isn’t really what I got in it for, and we need to build this expertise on our team”, and we were indeed doing that, and hiring people, and just learned it ourselves as we went, and so forth… But I just saw that as being potentially a really good fit, in the sense that they had that expertise, and we could be the ones that were on the developer experience, application focus, in touch with these kinds of current cutting edge application development practices, like decentralized revision control, and so forth.

So it seemed like a really good fit… And oh, by the way, everyone involved would get rich out of this, right? Obviously, of course, you think about yourself, but also all of our employees, many of whom were – this would be really life-changing for them. Also, the investors who had believed in us. Also Y Combinator, which at the time was an unproven thing.

So I wanted to find a good home for the business to continue what I was doing, and indeed, the way that we thought about it a little bit is – it’s not that different from taking venture capital. You’re getting another source of funding that comes with it, and of course, all sources of funding comes with strings attached. This will come with some strings attached, but it also comes with some real strategic benefits.

So we were pretty far in that discussion, and I think I was ambivalent about it. I didn’t know what to expect from it. I saw a lot of ways it could go wrong, and most of all, I was not done at all building the product. I had way more to say, particularly around what we were calling the polyglot platform, being able to deploy languages other than Ruby, and so forth. So I was really trepidatious about that being interrupted by something like an acquisition like this. But I also saw all the benefits that potentially were there.

[59:49] So that was a VMware deal… And that was eventually interrupted by one of our investors, Steve Anderson, who basically came to our office one day and pulled us into a room and basically said, “You know what? I think this is too cheap.” And we’re looking at him like “He’s crazy, because –” I mean, $70 million… I couldn’t even fathom that number. I don’t know, maybe – again, maybe Silicon Valley these days is just full of like eye-popping numbers if you’ve been around in that, but I’m just some kid from Los Angeles, I’ve started a lot of businesses, but most of them I’m lucky if they get into the millions in ARR, at best. And so just these eye-popping numbers, and him saying “No, there’s more potential here. You’ve gotta hold out” was quite striking. But he was really passionate about it, he made a good case for it, and we basically put together a new financing round that would let us let us continue independently.

Maggie Johnson-Pint: Sure. So the idea behind distributed trace is that you can watch a request move through your production system, and you’ll watch in these chunks that we call spans, you’ll watch it move through each server that it’s going to interact with, and you can get an idea of how long does each span take, and where are we seeing bottlenecks, where are we seeing errors [unintelligible 00:16:20.12] OpenTelemetry is this standard for distributed trace that is actually being bought into by a lot of the cloud providers. We were working on it at Microsoft.

And so what you start to see is that request can pass through cloud services, and then into other things, and you keep retaining that OpenTelemetry-based trace throughout your infrastructure and the providers’ to give you better information about how things are going. And that probably was very much the first standard that we needed to see come unlocked to be able to start getting tooling to unify in the observability space.

As with all standards, there’s certainly a lot of discussion around OpenTelemetry, and is this the right way to approach this problem…? It’s actually really interesting, I was talking to David Cramer, the CTO of Sentry last week about this, and he was calling out some very real (and I’ve known about these for a while) issues with OpenTelemetry, because it was really thinking in server terms, because he came out of this SRE space… And for us as client devs, there are some things that we need around ordering guarantees when you’re outside of the server world, that it doesn’t have yet. So I think there’s still a lot of work to be done there to think about how we reconcile some things in that space… But that’s the first standard that’s coming out.

Other stuff that we see that I wouldn’t call it a standard, but it’s a de facto kind of standard that people are starting to understand from the SRE movement… The SRE movement, if you’ve ever heard the term SLO, or service-level objective - that came straight out of Google SRE. An SLO is this concept that you would have an objective for how a service would behave. It’s not the same as an SLA; an SLA is what you sold your customer.

[18:05] An SLO is an objective for what your real goal is, and it can actually have multiple indicators or multiple sub-metrics to it. And the idea behind an SLO is as long as you’re meeting your SLO, then everybody can keep shipping. But if you fall too far out of your objective, then everybody needs to slow down their ship and work on repair work to get the system back to the level of availability that’s required.