Seven shipping principles

There’s this new website, 37signals of.com, which has three long thoughts as of today. The first one is group chat, the best way to totally stress out your team, the second one is the 37signals guide to internal communication, and the third one is seven shipping principles. As this is Ship It episode 77, we will kick it off with the seven shipping principles. David, welcome to Ship It.

Thanks for having me.

So David, why is the seven shipping principles thought important to you? Because you wrote it, there must be something to it. I’m wondering about the story behind it.

Well, they came about because I wanted to share internally at 37signals how we approach shipping software. We have a lot of new people at the company, we’ve grown the company much faster over the past year-and-a-half than we’ve ever done in the past. So a bunch of new, wonderful programmers and designers and other people involved with the product have shown up at the company, and they don’t have the 20-year history that Jason or I have building software, or the 10-year history that some of our veterans have… they have to internalize how we do it, what is important to us, what is important in terms of when we say go, when is something good enough… These principles try to encapsulate that approach.

And once I’ve written them for internal use, I read through them and I was like “Hey, this is actually a set of principles that could apply anywhere.” Anyone at any other company could choose to look at shipping software in the same way, and perhaps be inspired to set their own bar in a certain direction, and be explicit about it, be explicit about when is something good enough, how do we go about, for example, getting confident enough to ship the software we do, where do we draw the line between testing and QA, and all the other phases you might involve in the shipping process, and at least start questioning that.

I think these seven shipping principles can serve as a way to pose these questions internally, in your own organization, “How do we how do we want to ship software?” A lot of people just take it for granted, that like, “Well, we just ship software when it’s done”, or “We have this process”, or “We have these phases that we have to go through, and these check-offs that we have to do”, without necessarily taking the time to consider “Is that a good way? Is this how we want the organization to operate?”

And in any case, here are seven shipping principles that we try to live up to at 37signals. This is how we want to build software, and as we always do, we share how we build software. We’ve been sharing how we build software, both in terms of our methodology, with the Shape Up, and we’ve been sharing the actual tools we use to build software, with Ruby on Rails and with Hotwire, and here’s a set of principles we’re sharing on how we decide that something is good enough to go out the door.

[06:02] The first thing which I noticed about the seven shipping principles is your unmistakable writing style. I’m really enjoying it, because it’s short, it’s punchy, it’s to the point… You know, I really, really love it. And I’ve had the benefit of seeing that style change and morph over the years. That’s been great. And almost, like, I can recognize DHH from a mile away. You know, there’s like this thing about it.

And it’s funny, because I wrote to Lars Wikman only - he’s a very good Changelog friend - about his newsletter. He has a newsletter, and I said, “Lars, that was amazing. I really enjoyed your writing style.” And he asked me, “Why? Is it because like you’re in a hurry?” And I said, “No. The older I get, the less time I have for bulls**t. That’s what it is.” So what is the story behind your writing style, David?

Well, the short story is a lot of practice. And the slightly longer story is one of inspiration. I have tried to hone my writing craft over the years by sort of following other writers that I like; and especially that, I should say, no-bulls**t style. I don’t like reading long books. This is one of the reasons why the many books we’ve written, Jason and I together, about four of them - they’re all quite short, and you can read the whole thing. Even if it looks thick, we use a lot of typography tricks to make a short book look like a long book. You could finish it in like two and a half hours. And it gets to the point without too much of a flowery language, but still just enough that it also is interesting, right? Like, it shouldn’t be dry, but it should be clear and to the point. And it should be the kind of writing that I would want to read.

I sort of run into kind of a block often when I try to read a book where the language sort of doesn’t flow; even if the ideas are interesting, I need to be served those ideas in a compelling way. So I tried to write what I would like to read, and that’s how this kind of came about.

But then of course, also - I mean, I’ve literally been writing for the internet for about… I was gonna say 30 years; it’s not quite 30 years. At least 25 years I’ve been writing for the internet. So over those many years, and all the books and so on - a style just develops; that’s part of it. I don’t think anyone is sort of necessarily born with a particular style; they might have affiliation for a certain style, but then you hone it over the years… And I write everything in the same way, and with the same standards as I would write a book,

I really like the flow. I really like – I like all the books; I have them there in my library. I reread them… I mean, it’s great, because a good book - you’re supposed to reread. If it’s easy to read the first time, it will be easy to read the second time. And there’s always like nuance that you get. So that flow is really important. And I can see a link between the flow in writing and the flow in shipping software. Because shipping software - it’s like, you’re just getting it out there. So focus on the flow, make it easy, make it smooth, both to produce and to consume. And then just keep doing that. It’s a pleasure for everyone.

Yes. And I think the other part of it is I try not to be too precious about the writing. Like, the Seven Shipping Principles were written in, I think, 40 minutes. And that was just it, right? A lot of the punch comes from the immediacy of it. It’s kind of like - I remember hearing musicians talking about how the first riff, they go into the studio and they’re just like, “Alright, let’s get this song out. We just wrote it the last night, we’ll get it out.” And that first version, even with its imperfections, is the best one. When they then go back and try diligently to sort of recreate it, it doesn’t have it.

Now, that’s not to say that I don’t edit. Perhaps my favorite part of writing is editing, and really honing it down and making something that’s a little fuzzy, clear.

[09:58] So that’s just an enjoyable process. And to me, it’s very much the same process as writing software. The reason I like writing Ruby and working on Information Systems compared to so many other kinds of software that someone could build - video games, for example; I was greatly into video games as a kid, and I thought, “Hey, maybe this is something I want to do.” And then I looked into what it actually – what kind of programmer you have to be to write video games, with 3D engines, and I realized, “That’s not me at all. That is far more in the math department, not necessarily my strongest suit, or the thing I enjoy the most”, versus the kind of software that I write today is very solidly planted in the writing department.

This is also one of those reasons why I don’t think of myself as a software engineer. I don’t actually even really like that term. I understand it’s something the industry have more or less coalesced about… they’re like, “Oh, we’re hiring software engineers.” No, not at our company. I’ve fought long and hard for the term programmer, or software writer. That’s what I identify as, as a software writer; not as a software engineer. And it aligns to what I think is important for the kinds of systems that we build, these kinds of information systems built in object-oriented programming languages, where so much of the task of writing that is the manipulation of words, and building stories, building coherent, small universes.

One of my favorite programming tools is the dictionary. I use the dictionary all day long. Whenever I’m writing a piece of code, finding just the right word to describe an object, to describe a method… I use Thesaurus all the time, antonyms… Like, all the things, trying to figure out, “How do we get a coherent domain language?” That was actually one of my all-time favorite books. I have this list of five favorite programming books. Eric Evans, Domain-Driven Development is on there as a – it’s funny actually, because that book for me was a little bit of a struggle to make it through due to the writing… But the concepts and the ideas in it are spectacular, and something that I continue to follow.

So this idea that what we’re working on and how we’re working is a form of writing is incredibly important to me, and I try to express a care for language, both when we communicate to other humans, or when we communicate to the computer, or when we communicate to other humans through the computer, which of course, a lot of programming is. You’re writing for maybe your future self or your future colleagues as much as you’re writing for the computer.

As important as that is, we all know that we need to ship it, we need to get it out there. It’s such a temptation to endlessly refine, and change, and make it better. And it’s infinite. And this is actually something that you called out in the article. So how do you balance out this need to make it perfect, with good enough and getting it out there? How do you think about that?

Constraints. I think that we are not special at 37signals in the regard that if we were giving ourselves unlimited amount of times, like we were going to ship it when it’s done, we were going to commit the same mistakes that so many other people who have been developing software for the past 50 years have committed. Start thinking, for example, that we can estimate how long software takes - no one knows how to estimate software, and that has been repeatedly proven, and the greater the scale, the bigger the errors.

So we try to approach software development in a different way, and it’s primarily through constraints. Simply saying, for example, that we have a budget for this feature, and the budget is four weeks. We don’t specify the scope in minute detail; we have not outlined everything in a wireframe. There’s an open space for us to adjust the scope as we go along, but we’re going to ship a good version of that, at high quality, after four weeks.

And those types of constraints I think is the kind of mental guardrails that lead you to a, in my opinion, proper way of building software; a way of building software that’s not based on deadlines and distressful sense where you’ve codified what the scope is going to be down to the last comma, and then you just find out “Oh, wow, it’s gonna take longer than I imagined to do it, therefore, I must just work longer nights.” It must involve more people. The history of software development is also full of examples of how that doesn’t work; that what one person can do in one month, two people can do in two months, right? You cannot actually just add additional programmers to a late project. It’ll only make it later. That’s the Mythical Man Month, and all this sort of cannon that has existed in software development world for a long time.

But the constraints, the keeping the scope flexible, the focus on running software, and then focus on shipping is incredibly near and dear to us. When I compare our organization at 37signals to a lot of other software organizations that exist at our scale of customers, and whatever - we ship an unreasonably high degree of everything we make. And it’s to the point that it’s kind of like a special occasion for us if we’ve worked on something and it doesn’t ship.

[18:01] For me, the satisfaction, the dopamine hit that is involved in building products comes primarily from shipping it. That sense of accomplishment that you’ve not just made something that you like, but you’ve made something that you like enough to share it with other people, and feel proud about the work that you’ve done, is incredibly important. It continues to be perhaps the key driver for why I’m still doing this.

I also very much enjoy the building part, and very much enjoy playing with the domain language and all the aspects involved with that, but it kind of like – it all leads up to the point of shipping. And keeping that front and center is incredibly important, especially as you grow your organization; it’s very easy to end up in a situation where, “Hey, yesterday you were 40 people, today you’re 80, but you’re getting 10% more done.” Or in some cases, negative percents more done.

Oh, yes.

It’s very easy to lose the connection of shipping that is so pertinent for survival in the early days of a business, where you simply – either you ship or you die. When you get to have a long-running, stable business, you can actually coast for a very long time. You can talk about things for a very long time, and it does not have immediate impacts on your ability to survive as a business. So I feel that it is our obligation. Jason and I in particular, as the founders of these businesses, to continue to keep the pace, to keep the expectations of the pace of what does a successful project look like. What is the, as we like to say, return on effort? Are we spending a lot of time talking about things, worrying about things, or building vastly over-built pieces of software? I think in many ways that’s as much of a dangerous under-building software, or making software that is full of bugs, is to overbuild it. Gold-platin it, as it is also called, where you end up with a system that just overshoots. The customers don’t care about all the bells and whistles that you’ve added to it, but you feel like “Well, we’ve done a good job.”

And you can do that when you end up in this situation where your survival is not depending on shipping. And that’s, I think, perhaps the key constraint that is so powerful in those early days of the business, is that you simply can’t afford that. You can’t just hire 20 people and send them off to work on something for 18 months, and then they come up empty-handed. If that ever happens in a startup, it’s dead; it will not survive.

So you have these pressures, you have these constraints that force in the early days, and in the later days, you have to recreate those constraints through sort of willpower. And that’s actually much harder; it’s so much easier to become flappy when you don’t have to be tight, when you don’t have to be clear.

So I write these shipping principles as much to myself, as much to Jason as I’m writing it to everyone else at the company, that we need to remind ourselves of what we believe to be true. And then we need to live it. We’ve been going at this, with Basecamp, for example, for 18 years, right? The threat is there constantly to lean back, and just – when I say coast though, I don’t mean relax. No one ever relaxes. Everyone is busy all the time. But there are a universe of people who are busy all day long, all week long, all year long, who accomplishes very, very little. Who are not effective; they may be very efficient, and they may use their eight hours a day in a way where things move around, but they don’t necessarily move forward. And I think that’s the key for me, with this focus on shipping. The shipping is the moment of truth. The moment of truth where whatever you’ve built now has to meet the market. And the market

is like turning the lights on. Right? You can have all these ideas in your head about the quality of what you build, or whatever. They’re just illusions. Until you turn on the light, until it meets the market, they’re fantasies, and I love dispelling fantasies; in either direction. Either the fantasy actually turns into reality, and we build something wonderful, and people love it. That’s obviously great.

[22:21] But I also think it’s great occasionally, to release something that people don’t love, right? Like, it’s a way to buttress your own humility here, that building software is hard, even if you’re very good at it. Even if you’ve been doing it for a long time. Even if you have a successful product, building software continues to be difficult. This is why we have not automated it away, despite the numerous attempts over the decades to like, “Oh, actually, programming - we’ve solved it. We’ve solved programming. Here’s fourth generation languages, here’s no code, here’s all these other things…” And it’s not that these things don’t have a place, it’s just that they don’t actually solve sort of the crux of programming, because the crux of programming is to make decisions that matter.

Now, if you can abstract those decisions away, that’s wonderful. It probably means you work in a quite constrained domain. An Excel spreadsheet is one of the most powerful programming environments that have existed in humanity. Right? And lots of programs are written inside Excel spreadsheets. Great. Can you write Hey, an email system in Excel? No. You’ve gotta go down to the building blocks, at the level where the decisions matter. And again, this is this wonderful horizon that I care so much about.

Ruby on Rails, the framework that we use to build all our software, and that I extracted and created from the initial version of Basecamp, lives at this horizon. Can we take more of the things, more of the decisions that don’t actually matter that much, that don’t express themselves in a particular articulation of software, and abstract those away? Wonderful, I love that. That is the domain of conceptual compression, which is one of my favorite things to do, where you take something that’s a big, fuzzy, difficult domain, and then you compress all that complexity, and you make it easy, or at least approachable. And we’ve tried to do that in so many domains.

Ruby on Rails obviously have Active Record for dealing with the database; it makes it so much simpler. And most recently, Hotwire, the work we’ve been doing on the frontend, to dramatically compress all the, in my opinion, astounding complexity we found ourselves in when it comes to JavaScript development, down to a very small, core group where more people can build the kind of competitive apps that we all want to build, right? But with an iota of the effort. Hey I think is a great example of that. When we first shipped Hey - an email client, by the way, that goes head to head with Gmail, that competes with Gmail. Gmail shipped a - or ships still; when you load the Inbox, it ships 4.8 megabytes of compressed JavaScript, which actually expands to, I think, 29 megabytes of JavaScript. An astounding amount of code, right? Hey, ship, the first bundle - 30 kilobytes. 30 kilobytes with an application that goes head to head with Gmail, in a modern way. That’s the kind of conceptual compression where we’re talking orders of magnitude less stuff to solve the same problem, in a, in my opinion, far more pleasurable, enjoyable way to work, a far more labor-intensive way to work, and one were, we shipped, we met the market, and the market says, “This is good.”

[25:44] I was that market. I was part of that market. I remember before it came out, I was like, “Wow, this is amazing.” I mean, I’ve known Ruby on Rails, and I’ve known Basecamp for many, many years. 15 years ago when I discovered Ruby on Rails, I was like, “Wow, this thing is amazing.” And that was the context in which I learned infrastructure and operations. And the reason why I could do that is because the application was simple, was easy. You’d look at it, there was like - the material, the books were great. It just like kicked off this entire change in the industry, because the application was simple. You had none of that Java complexity. You had Ruby, you had the language, you had that beauty; you could focus on other things. That was amazing.

And I remember the JavaScript back then was still a problem. And while I have left the Ruby on Rails ecosystem since, Hotwire sounds amazing, and it addresses one of the pain points that was there. Basecamp - an amazing tool that helps you write. I was like – again, that’s a different story for another time. But let’s come back to Hey, because Hey, changes the way the world views email; the way just the world emails. And I really liked it when it came out. The thing which I really missed was “How do I integrate with Gmail for sending emails?” That was the problem. I was so happy when you introduced that feature not long ago; like, “Yes! I can finally combine these two things, and I can start doing my migration.” I was super-excited.

Yeah, I think it’s one of those things where – we always build software for ourselves first. The version one of any new product that we put out is the version we want. And we’re very deliberate about the fact that we don’t want to pollute that process by taking customer feedback too early. I think there’s a lot of software that’s built by committee, that’s built by focus groups, that’s built by all sorts of forms where it waters down the key sharp points that dulls the edge of what you’re trying to do. We weren’t trying to replicate Gmail; we weren’t trying to make Gmail with a different set of colors on it. We were trying to make some fundamental changes to how people use email, and change their relationship with email… And that had to be a sharp edge. But it’s also then true that that is the most, if you will, extreme version of that. And then we, after release, then spent tons of time listening to everyone, involving them in, and that integration with Gmail was clearly a huge blocker for a lot of people. they wanted to try Hey for real, which meant that they had to both send and receive email, but they didn’t want to tie themselves to a new email address until they were absolutely sure that they wanted to stick with that.

So I’m really happy we did that. It’s actually an interesting story, because right from when we launched, we had this feature, just not for Gmail. Because Gmail is essentially a de facto monopoly in the US. And they use that monopoly power to exclude normal forms of integration. You can’t just use it as a sending box using IMAP, or POP3, or SMTP. No, you actually have to go through their API. And to go through their API, you first have to be permitted to go through their API. There’s a months-long application process, which amongst other things, requires an external security review by a set of firms that they designate to be worthy of doing these security reviews, to the tunes of tens of thousands of dollars and months of effort to do it. And at the beginning, when we launched Hey, we looked at that and said like, “You know what - people clearly want this.” Something like 80% of all Hey customers - maybe it’s even higher - 80% come from Gmail. Gmail is just the monopoly in the email world, particularly in the US, but also around the world. they have 1.5 billion users, or something like that.

[29:40] And as such, I was just like, “We are not going to just fall on our knees for that monopoly.” The whole point of releasing this is to start a fight with this monopoly, in the sense that because Gmail has captured this monopoly, they’ve let email stagnate. they’ve let the entire industry of email stagnate, because they’ve set the kind of boundaries for how email is supposed to work, all the expectations, and they were poured in concrete in like 2005. It’s essentially been an ice winter when it comes to new thinking in email systems for 15 years.

Now, I know there are other email clients that have done interesting work over the years, but the vast majority of those have just been clients. they integrate with Gmail, so you’re still bound into that. And we wanted to make a series of changes that we believe were much better done if we were an email service; not an email client, but a service that could take on Gmail head-on, and it essentially allows someone to escape Google entirely if that’s what they wanted to do.

I think one of the nasty, unfortunate parts of today’s technology is that even when we don’t like a supplier, like say Google, it’s very difficult to actually escape; it’s very difficult to live your digital life cutting things off. And we wanted to take one of those big ones, email, and take that out of the hands of Google, and say, “Here’s a modern email product that someone would legitimately prefer over Gmail, and it will allow them an escape hatch.” Once they’ve gotten out of Google land with their Gmail, perhaps it’s not that far of a reach to replace the other parts of it… But it was just such an onerous process. I mean, the whole thing was onerous, right? We’ve spent essentially 20 years getting ready for this moment. Running your own email service is difficult. Building your own email service - difficult. At a whole other level than anything we’ve actually tried to tackle before. And then going head to head too with the likes of Gmail. Difficult.

I mean, we’ve long operated in a blue ocean with Basecamp. Now, Basecamp has a lot more direct competition now. I mean, for years and years, I thought like, “Where’s all the competition? Why is no one showing up? This is a wonderful business, we have tons of happy customers”, and it just took an extraordinarily long period of time before direct competition showed up. I mean, Basecamp has had competition since day one, but it was always sort of like different kinds of things. And now there are a lot of venture-backed companies who’ve realized, “Oh, this is actually a good business.”

So now there is direct competition. But with Hey, we didn’t have that luxury. We didn’t have a runway of a sort of Greenfield, with no competition, no direct competition. We had incredibly tough direct competition. The toughest of all, free competition. And free, reasonably good. Within the constraints of what Gmail does, Gmail is good. Like, Gmail is the best email service anno 2004.

Yup.

It’s just that it’s no longer 2004. A lot of years have passed since 2004, and I think it’s actually not that great of an email service anno 2022. But it is free. And obviously, that’s very difficult. We then show up and say, “Hey, it’s 100 bucks a year.” Even if that amount of money for someone who actually uses email on a frequent basis, you’re like, “That’s not even 10 bucks a month.” But that is the power of free.

So that’s all the stuff we were up against. I’m really happy that we’ve now been able to, as we’ve called it, fill in a bunch of the space underneath. That wasn’t the reason we set up to do this, but now more people can enjoy Hey, so… Sorry, that was a bit of a tangent.

No, that was a good tangent, that just like fills in a lot of blanks, which even I had. So that’s very helpful. The one thing which I remember from the Basecamp days - this is a picture, and I will look it up and post it in the show notes… I forget who posted it; maybe it was you… But it was a picture of what one terabyte of RAM looks like. And the context – was it you?

Yes, it was me. Yes.

[33:53] Alright. Okay. Wow, amazing. And I was like, “Actually, you can scale these systems fairly simply by just, you know, putting more RAM in them.” And that was amazing. That was like in the context of a database, just like it was showing how far you can take a MySQL database that Basecamp was using at the time; maybe it still is now, I don’t know. But I’m wondering, from the infrastructure perspective, Hey was very different to Basecamp. Hey was this new thing, the scaling requirements were different… How did you approach the infrastructure operations of Hey differently to the Basecamp ones? Because I don’t think you bought a terabyte of RAM for Hey, and there you go, off you go. I think that looked very different.

It does look very different. Email is several orders of magnitude more difficult than almost every other type of information system that you can build. And a big part of that reason is the volume of data is incredibly high. And it’s not just volume of data like if you’re hosting videos, or something; you’ll have a lot of data. But that’s in a – I should say, to some extent, I know there are complexities too, but like a regulated format. Email is still the Wild West; email is still absolutely all over the place, in part because it is this wonderful, we should recognize, open standard where there are literally thousands of different clients. And then it’s perhaps less fortunately also an open STDIN the way that anyone can send you email. And however much they want to send, they can send. And it’s incredibly cheap to send millions, if not billions of emails. That’s how we ended up with the whole spam issue, and thankfully, advances have been made since the dark days of spam, where most of the spam ended up in your inbox… But it’s still an incredible volume of email. I mean, our databases for Hey, they grow by like tons of gigabytes every day. Every day. That’s a completely different scale than - you think of something like Basecamp - how long would it take, or how much, how many comments, how many posts, how many to-do’s would it take to create gigabytes of data? I mean, it just happens at a completely different timeframe.

So Hey necessitated a very different approach. And one of the things we did different was we went straight to the cloud right away, which is one of the things I have, to put it mildly, mixed feelings about. And in fact, we’ve now set a mission for ourselves that we want to get off the cloud, at least off the public clouds… Because I don’t want the entire internet to be run by five companies. And that’s where we’re headed right now. Not even five; I think we should say three maybe.

Three, that’s it. That’s what I was thinking.

And this idea that we take this beautiful, beautiful system that is the internet, that was designed from day one to be resilient, to be not owned by single companies, to be all these things, and then we take all of that and we hand it over to three companies to run all the servers? And if Amazon US East 1 is down, then like 20% of all websites go offline… That’s–

Crazy.

It’s not only absurd, it’s actually revolting. I think we’re spoiling one of the greatest achievements of humankind, the internet. And we’re willfully regressing on all of the key components that made it so unique, and we’re handing it over to a bunch of monopolists to run as they see fit. That’s just dystopian. And I feel that because I’ve literally been working with the internet since the very first browsers came out. And I owe my entire career to the internet. I owe much of my life to the internet. So to see us spoil it in this slow-boil way is just heartbreaking.

Now, I understand why. It’s not like there aren’t reasons why people enjoy, or perhaps even prefer the cloud… But a) I think those reasons are overstated. I used to assign to those reasons. This whole moniker of like, “Well, you wouldn’t create your own power either. What, are you going to run a power plant in your backyard?” That was very compelling, I think, in the early days of cloud, and now it’s very uncompelling, as people have realized that it takes just as much f****n’ effort to run in the cloud as it does to run on-prem. In many ways, it actually takes more effort. And in many ways, the complexity of racking a server and connecting some Ethernet cables and so on is dwarfed by trying to understand the 150 services AWS puts out, or how you do permissions, or how you do cross-routing; all the other complexity of the cloud is there, it’s at least as much.

[38:24] Now, that is not to say it does not have distinct advantages, and we enjoyed some of those advantages very much when we launched Hey. When we launched Hey, we had scoped and set up the infrastructure to accommodate I think 30,000 people trying out and using the system over the course of several months. Well, we had over 300,000 people show up in three weeks. On the cloud, we could push a few buttons - it wasn’t quite that simple, but at least the mental image of it is simple, where we go like “Boom! Now we have like 10 times the capacity.” Now we can deal with 300,000 people showing up in a few weeks, instead of 30,000 people showing up in a few months. That is a difficult proposition if you had been on-prem. So that’s what the cloud does really well, it does spikes very well. So if you have this incredibly unpredictable business, the cloud does that better than anyone else. Now, is that generally our business? No. I can predict the amount of storage, for example, we need for Basecamp a year-and-a-half in advance, because it’s such a stable, predictable business. The vast, vast majority of SaaS software, and certainly all of b2b SaaS software is incredibly predictable, which is why in part it’s such a good business. But it also means that the key advantage of the cloud is not that pertinent. And in fact, it is almost sort of – it’s the downside, right? Like, you rent your computers.

Now, what do you think is cheaper - to rent your computers or to buy your computers? Now, it is cheaper upfront to rent the computer. So if you have no money at all - great. Okay, fine; you don’t have to buy the machines. But if you have to rent the same computer for five years in a row, you’re going to pay for that computer five times over, at least; in some cases 10 times over. Right? The expenses involved with the cloud are evident in the profits that AWS posts every quarter to its parent company. AWS is responsible for something like three quarters of all profit of Amazon. Now, imagine that - Amazon sells billions and billions and billions of dollars worth of merchandise every day on their website, and it’s barely profitable. They rent out a bunch of infrastructure to tech companies, and that is wildly, obscenely profitable. What do you think that money comes from? It comes from your f****n’ pockets, right? Now, there are some economies of scale, so they can obviously buy some things… But those economies of scales are not like 10 to 1. It’s not like Amazon could buy the same CPU for a 10th of the cost that you can buy that CPU. I don’t know, do they get 20% discount, 30% discount? Whatever it is, the math doesn’t work on the long term if you have a stable workload.

And that’s partly where we have been with our business since the beginning. Now, Hey, was different, but now it’s not so different. Now it’s a far more predictable thing. And when I see the amount of money we send to AWS every month from Hey, I cry a little, because it just looks like, “Geez, if we take that money and we invest it in our own software, a) we get out from under the big tech monopoly of this software, b) we’re not contributing to the death of the internet, where we reduce all the computers that run it to these three data centers, and c) we’ll actually make a bunch of money. We’ll own a bunch of software.

And what I’ve also found, particularly right now, we’re in this fascinating phase of hardware infrastructure where advances are actually happening again. Now, for the longest time, 2010 through ’18, or something, how much extra power they would get out of those Intel CPUs? Bubkes, Nada, Niche. You could barely tell the difference.

[42:07] Suddenly, TSMC and Apple have turned up the heat on the manufacturing processes, and we have AMD putting out five nanometer chips, and you’re suddenly seeing like 20%, 25% performance jumps generation to generation. And Apple has been doing this for multiple years running straight. It’s why those M1 and M2 chips are so ludicrously more powerful, in particular on the laptops, than any of the competition, because there’s finally stuff happening in the hardware realm again. And you get to be involved with that if you buy your own *bleep* computers. That’s fun. Fun in the same way as that example you refer to where we laid out all the RAM we bought for the machines back in 2012.

And the second thing is, computers are getting ludicrously powerful, particularly in terms of storage. And that is one of the things that just fascinate me. What changes, at which point do we have tipping points where - for example, Redis is a tool we’ve used for the longest time, and we used it for a bunch of storage, and that obviously runs all in RAM, but now you have NVMe cards that can do 10 gigabytes a second with a PCIe 5 and you’re going like “Whoa, maybe we don’t need to use RAM for caching anymore. Maybe we don’t need to–” It just opens a whole new domain; it feels like we’re a little bit back at the frontier, in much the same way as I recall like the Pentium days; we went from a 486 to a Pentium and you’re like “Holy s**t, we can do new things in new ways.” There’s some of that going on right now, that I don’t even think the industry at large have internalized, and part of the reason they haven’t is because so much goes straight into this amorphous cloud. If you now knew how large of a site or an app that you could run on a single damn computer…

I’ll give you an example. We just bought some new NVMe cards for Basecamp. 12 terabytes worth of storage - $3,000. You’re like “That’s so wild.” In 2012, when we were spec’ing out the hardware for Basecamp 3, we were buying these exotic, not available to mainstream Fusion IO NVMe cards that were like - I think it was $30,000 for like two terabytes, or something… Like, orders of magnitude less. They were orders of magnitude slower. I don’t think most people realize just how much computing power, how much storage performance you can get in a single box. I would say that probably 95% of – I mean, I’m just f****n’ shooting in dark here, but for comedic effect, 95% of all b2b SaaS software could run on a single box under the sysadmin’s table at the company. And there’s so many companies spending 20k, 30k, 100k, a million dollars a month renting a bunch of s**t from Amazon or Google or Microsoft, when they could just buy some of this new, amazing tech and stick it under the desk.

So I think we’re due to a pendulum swinging back here, where people realize, “Hey, you know what? The cloud was not actually that great.” And the reason it wasn’t that great was it was no less work; it has just the same amount of complexity.” Now, that’s not true in all regards. If you use something like Heroku, or Render, or some of these other fully – there, you can get away with an actual reducement in complexity. But if you are trying to set up Kubernetes, or run that yourself, or run that through Amazon, or whatever, the complexity and the expertise needed is just as high as it was on-prem, and the costs are increasingly out of sync, in my opinion, with what you can do yourself.

What were you thinking in terms of hardware, and - like, going back to the data center model, going back to your own hardware… How does it look like? Or what are you thinking of running for Hey, now that you’re thinking of going from the cloud, to on-prem?

Now, the good thing is, we have deep expertise, because this is what we’ve mainly done for 18 years. So we already run out of two data centers. Of course, you don’t run your own data centers, or I mean –

You could, but yeah… [laughs]

I shouldn’t say that with scorn, I think actually there would be something interesting again. I remember when I started in the internet, the common thing was you ran your own servers, sort of in your own building, right? That was not at all an anomaly, that you would literally have the servers to operate your website or web service in your office. That was totally common and normal. I would actually love for us to get back to that. But let’s just say you don’t. Let’s just say you go to a professional data center, you rent a cabin or two, you rent some power, you rent some space… You don’t even have to do the thing anymore; you don’t have to rack the servers anymore. All these data centers have these white glove services; you order a bunch of machines from Dell or whoever, they get delivered straight to datacenter, someone will take those machines, they’ll slot them in and they’ll connect the power and they’ll connect the Ethernet, and then boom, you’re up and running. And that’s essentially the approach we’ve been using.

So we buy a bunch of hardware, a bunch of machines, we’ll rack them in two different data centers, so we get that kind of redundancy, and then off we go. Now, of course, reality is slightly more complicated than that, especially when you have a very large existing operation that is already in the cloud… So migration, and so on - internally, we kind of have the goal that like in a year from now, we’d love to run Hey on-prem. So you can imagine that there’s quite a lot of work involved with that, when you take an in-flight service like that. But we’ve also made the decision that future products from 37signals, unless they have a unique need to be dramatically scalable from Monday to the other, we’ll start them with our own stuff, again.

Interesting. And once you get that hardware, what happens next? What operating system do you get? What runtime do you get? How does basically code make it onto those machines, and what services do you have available? Like, do you use containers? Do you use something else? What does that look like?

Yeah, so that’s the other interesting thing, is that when people think on-prem, they think like, “Well, that was a single box, and you just installed the stuff directly on it, and then you manually have a bunch of system administrators manually tinkering with conflicts, and so on.” Of course not, right? All the lessons of the cloud apply - most of the lessons of the cloud - apply to your own hardware as well. So what we’re essentially trying to build up now, particularly for Hey, is to build up our own cloud, if you will. And by cloud I mean you can take a bunch of physical servers, and then you can slice them in virtual ways.

[49:49] So we’re looking at something – I believe it’s called harvester, that SUSE runs, which is kind of like a VMware virtualization setup. So you take a big physical box and you break it down into small virtual ones, and you put Docker images, or your MIs or whatever, on those machines, and you operate them in much the same ways as you would operate a cloud setup. In fact, for our case, we’re even looking at Kubernetes. I mean, for all its complexity, it also does offer some very interesting approaches. I don’t know if we’ll end up committing to that and run that ourselves. It is kind of a beast… But there’s a lot of the techniques and the tools that we’ve been used to using in the cloud you don’t have to give up just because you buy your own hardware.

Now, there is a line where we need to figure out where to cut it, right? Like, for example, do we actually want to run Kubernetes ourselves? There’s complexities involved with that. But it doesn’t to also some extent really matter that much. We have, for example, setups where we just don’t use Kubernetes, and we just use images directly, and we’ve sliced these physical servers into virtual setups, and you can move these images around… So it’s not like you have to go back to the stone ages because you want to run your own hardware.

Just to change the subject a little bit as we are preparing to wrap up - we have maybe another five minutes - there’s something which I’ve been waiting to this moment to ask you… Now, we’ve been talking about fast NVMe’s, and things like that… I think you’re one of the few software writers - let me use the term that you prefer - that knows what it feels to be fast… Because this year you participated in the Le Mans race. What does it feel like to be in a race car, in an LMP2 one? I’ve never been in a race car, so describe that feeling for us.

The best parallel I can give to a programmer is it is like entering the state of flow on command. So this state of flow, where you can at times lose some connection to time itself, where you think like, “Oh, I’ve been programming this hard problem. What - I’ve been sitting here for 20 minutes?” and you go like, “No, it’s actually been an hour and a half.” It is this moment of intense focus to the point where all these other things you might be thinking about disappear.

And there’s this wonderful book and concept of Flow, that’s been exhaustively studied, that essentially pointed as these are the moments of pure happiness for most people. That these moments of pure happiness are these moments of total focus, where you’re activating all your human potential, you’re using your skills in just a way where it’s just beyond your reach; you’re not trying to do something so hard that you will constantly fail, but… As when I’m working on a new system, and I’m trying to come up with sort of the best class names, the best object names, and so on, and I can get into this state of flow where there’s nothing else in my head.

But I’ve found that that’s, that’s difficult to do on command with programming. I mean, I feel blessed when I fall into this state of flow, and it’s really all encompassing, and I make these huge leaps in solving a difficult problem, or devising beautiful architecture. With a race car, you shut the door, you turn on the engine, you drive up to the end of the pit lane at 60 kph, then you push the pit lane button, and you’re flooring this thing - okay, now it’s only about that. There’s nothing else in the world. The corner’s coming up, particularly if you’re driving in a race… Like, at Le Mans, you come up to the Porsche curves; it’s a fifth gear corner, you’re entering pulling three and a half, four G’s, you’re going 250 kilometers an hour, there’s a wall, I don’t know, 10 meters from the edge of the track, or five meters from the track… If you do not give that your full and undivided attention, you could get seriously hurt, or worse. That is such a forcing constraint on your brain that it basically says like, “Hey, do you know what? The state of flow is necessary in this environment for you to– to put it to a point, survive.” That is a very gratifying state to be in, and to be able to activate it with such regularity is truly a remarkable experience.

[54:10] And then the other thing is, driving a race car well is very much like optimizing a piece of code, or editing a piece of prose. It’s a closed loop, and you get a verdict back at every lap, usually two minutes on a normal track, four minutes at Le Mans… Did I do better? Did I do worse? That sense of immediate feedback from your application of skill is intoxicating. It’s intoxicating to know that there’s a stopwatch that keeps track of whether I’m doing better or worse, and I can measure myself against others on the basis of that time. Am I progressing? Am I becoming a better driver? It’s truly a thrill.

And then I also just enjoy kind of the physical aspect of it, consistently pulling for example 4g, or having to wake up at 3am in the morning to go into a car and drive for two hours at 300+ kph… I think when we were driving the fastest, we were always driving 350 kilometers an hour… That is a unique experience that you don’t have access to in daily life in the same way. I was about to say civilian life… There is almost something militaristic about this, this fusion with the machine, which is - perhaps of all the things that having run a profitable company for 20 years have given me access to that I would not have access to otherwise, that is the one thing where I go like, “Okay, yeah, I enjoy that.”

That is amazing. Just listening to that, I can imagine it, and I can imagine the discipline that that takes, whether you’re preparing for it, whether you’re there… Like, mistakes are very expensive. You know, you don’t have the luxury of “You’ll just push again, that’s okay.” You know, like ship the fix, or roll back. There’s no rollbacks, okay? [laughs]

I just had this - two races ago, I did a terrible mistake, where I ended up smashing the car pretty substantially… And not only was it very expensive, it was also one of those things where you just go like, “Yeah, there’s not a rewind button here. Like, this thing just happened. And it happened because I wasn’t good enough. It happened because I made a mistake, and I miscalculated my approach to it.” And knowing that that’s there, that that consequence to your actions is so readily apparent is one of those other focusing functions. Now, that is in my mind, that like “I make another–” And you think of those mistakes, you think – they don’t have to be very big. I was off the corner by less than a meter. I was less than a meter off from where I needed to place the car, and the consequence of that was that I kind of smashed it. I ran over some curb, the car pitched left, then it pitched right. Good tank slapper, and I flew straight forward into the wall. And even hitting the wall net and I mean, I was actually incredibly fortunate. There are people who have done the same mistake and then completely destroyed the car, flipped it, or gotten seriously hurt. And you go like, “Sometimes part of the enjoyment comes from the fact of knowing that risk is involved”, right? That the purpose of life is not just to minimize all risks to a minimum. I see that with my kids as well. They enjoy physical exercise and physical activities more when they know there are some consequences; like, they’re swinging on something, and if they miss it - it’s not like they’re going to break their skull, but they might get hurt a little bit. That’s actually good. And I think one of the ills of modern society is our strive for safety-ism to an extreme degree, such that we cannot tolerate any form of risk anymore… You end up with dull people like that.

That is an amazing takeaway. That’s all the time that we have today. DHH, it has been an actual pleasure. Thank you very much for today, and I’m looking forward to next time. Thank you, David.

Thanks so much for having me on.