The foundations of Continuous Delivery

I remember how difficult it used to be to get code into production. FTP used to be used a lot, rsync used to be a thing, getting thing out there… And something happened around 2010. There starts to be a shift in 2012/2013; there was an acceleration of just git pushing, and the code would get out there. And I thought that was amazing. Like, why haven’t we been doing this all along? I mean, what else do you need just to get it out there for the users to tell you “Does it work, or doesn’t it work?” or “You’re missing this.” The quicker you can get to that point, the better off you are. And even if you make mistakes, that’s okay. How do you learn if you don’t make mistakes? So don’t try to not make mistakes, try to make them so quickly and fix things so quickly that no one even notices. By the time they notice there’s a problem, you’ve fixed it, and it doesn’t exist.

I think you, Dave, had something to do with this, because around 2012 you published this book, you co-authored this book with Jez Humble, which was called “Continuous Delivery.”

Yeah.

[04:04] And even though, hand on heart, I haven’t read the book, but everything that you capture in that book I sure have practiced for more than decades, and I cannot think of working any different. So how did you come up with the concept of continuous delivery?

I’ll start off by admitting that I’m very old and I’ve been doing this for a long time… So mostly how I came up with this is by doing it wrong in lots of weird and interesting ways first, and finding out what didn’t work. I had a kind of formative experience, several formative experiences… But one time I remember in the late ‘90s building some reasonably complex software for an insurance company, and we were supposed to deploy it by writing a manual script that somebody would then take over and execute. And the system wasn’t configured in the way that we expected. We didn’t know what the configuration system was really, so that didn’t work very well. So I remember me and a friend spending two days, stood up in a server room somewhere, trying to manually install the software… And I was thinking “There’s gotta be a better way of doing this…”

Later on I worked on a project for a points-of-sale system when I worked for ThoughtWorks in the early 2000’s… And we were doing extreme programming at some scale; at the time we thought that it was probably the biggest agile project in the world. There were about 200 people working on this agile project on three different continents… In those days it sounded really weird, because agile projects in those days were just small projects. And we were kind of trying ideas out.

So we started to get more disciplined in our approach, we started seeing the things that were going wrong, and we started to introduce more of the deployment automation, more of the configuration management infrastructure as code, better approaches to automated testing, and starting to formulate basic deployment pipelines out of those. And it was during that period that we started playing with the ideas and pulling something together and thought that we had something.

The book itself came out of a notion – there were a bunch of us doing different things on different projects in ThoughtWorks, and we thought that we were on to something. We were starting to see patterns that worked, and we were starting to apply those to the project, so we would go in having a better sense of what to do to begin a project and have some success with it.

So we thought we’d kind of write a book that was initially meant to be a series of essays, and there were a bunch of us that said “Yeah, we’ve got some stuff to say” and started talking about it… Actually, when it came down to it, only Jez and I did any writing, so that’s why we ended up writing the book. [laughs] And the book morphed into something else. It certainly didn’t start out being called “Continuous Delivery”, but it morphed into something else, and both of us, I think, were a little bit wary of thinking of this at the time as – and I do think that it’s kind of a methodology. It’s a way of approaching software development in its own right, and I believe that it’s an engineering practice. I think that engineering in the sense of amplifying the impact and the talents of the people that are making the changes. as you said, software development is weird stuff, and one of the really hard things is knowing how well you’re progressing, knowing how good your ideas are, and so you want to be able to get those out into the hands of users quickly and efficiently, so you can learn from that and adapt and change.

I think you put it perfectly when you were describing it in terms of we wanna give ourselves the freedom to make mistakes. We want to be able to start – I am a popular science nerd. I love reading about science, and physics in particular, and I think that we can learn a lot from the fundamental philosophies of science. I don’t mean Six Sigma accuracy and statistics or something like that, but just applying scientifically rational thinking. You know, start off assuming that we’re wrong, rather than assuming that we’re right. Test out our ideas, try and falsify our ideas. Those are better ways of doing work, and it doesn’t really matter what work it is that you’re doing. That stuff just works better. And certainly, the ability to move quickly, make small changes quickly, observe the impacts of a change so that you’re in effect controlling the variables, limiting the scope, the blast radius of mistakes is a fantastic way of making progress efficiently.

[08:41] One of the things that I am obsessed with at the moment is watching Elon Musk and SpaceX build starships to go to Mars and blowing them up in Texas, because that’s how you learn. That’s how you do great engineering. I think we’re on to something here, because I think continuous delivery is an approach that allows us, facilitates that kind of thinking and that kind of approach to software.

You’ve made so many great points that I have difficulty tracking all the things that I wanna mention to all your points… I mean, there’s just so much, so let me start with this. First of all, thank you very much. You have no idea how big of an impact your approach and your teachings and sharings had, not just on me, but on everyone I know. The stuff that you do and the staff that you’ve been promoting for decades now have been part of me in many different ways.

For example, I was also into physics when I was in high school, and I was convinced that I’ll go to university to study physics. But then, when I was at the University of Puget Sound in Tacoma, there was the “100 Years for Max Planck” talk. That was a fascinating conference. But I discovered the Macintosh. And that changed my life.

So I was so good at making mistakes and learning from them, and deploying, and figuring out what doesn’t work, that people said “Can you do my website? Can you host my thing? Oh, how do I do emails? I don’t know how to do emails.” This was like the early 2000’s. That’s when I started doing these things properly. And I realized “Well, this approach of trying to figure out if it works - it’s so good that you don’t need anything else. Keep learning, keep iterating, keep improving, and that’s all there is to it. Nothing else.” And continuous delivery is so fundamental to this approach of working that everything changes and you don’t want to go back. I can’t imagine–

Absolutely. That’s one of the things that I’ve observed. First of all, thank you for saying thank you; that’s very kind. But I’ve had the privilege of working with some great teams over the years, and I have yet to see one that has adopted continuous delivery in the way that I would recognize it that ever wants to work in a different way.

One of my other formative experiences was I was involved in building a very high-performance financial exchange, while I was in the middle of writing the continuous delivery book… And that was an exercise in genuine engineering. We were doing some really hard stuff, some really difficult stuff to be able to build this ridiculously efficient software system. But we were starting with a blank sheet of paper. I was the head of software development, and I dipped in the middle of this continuous delivery thinking, because I was in the middle of writing the book… So we built the organization from the ground up as a continuous delivery organization. And the commonest message that I still get from the people that I worked with on that project is “Oh God, I miss what it was like there…”, you know, if they’ve moved on and they’ve gone somewhere else.

One of my friends is now in New Zealand, and he regularly grumbles to me that “If only we were doing those things…” People don’t wanna go back to a different way of working. This stuff works better. And fascinatingly, we’re gathering data to back those sorts of statements up for science nerds like you and I. You shouldn’t just trust what we say, we should also gather data, and you should try that for yourself, and all those sorts of things. Because if we are right, it’s a reproducible thing. It’s not some kind of magic.

[12:07] I love that. Starting from “I’m wrong. Let me figure out what right looks like.”

Yes.

And if you always assume that you’re wrong, even with all your experience and everything you know, you will never be wrong - that’s very weird, because you will figure what right is. You don’t know what right is. It changes all the time. It’s contextual. And most importantly, it’s the people that you work with. They always change. You replace a team member, you have a new team. Someone leaves, someone joins - you have a whole new team. So how do you stick to those principles and how do you promote those healthy principles that everybody respects, abides by, and then magic happens? And I think you were capturing a little bit of that magic with your friends that you use to work with, and you became so much more than co-workers. That’s amazing.

Yes, certainly. And I think that’s certainly the difficult problem. We’re technologists, so we often get lured by the technology, and your joy of discovering the Macintosh and all that kind of stuff… But the hard parts – I know it’s trite, but it is true that the hard parts are the people parts. It’s not that continuous delivery or its practice are particularly difficult. In fact, I would argue the reverse. I would argue that one of the lures of this way of working is that it’s a much simpler way of working, but you have to put some work in and you have to think about things differently, and you have to discard some of the baggage from previous ways of thinking about things. And that is so incredibly difficult for people and organizations to do that these days I make a decent living with helping people to try and make that change… But it’s incredibly difficult.

One of my proudest boasts is that in the organization that I’ve just mentioned, where we’ve built the exchange - it was an organization called LMAX. It still exists, still trading, still running these exchanges around the world built on our technology, and the culture is still fantastic. Funnily enough, I saw a job advert for developers for LMAX cross my event horizon today, and I was reading about it and I was smiling to myself, because I probably could have written that job advert seven years ago when I left… Because the culture is still there, the behavior is still there, and that’s continued. So we started something with the group of people that were there at the beginning that’s been durable in the organization; we established a development culture that has been not only lasting, but has been communicated to subsequent work generations of developers and other people working in that environment… Which I think is fantastic. I’m incredibly proud of that.

It is difficult to get people thinking differently, to change their minds, to jump out of the ruts of their old habits and jump into a new way of thinking about things, which these days is a lot of the thing that I get pleasure from, is trying to help people just think about ideas differently.

So would you say that the people influence these practices and the people contribute to how these practices work, or is the inverse true, where the practices influence the people to behave in a certain way, and then sustain these practices long-term?

I think it’s both. I think it’s a combination. I think it’s a little bit too trite… So we’ve always said things like “What it takes to build great software is you need great development teams.” And that’s true, you need good people. But it’s not enough. I’ve worked with some genuinely brilliant developers, building bad software. And that’s not their fault. A bad process will break good people every time. And so there’s more to it than only that. And this, again, is one of those things that - I refer back to science quite so frequently… Often, developers and development teams and organizations are somewhat disdainful of process, because they assume that software is a heroic exercise carried out by geniuses toiling against the code mountain…

[16:14] But you could say the same kind of thing about science. Science is this terrific endeavor, a human activity carried out by fallible, mistaking human beings… But by organizing their thinking in a certain way, they eliminate whole classes of errors and biases that are built into us through our biology. And if you want to do engineering, which I would count as an application – so practical science, to a practical end, is the way I think about engineering. If you wanna do engineering of that form, which is what I’ve come to think of what we do as, when we do this sort of stuff that I’m talking about - if you wanna do that, then you’re gonna have a better outcome, you’re gonna improve your chances. It’s no guarantee. It’s not going to make a bad development team great. It’s not gonna make a bad development team build world-class software, but it’s going to improve the quality of their work. It’s going to amplify their talents and their skills to an extent so they can do better than they would do without it. And that’s true of world-class developers, too.

One silly example, one of my good friends who I worked at LMAX with - he was the CTO, I was the head of software development - is Martin Thompson, and I regard Martin as at least one of the best, probably the best programmer that I’ve ever met. He’s genuinely brilliant. Really, really talented guy. And I’ve known him for a long time, I’ve known him for many years. We first met in the ‘90s. He’s a bit younger than me, and he was a young man then, and he was very, very good then… But I taught him to do test-driven development while we were at LMAX, and he and I think he’s a better developer now than he was before.

There are some of these techniques that however good you are or however bad you are can improve you. And if we were to be able to identify something that we could class as an engineering discipline, then I think it would have that kind of property, which is really what I’m talking about. I think we are odd in software development circles in that we tend to take terms and change what they mean. I think in nearly every other context that we can think of outside of software, if you use the term “engineering”, it means the stuff that works. In software development we’ve turned it to mean something else, usually something more complex and something that we don’t like very much.

I think words have some power, and I think that I like to use a reasonably strict definitional approach to parsing things to be able to form ideas… And if we think of engineering in the terms of the practical application of science, then it ought to work for software at least as much as it does for anything else. And we have a bunch of advantages in our favor, too. We’ve got one of the most powerful experimental platforms that happens to also be exactly where our software lives, in a computer.

First of all, I really like that you point out this distinction between engineers, software engineers and developers, because it’s a very important distinction. People don’t even think about it, and they use the terms interchangeably… But they mean very different things. Now, I think we can have a show just about that, whether we are software engineers or software developers… So let’s park that there, recognize it for what it is, and move on to the other thing, which I – when you mentioned Martin being one of the best software developers, engineers…?

Yeah.

Software developers or engineers?

Both.

Okay, great. So being Martin is the best software craftsman, software person, TDD made him better. Before TDD, what made him so good, in your eyes?

[20:01] There were lots of things. He’s a very smart guy, which helps. It’s not enough, but it helps. One of Martin’s great talents is he’s got a laser beam focus on simplicity. And one of the things that I learned from developing software with Martin is I’ve really strengthened one of the tools in my toolbox, which is focusing on the separation of concerns. So Martin is absolutely brilliant; he sees the least piece of code that is doing two things and immediately he’s pulling it apart to try and separate those two things, so that each piece of code is focused on achieving one outcome, and then he’s growing it from there.

Martin’s code is almost like reading prose. It’s readable, it’s modular, it’s cohesive… It’s just nice code. It’s also blisteringly fast. Martin’s one of the world’s experts on high-performance and concurrent systems, and he’s widely recognized as such. The people on the Java team occasionally ask him for advice about how to speed things up, and that kind of thing… He’s well-respected in the industry. But the thing that I value most is the focus on the separation of concerns as a driving force in the design that he applies to code. I kind of had a more informal use of that kind of technique. My design skills were pretty good. I’m a decent coder, I’m not a bad developer myself, but Martin was always so focused on it, and I’ve picked that up now. And now I’m always looking “Could I pull something apart here?” and my code is much nicer as a result.

I can definitely see how the TDD would have enhanced that property or that aspect, because it forces you to focus.

Yes.

I mean, you could be focused or not as you write code, but when you write TDD, if you do it properly, if you start with red, which is always the first one… Do it Red, go Green, then refactor. You even have a video on this; we’ll introduce this a bit later, but that’s fascinating to see how simple it is if you really think about it. And one of the things - again, I’m jumping here between things, but there’s so many things I wanna talk to you about… It’s how it took you a really long time of thinking, working in this space of extreme programming, agile I’m sure it’s a big part of that, continuous delivery, test-driven development, to not only hone your skills, but also share your skills and share your knowledge with everyone that wants to listen, or is interested in these things. So in a way, it’s not just the focus, I would say, but also the consistency and the perseverance to stick with it. I mean, it’s been decades and you’ve been sticking with the same thing, on the same hill… I mean, sure, you’re sharing it with others; I know that Jez Humble has a same hill, or maybe a different hill… The point being, these things stand the test of time, and focusing on that one thing long-term is what recognition, admiration, thankfulness, respect - that’s how you build them. Success… Whatever you wanna call it, it’s all related, I think.

Okay, that was a very nice story. Thank you for that.

It’s a pleasure.

Now as we come back, I would like to dig a little bit into the technology that you used back in the day, so the specifics around the CI system, the CD system, the programming language, the frameworks, how that used to work, if you had any project tracking tools, or how you would organize work in the days… And I would like to dig a little bit deeper into those specifics - time it took, which cloud provider you used (if any), where you would run these things, and how that changed over time.

Yeah.

So you’re telling us some very good stories, Dave, about your time at LMAX that you were very fond of, some great people that you’ve worked with, that you’ve been in contact ever since… And you were part of that family, in a way, right? Your work family maybe, your continuous delivery family, whatever you wanna call it…

I would like to dig a little bit deeper into the specific stack/technology that you used at the time, and also how that changed over time.

In the early 2000’s, what did a technology stack look like? The programming language, the framework, the CI/CD, how did you organize work… That type of thing.

I’m more than happy to talk about tools and technology. I should begin though by caveating it, because I’m not very technology-driven, which is weird for a technologist to say… But I think that the tools are secondary, I think that I value design and design thinking more than I value the tools, and I think you apply that in different tools.

Having said that – so my background was largely in the C family of languages. I did a lot of programming in C in the early days, C++ later on… During the early 2000’s, building the points of scale systems and stuff like that, I was doing a lot of work in Java, sometimes C#, bits of other things… Python… I played with Ruby slightly… But mostly, the early days of continuous delivery were mostly on Java projects. One C# project that I can think of… So we were mostly using the technologies around at the time. In the early 2000’s there weren’t many tools, certainly no continuous delivery style tools. In the early 2000’s we’d just started doing continuous integration, really… Or at least it had become popular.

So people have been doing continuous integration for a long time, and I was doing some version of that in the early ‘90s, but we had continuous build… But it was all just done in shell scripts. Tools to manage a build process and those sorts of things didn’t come along until about 2000, and the first one was I think CruiseControl, which is an open source project from ThoughtWorks.

When we built the exchange LMAX, we started off using Java and CruiseControl, the starting point for building our deployment pipeline. And we built a very sophisticated deployment pipeline with different instances of that, using mostly things like Ant files as the glue between stages, and those sorts of things, to encode more complicated bits of glue between the different pieces.

We did a lot of development of sometimes reasonably sophisticated tooling of our own. We built our own deployment mechanism, which was similar in some ways to something like Chef or Puppet. It ran a little agent on the server, and the server called back to some master repository, and it pulled down changes and deployed them for us…

We were doing early things with infrastructure as code. I remember we wanted to be able to version control the configuration of network switches, and the only way that you could configure the network switches was through a firmware admin console, web-based firmware, like you get in a home router, or something like that… So we wrote a little domain-specific language that we could program the configuration of this thing in, which then was backended by – I think it was Selenium, or something similar.

[28:11] Yeah, that’s right.

It would then drive the web app to poke the values into the router. So we were doing a lot of messing around with those sorts of things; very ad-hoc, very – incrementing those as we needed… And as I said, we did some fairly cool, fairly sophisticated things.

One of the things that one of my colleagues, Mark Pryce, wrote at LMAX was still the best version I’ve seen ever of a test distributor. So we were managing a fairly large set of infrastructure to be able to get our feedback fast enough. We had one big repo; we’d put everything in one big repo, and then we could build and test and deploy everything together. And we could be more certain then in our changes; it meant that we didn’t have to worry too much about how loosely coupled. We wanted good design, to make sure our designs were loosely coupled, but we didn’t have to have them independently deployable, the pieces, so we could test them together first.

So we did that, and that was efficient. We ended up with a network - when I left, it was about 48 different server instances that ran our continuous delivery infrastructure, and a dynamically managed compute grid to evaluate these things, which Mark Pryce wrote the software called Romero to be able to manage all of these different instances. So we did a lot of various tooling of that kind.

In that project in particular we weren’t very big consumers of other people’s software, to some degree. We wrote a lot of stuff of our own, partly because of the performance demands, most of the third-party software that we had wasn’t fast enough for what we were trying to do with our exchange, so we wrote our own collections, for example…

Okay.

…a HashMap in Java, for each entry, at a time, creates five objects. So you’ve got five garbage collection problems for every item that you added to a HashMap in those days. So we wrote constant memory footprint HashMaps and stuff like this so we can go fast. So we did a lot of stuff at different levels of abstractions, very low-level technical detail stuff, to bigger picture things.

That’s really fascinating to capture the context in which these ideas came to be… Because while different people may have had similar thoughts, first of all, ThoughtWorks was a consultancy at the time, I imagine…

Yeah.

…so that’s how it started; that’s what I know about ThoughtWorks. So not only you had to come up with these ideas, but you also had to build the tools, which didn’t exist… Open source I think was only just getting started… This was like early 2000’s, so it wasn’t really a thing. Git was only just starting around that time… GitHub didn’t exist, by the way, and we know what an important place that is for open source. Twitter didn’t exist, Facebook didn’t exist… A lot of the platforms that we have today didn’t exist. And the CNCF, the Cloud-Native Foundation didn’t exist either. So you were like – I wouldn’t say avoid, because that sounds negative, but you were in a big sea, with no islands, with no towns, no harbors inside, and you had to figure these things out. And that was really challenging. Not because it was “not invented here.” It was not that syndrome. It just didn’t exist. And the communication was very different at the time as well.

So that must have been very challenging. And even so, you built those things, you shipped those ideas, you shipped the code, and many people benefitting so many ways, decades after you started. 21 years later we’re talking about how this started and how relevant it is, and it feels to me like the whole world, in a way or the other, the whole software world is revolving around the principles that you sat down then.

[31:56] There’s a bit more history than that. So there were people doing good stuff before us, and open source had been around as an idea. It wasn’t as big as it is now, there wasn’t as much choice as there is now, but you know, Linux was around… Linus Torvalds had released that as an open source project considerably before then, and so on. A lot of the ideas that we were building on.

ThoughtWorks was an interesting place. There was a brief spell – I feel privileged to have worked for ThoughtWorks at a time that was very exciting. And ThoughtWorks in London in particular I think was not quite on the same scale perhaps, but it was almost like an agile Xerox PARC; it was a place where some interesting fundamental ideas were introduced.

As I said earlier, Agile at a big scale, Agile in a more commercial setting - we were doing it because we thought we could make software more quickly and better quality software using these techniques, so it would have a commercial advantage implied. These were the reasons why we were doing some of these things.

BDD, continuous delivery, mocking - these are ideas that came out of that office in ThoughtWorks. Several books that are famous, Growing Object-Oriented Software, Guided by Tests, my Continuous Delivery book… There were people that are now well-known in the industry that we were all part of that group of people working there at the time. So that was an exciting place to be. We were consciously experimenting and playing with new ideas, and trying to find better ways of doing things.

The software industry had been through what I think of as a fairly rough time of trying to industrialize it through the late ’80s and ‘90s, applying techniques that people thought would work to make it more effective and productive, and they didn’t. And the Agile movement was a bit of a reaction against that, I think.

What I’m trying to say is that we were building on the shoulders of giants. People that did stuff before. Continuous delivery I think of as second-generation extreme programming. It’s extreme programming, but just with some other ideas added to it, that help you get there a bit more easily maybe, in some ways. But if you’re doing extreme programming, you’re not doing it wrong. That’s a pretty good starting point.

I see a lot of the new systems, for example Argo CD - that’s something which fascinates me right now, how it takes the concept of a pipeline to like a new level, and you have workflows, you have a programmable API, which is the Kubernetes, and this control plane where you define these custom resources, and then things happen, all these relationships emerge between them… Event sourcing is another big thing, which maybe is not as popular these days; I don’t know. I’m not too into it, but I keep hearing it; it keeps coming up. But there have been so many CI/CD systems that appeared in the last five years, which seem to have exploded recently. There was Drone CI, there was CircleCI, there was GitHub Actions, which wasn’t a CI to begin with, but it became one over time… And all these other systems. I mean, Jenkins - I think that came after CruiseControl, I remember…

We switched to Jenkins at LMAX; we refactored our pipeline to use Jenkins later on.

Interesting. So there was this transition. And now we’re in the era – I think it’s almost like the third one, which is a cloud-native one, where we have so many projects. I’m not sure whether you looked recently at the CNCF landscape, where you have all those projects. There’s so much things there, and you can’t even keep up with all of the updates, that’s how many there are, never mind try them. It’s impossible. There’s not enough days in the week. Or hours in the day. You know what I mean.

Both.

So I’m wondering, how did the cloud-native landscape shape your ideas of continuous delivery? Was there an impact of that, or was that happening in parallel? What influence, if any, do you feel coming from there?

[35:45] I think that the gestation of the cloud was kind of in parallel with the starting points of continuous delivery. We published our book in 2010, and that kind of put a name on these practices, I suppose. And people these days talk about continuous delivery and continuous integration and all these sorts of ideas… I think we helped to popularize that through our book. But we’d been doing it for several years before that. And certainly in the early days, the cloud wasn’t around, so all of the projects that continuous delivery began with were cloud projects.

I think that the cloud makes some of the continuous delivery thinking more obvious. I mean, you’d be absolutely insane to be an organization like Amazon or Google and to manually configure your servers. It’s such a bizarre idea; it’s just laughable. You couldn’t do that. You’re gonna automate that, unless you are crazy.

So ideas like infrastructure as code just seem obvious in the cloud. And they didn’t always seem obvious to other people. People thought we were strange when we started automating those things, and making our servers in our data centers more repeatable and reliable. There’s that kind of stuff.

If I’m honest, I don’t think that cloud had a huge impact on the kinds of projects that I was working on during that time. Certainly, up until after the continuous delivery book came out. Of course, like everybody else, it has an impact on me now in the way that I think about these things and the way that I advise my clients how to approach solving problems.

I am, I suppose, an old school developer. My formative years in software development probably predated the cloud… The cloud is obviously a good thing; it gives us lots of opportunities. It commoditizes compute, but to my mind, it’s not a fundamentally defining thing. It certainly changes some of the dimensions of design that I care about, the way that I would think about design…

One of the things that seems to me that changes is the economics of the design from an architectural point of view. In the old days, when we were doing stuff in our own data centers, we were probably more worried about managing storage, because storage was expensive. And that’s largely become commoditized. And the price per bytes of storage has kind of just dropped through the floor with the introduction of Moore’s Law and the introduction of cloud-based services. The big difference with the cloud now is that the unit of cost is really around compute. That’s certainly if you start thinking about serverless things. So that ought to change the way in which we apply our design thinking. For example why bother normalizing data anymore? Why not just shard it out and make everything separate to optimize the compute cycle? You could do that. We could have processes sucking in data and just allocating them out in a way so that they’re all more parallelizable… And I think that’s kind of interesting, that it has those sorts of things. The easy access to be able to spin up some compute resource or storage resource or whatever else is fantastic… And the ever-raising of the bar of abstraction that the cloud services add is kind of interesting. And I think there’s much more to come.

One of the things… We were talking about the continuous integration and continuous delivery tooling - I don’t think we’re there yet. I think there’s more to do. I would like to see tooling that’s more opinionated. I would like to see tooling that just – bang, gave me a deployment pipeline. If I follow the rules, it’s just going to run my unit tests, run my acceptance tests for me and deploy into production. That’s doable, it seems to me. We could do that. I’m hoping to see the continuous delivery cloud vendors do more of that kind of thing - be more opinionated.

[39:45] One of my favorite technologies - we were talking about tech earlier on - is Gradle for build systems in the Java space. And one of the things that I always loved about Gradle is that if you don’t care, if you just are willing to buy into its model, you can write your build script in one line. You can just say “I’m doing Java” and it’ll do it for you. It’ll compile to Java, it’ll run the tests, it’ll do all of those things for you. But if you want to override almost any behavior, you can do that, too. It’s a whole programming language built on top of this well-designed domain model for builds, is what Gradle really is.

I like opinionated software. I like opinionated software that says “Do it like this”, and if you don’t like it like that, it’ll get it out of your way really quickly. I’d like to see more tools like that, because I think there’s a tragedy of the commons kind of thing goes on a little bit. If everybody has a choice, everybody’s rediscovering everything from scratch, and I think we ought to be able to build a little bit more things and the cloud is one of those things that is doing that in some context. It’s not doing it enough to my taste for build systems.

So I am currently in the middle for a piece of work that I’m working on to demonstrate how to build deployment pipelines, building a little sample application using GitHub Actions. It’s alright. It’s nice. I quite like GitHub Actions, it’s okay. But it’s too fiddly. I’m fighting with it, trying to get my Docker images to communicate with each other so I can run my acceptance tests, and this sort of thing. I’d like something that just worked. I’m not really interested in that part. I’d like something that just works if I wanna do something simple.

That’s really interesting, because I have seen – so these trends have been emerging in… I think build packs came closest to that, where it would automatically detect your application and it would know what to do; what is the build step, what is the run step, what is the package step… So it had this stuff built in. I think Heroku were the ones that made it popular, Cloud Foundry - that was the enterprise version of that… I know that there’s other newcomers. Render is one of them, Fly I believe tries to do something similar… The point being there’s some good concepts, but I don’t think they’re standardized. The one concept that was able to be standardized was the Kubernetes API, and it’s amazing because it was unified API, and you can have almost anything via the same API. Do you want, for example, a VM? Well, you’re programming in the same API and it spins up a VM. Do you want a SQL instance somewhere in some cloud? It’s the same API. You can start doing arbitrage, you can start doing some really clever things.

There’s even like a control plane which controls all the Kubernetes deployments, all that DNS, your CDN, all the things. So I think that is fascinating. Could we have something similar for CI/CD, this concept of a pipeline? I think we should. But I don’t think there’s any one clear winner as to how to approach it. It’s just YAML, really. And that’s okay… I mean, Tekton CD - that’s trying to do something; it’s using the same Kubernetes API to declare your pipelines and your inputs and your outputs and how that works… But you’re right, it’s still fiddly. It’s almost like we need the next building block. And I think this is something that happened with the cloud - you had this fixed compute before, which was your cap ex; you’d buy the hardware, you’d invest and that’s it… You know, you’d spend the money, so you have to use it… But it was very difficult to increase, or very slow to increase. Then the cloud came and you could have almost like infinite capacity. Do you want 1,000 CPUs? Two minutes later you have them. Do you want petabytes of SSD storage? You have it. And then storage wasn’t an issue anymore. But it was difficult to scale that down to zero, and that’s where serverless came.

Serverless - you can have an infinite capacity that you run for milliseconds, and then it spins down again. And that’s a very interesting take. But you’re right, how do you declare the pipeline or the thing that kind of controls all of these things that need to happen, because shipping the code out there, just part of it - you have tests, as you mentioned, you have the build, you have the dependencies you need to resolve… And that pipeline would be really big. So if you had to imagine it, it’d be massive. How could you declare it? And I think there’s a lot of variance in pipelines.

[44:11] There is. But if I’m honest, part of why I’m complaining and being a grumpy old man about this is I’d like people to take my opinion. [laughs]

Right, okay. Same does everyone else.

Yeah, of course. [laughs] But one of the things that I wish people had picked up from the continuous delivery book that they didn’t was that the book outlines a pattern for what a deployment pipeline is. Jez and I each wrote equal amounts of that book. I started off writing the beginnings of the pipeline bits. He contributed on top of it. But when I wrote the pipeline bits, what I meant is that I think this is the starting point for a pipeline. So yeah, absolutely you vary from there, but I think it’s a bit like patterns. I think that if you wanna write software, what do you want? Well, you want fast feedback during the development phase to confirm that the code that you’re writing is the code that you think it is. And then you want to be able to check that that code works as a system, is deployable, is configured correctly, it delivers value to customers. So there’s a separate, different focus of testing that you need to establish that. And then there might be other things that are optional; maybe performance testing, security, whatever else.

So you could imagine – so my minimum deployment pipeline is pretty fixed. You start off with a commit stage, we give you fast feedback, it’ll search your coding standards, runs all of your unit tests… If it succeeds, it builds a release candidate. That release candidate is a deployable thing. You deploy it into an acceptance test environment, you run a bunch of BDD style acceptance tests against it to check that it’s deployed correctly, it works correctly, it does what users want it to do… And then you can deploy it into production, because it’s deployable.

That’s my minimum deployment pipeline. I would pay my own money to be able to have that at the push of a button for a Java project or a Python project, or a C# project, or whatever it was I was doing… And I can see no reason – in fact, I have built that internally in organizations in the past. I must confess, when I started my own business and working for myself, my ambition was to earn enough money to pay me to have enough time to write some code so I could open source this model like this. And I earned enough money, but didn’t have enough time. [laughs]

I think that worked out really well, because what you did manage to do was spend a bit more time on those videos that I’ve alluded to in the past… And that is actually how I came across, like “Oh, Dave Farley? I’ve heard that name. Continuous Delivery? Okay, I haven’t read it, but I’ve heard of that book, and now I know I have to read it.” Based on what you’ve said, there’s some very important information there, which I need to get… So that’s the first step.

The second step is you were able to capture some concepts in very simple terms, in very good terms, and these concepts stood the test of time. So that’s out there, it’s super-valuable, and it will continue being valuable for many years to come, I’m sure of it.

If someone’s listening to this and wants to do this, that would be really interesting… Like, what would a GitHub Actions pipeline look like, for example, that resembles Dave’s ideal pipeline? I think in your videos you even have – like, that graphic keeps coming up. Do you have a specific course or book that talks more about that pipeline?

I do. So the Continuous Delivery book that we’ve been discussing talks in broad principles about continuous delivery, and the deployment pipeline is kind of the core of the book, but it’s not all the book’s about. I have another book that was released this year on Leanpub, which is more of a focus manual on how to create deployment pipelines, and this pattern that I’m describing really, I suppose. So what are the key stages as I see them, and what should those be doing.

[48:06] It’s a pattern, so you would expect it to evolve over time and to morph into different shapes, so you take it parallel, and so on. But it seems to me there are some fundamental things, is that your fast feedback on the technical quality of your work as a development team, and then you need confidence to know that your software is releasable. And the latter involves tests that are more expensive to run.

So you need to think about this as kind of a machine, a parallel computing algorithm, if you like, so that you can kind of trade-off - getting the fast feedback, and then moving ahead in confidence that you’re likely to get good feedback from later stages. So if thinking in those sorts of terms helps you model it… I think that’s a good pattern, and I would make that the starting point.

It’s almost like a template for pipelines.

Yes.

If there was a template that captured these important elements that need to be present, and then from that, it’s almost like an RFC, and then from that you have a specific implementation which is for maybe a specific CI, and then you have variations of that implementation in whichever CI it is.

Yes. And there are key stages that I would a system of any complexity would probably want to parallelize and grow, to get fast feedback… What people took from the idea of continuous delivery was – when people think about deployment pipelines, I think what most people probably interpret that to mean is basically a build script that can deploy stuff at the end. And it’s much more than that. There’s more to the model than that, in my mind. This is one of the things that I can be definite about; I’m usually not definite about things, but in this case I can be, because I invented the term deployment pipeline, so I can be definitive about what I meant when I said it. A deployment pipeline goes from commit to releasable outcome. That’s its job.

If at the end of a deployment pipeline you’ve got more work to do, it’s not finished. It’s not a deployment pipeline. The objective of a deployment pipeline is go commit to a releasable outcome. It doesn’t mean you have to necessarily push the thing into production. That depends on the business, whether that makes sense or not. It makes sense to go frequently, but you don’t have to… But working so your software is always in a releasable state is the way that I tend to describe continuous delivery; the deployment pipeline is the thing that determines releasability.

So that’s what we’re trying to get to… And if you’re thinking about that - so what does that then take? Well, it depends on your system. But at a minimum, you want to know that the software works, it does what you think it does as a developer, and is deployable and does what the users want. As an absolute minimum, you must want to answer those questions in some way before you deploy it to production. So that’s my minimum starting point. That’s where I would start my template from. And then you could optionally plug in bits that would do performance testing, host where you put your performance testing.

So part of this is really to address the problem of how do you get organizations to start buying into this, and what you wanna do is that you wanna make it really easy to do the right things, or what you think are the right things, and possible to do other things. That’s my approach. So I would like it to be absolutely trivially simple that if you are willing to accept some small constraints on the way in which you organize your code, then I will be able to build it for you, deploy it for you, run all of the unit tests, because you’ve put them in the right place. And then once I’ve deployed it, I’ll be able to run all of the acceptance tests, because you’ve put them in a different place. And if all of those tests pass, I’m gonna give you something that you could deploy into production. And that’s pretty trivial and doesn’t really constrain you very much. All you’re asking is “Tell us what you wanna build.” You know, plug your build script in here, that’s fine, build the pieces that you want, but tell us where the tests are, and we’ll report them to you.

And now, as a developer, I’m gonna press the button, get my template out, start building my project against that template, and my deployment pipeline just starts functioning. Instead, at the moment, with all of the technologies that I’ve tried so far, I pretty much have to go through that exercise every time to set something up.

Okay, let’s do this… My assumption is that the pipeline that we use to push Changelog.com updates out is wrong. So my assumption is wrong. This is what the pipeline is, and I would like you to tell me what your thoughts are on the pipeline.

Okay.

The pipeline, whenever there’s a commit to the GitHub repository, it pulls down the code, it runs a build… By the way, it’s an Erlang-based project, Elixir-based project, so it has to fetch some dependencies, compile the code… So that’s the first step. And then it fans out into two other steps. One of the steps is to compile all the assets, and these are the static files - the CSS, the JavaScript, all those things - and the other one is to run the tests. The tests are mostly unit tests, but also some integration tests, because it does use a database… So it’s like some hundreds of tests. It doesn’t take too long to run, but a few minutes later it gets to the last stage. If both stages passed, it fans in; so there’s a small fan out and fan in… And the last one is to build the artifact that is deployable. And in this case, it’s a container image.

All those things put together, it takes maybe up to ten minutes to run. it can be a bit slow, but we won’t get into that. The point is that the pipeline ends at publishing this artifact to a repository. And this is like an artifact repository.

In production, we receive notifications, “Oh, there’s a new artifact.” And the production system - in this case it’s Kubernetes - knows how to pull the latest version down, how to do blue-green deploy, and there are checks which make sure that the new version actually works. It connects to the database, the health checks pass - all those things pass. Then it gets automatically promoted to be the new version.

All this happens within 15 minutes. A lot of it is just like slow workers… Anyways, it’s a lot of free infrastructure there, especially on the build side. But within 10-15 minutes every commit goes out into production. What are your thoughts about this pipeline. I’m assuming it’s the wrong one based on your description…

[laughs] I think all of the things that you’ve said - I can’t be too critical of it, because it’s working. So I’m nothing if not a pragmatist. Let me critique it, nevertheless.

[56:10] Yes, please.

So I think what you’ve said - I think the implications of what you’ve said is as a developer I don’t know that I’m ready to move on to something new until after about ten minutes, when you’ve run all of your tests. That seems a little bit slow to me. I suppose it depends how big or complicated the code is. And part of the reason why it’s slow is that you’re conflating different kinds of tests. So the reason why a deployment pipeline is called a deployment pipeline is weird, and it’s all my fault. I’m a software/computer nerd, and what this reminded me of when I came up with the idea was instruction pipelining in Pentium processors.

Okay.

So when I say a pipeline, I don’t mean a straight line. What I mean is an instruction pipeline. And an instruction pipeline in a Pentium processor was a branch prediction algorithm. So at a point in which you come to a branch in the code, a Pentium processor will start three threads of executions, three processes internally. It will start evaluating the statement and the condition that you’re interested in, and it will also in parallel start executing what happens if that condition is true, and what happens if that condition is false. And then once it’s finished evaluating the condition, it will discard the computation that wasn’t useful. So it’s made progress. It’s made progress in parallel with carrying out the condition.

At the point at which a developer commits a change, my recommendation for a way of working is that you sit and you wait for the results of the tests. And at that point, what I’m looking for is a high level of confidence that if all of those tests pass in the commit stage, then everything else is gonna be fine. If my tests pass, I’m gonna move and I’m gonna start working on something new, with about 80% confidence that 80% of the time all the rest of the tests are going to be OK now. Now I can afford to run more slower, more complicated tests, because I’m making progress in parallel with executing those tests. And 80% of the time (or better) all of those tests are gonna pass, because I’ve got high confidence, because I’m doing the fail fast thing of testing things.

So I’m gonna run very fast, very efficient tests in the first stage, in the commit stage. It’s gonna be focused on a really technical evaluation of what we’re doing. Then I’m looking for the deployability of that. One of the things that you said was that the first time that you actually deploy the software is in production.

Yes. It goes straight into production, yes.

Yeah. So how often is that a problem? Does it ever cause a problem, or does it always work?

It always works.

Then I can’t critique it. For other kinds of software though, I wanna test the deployment of the system. I want to test that that works… Because it changes over time. If you introduce a new service, or something that’s different, then you’re gonna be evolving that over time, so I’d like to be able to evaluate those kinds of things, too. I’d like to be able to test the configuration of the system. How does it work if it’s in a thread pool, or whatever else it might be; I wanna test all those sorts of things, too. So I guess partly it depends on the consequence of things going wrong, how far you take that.

These days I make a living as a consultant, advising usually large companies on how to improve their software engineering practices… And one of the companies that I worked with was Siemens Healthcare. They’re building machines that can kill you if they get the wrong – you know, these medical devices in hospitals. There are chances that you don’t wanna take with that kind of software, so you wanna be more thorough in your approach to evaluating those kinds of systems than what I would for other kinds of systems. So it probably does vary.

[01:00:07.16] So I must say, I can’t really critique your project very well, because it sounds very good. It’s light years ahead of probably whatever average means in our industry.

This was really good, because even talking to you about how it works, I realized why certain layers are so slow. Why this takes 15 minutes. And it’s not the tests. The tests run in maybe 15 seconds. The tests are really fast. But it’s all the caches, of dependencies, of pulling things down, of running updates, of compiling things.

When we run this in the CI, there’s a queue. So your jobs may be queued for maybe 30 seconds or a minute. And you have multiple jobs, you have containers, you have to pull down images that may or may not be on the node or on the host where they run. And all those things, like the cache misses, can mean 30-45 seconds… Which in the big scheme of things it’s not a lot, but they add up, because you have so many layers.

What is the impact of something not working? Well, when we deploy into production, the reason why it’s slightly slower is because the first thing that we do is we back up the database before we’re running the migration, so there’s a full database backup every single time a new version starts. We backup all the assets to S3, so if we lose everything, that’s okay, we can restore the whole thing in 30 minutes. And in front of the app - it’s a monolith, by the way, and we didn’t have time to discuss about microservices and monoliths (another time, I’m sure), is that we have –

I have a good video on that topic.

I know you do. We will definitely discuss that next. And in front of the website there’s a CDN which serves all the content cached. So if the origin is down, if the app is down, that’s okay. Everything is cached worldwide. So we serve the cached content. So the impact on end users is none. They see the old content, but it doesn’t go down. So the uptime is always 100%, because it’s never down. It’s distributed across the whole world, again. And all that – there’s like a complexity in the system which makes certain things slow.

But anyways, I would love to talk more about this, but we’re running out of time, and this just shows how much we have to talk about… I would really like to talk about your YouTube channel next. So what made you start your YouTube channel? By the way, for those that don’t know about this amazing – it’s my favorite YouTube channel right now; it’s called Continuous Delivery. It’s Dave’s new YouTube channel. And week on week, every Wednesday, he publishes a new video. It’s one of the best tech videos that I’ve seen. They’re short, 17-18 minutes, but there’s so much information there. I highly recommend you check it out. So Dave, what made you start this YouTube channel?

The simple answer is it was Coronavirus.

Finally, there’s a positive… [laughter]

So it’s something that I kind of had in the back of my mind for a long time. I am approaching the end of my career. I’ve done a lot of interesting things… I am opinionated, as you can probably tell from my conversation, about software… And I think that the teams that I’ve worked on, I’ve found some things that are worth spreading and worth hearing, at least. You can dismiss them, you can disagree with them; that’s absolutely fine. But I think – when I’m being grandiose, which I sometimes am… When I’m being grandiose, I think that we are on the verge of discovering what engineering for software might really mean. That is in the same sense as Elon Musk blowing up Starships in Texas. It’s experimental. It’s about learning and discovery and trying out ideas and focusing on the skills around that kind of thing.

[01:03:59.00] I think that if people just did that, then they would find a dramatic, experience-changing improvement in their experience of building and delivering software. You genuinely can build better software, faster, doing these techniques.

Sometimes I err on the side of being too prescriptive about some of these things, possibly, but I wanted to start talking about those things. And I’ve been talking at conferences for some years, working as a consultant for some years, helping people to do this kind of thing. And I had in the back of my mind it’d be nice to play with a YouTube channel one day. The Coronavirus happened, we were in lockdown, and at the time I was travelling around the world, constantly, as a consultant. And that kind of fell of a cliff. I was at home and I thought “Wow, what am I gonna do now?” So instead of writing software - I should have built the system that we’ve just described. Instead of doing that, what I did was I started a YouTube channel, and that’s been a fascinating, engaging, delightful experience on the whole.

Sometimes some of the comments are not quite so delightful, but usually they are. Mostly they’re lovely. And I’ve had a fantastic time. I think it helped to keep me, my wife and my son saner than we would have been otherwise through the process.

We have released a video every week at 7 PM on Wednesday since the start of the pandemic, and we haven’t missed one yet.

Now, I have to thank you again… I find myself thanking you so much, because those videos - they are like a breath of fresh air. There’s so many videos obviously on YouTube; it’s massive. For me at least, and for our family, it’s like the new TV. We use YouTube way more than anything else. Netflix is there, Apple TV is there, but it’s YouTube by far. And I don’t know how it happened, how I came across your videos, but they were so refreshing. They were simple, they were to the point… And it’s not just me. If you look at the comments, the more positive ones, that’s what the majority is saying. The way you capture these principles and the way you convey them is so good, and it’s so simple… It’s like “Yeah, it makes sense.” At the end, after you watch a video, like “Oh, I wanna try this out.” It just makes you think.

I’m sure that some of the information that you convey - it will not hit home until a few months later, or maybe even a few years later. It’s simple, but there’s so much there. And my favorite one - you keep mentioning Elon Musk, by the way… If you know him, or if someone that knows him is listening, I really want to interview him, because I think he’s the embodiment of Shipping It. He’s literally shipping the human race to a whole new level. I’m so fascinated by him. So my favorite video is the “SpaceX and software engineering | How to learn” on your YouTube channel. The link will be in the show notes, by the way.

Now, I try to limit myself to three. This was my top. The other one is “How to build quality software fast.” That shipped yesterday. I mean, if you’re paying attention, there are videos that you’re just publishing which are top, so they are getting better, in my mind… “Why CI is BETTER than feature branching” - I would love to talk to you just about this. I’m a big believer in single branch, push straight into main/master (however you wanna call it). I would recommend main, the main branch. If you use Git, power to you; if you use something else, that’s okay too. As long as you have single-branch, you continuously integrate, you continuously deliver - that’s the place that you wanna be in, because you’re trying to learn. And you will be wrong, even when you think you’re right. So better think you’re wrong, and start thinking that you’re wrong, and it will be good. Trust - not me, trust Dave, because that’s what he’s saying.

Don’t trust me either. Try it out. [laughs]

Try it out, yeah. That’s the best one. And “What’s wrong with the state of DevOps?” That’s the one that I want to watch again, because that’s another very good video. We don’t have time to talk about the specifics, but if anything, I feel like we should have another interview. We’re just finishing this one, so I’m not sure how that’s going to work, but I would definitely like to get together again - maybe this year; if not, next year is fine as well - to do another check-in, see how it’s going.

Right now, you had like 53,000 subscribers, or 54,000. That was yesterday, by the way. It changes day to day. So let’s see how many subscribers you’ll have next time. When I started watching, you had like 5,000, 6,000, and then it just exploded. So yeah, the response has been positive. I hope you’re pleased with it, because I’m very pleased with this YouTube channel… And thank the pandemic that it happened, right? [laughter] It’s the weirdest thing to say, but it’s the truth. If it wasn’t for it, we would never have this YouTube channel.

Yeah. Well, it doesn’t compensate for the bad things… But it’s been a lot of fun, and a lot of pleasure out of making the videos, but also engaging in the comments and talking to people about ideas, which is fantastic. It’s all that any of us can do.

I’m interested in your selection. They weren’t the ones that I expected, to be honest… They’re not the most popular ones on the channel, some of the ones that you’ve mentioned… But they are ones that I like. I was slightly disappointed by the take-up of the SpaceX video, because I thought that was a good video. I liked that one.

Based on what I was saying earlier - I was saying that some of the things that you share, I don’t think people realize how valuable they are until maybe a few months or even years later… And I think it depends on experience, it depends on what you value. But I see, for example - SpaceX is such an important thing. Tesla is such an important thing. Not the things that they do, it’s how they approach it. How they’re able to build – that’s what fascinates me, and I know it fascinates you too, because you mention it in the videos. So which are your favorite videos?

Oh, last week’s, of course. [laughs]

Last week’s, okay. That was a good one. It’s “How to build–” No, that was this week’s. Which one was last week’s?

I just made always last week’s. [laughs]

Oh, I see. Okay.

No, there are some that I’m proud of. The early ones - I think there were some good ideas in the early ones, but my editing skills have improved significantly, and my equipment has improved a bit. It’s still not very professional, but it’s good enough now that it’s not gonna make people run away screaming.

I liked the SpaceX one. The microservices video in which I talk about the problem with microservices is a good video… I was pleased with last week’s video, which was “CI is better than feature branching”, which is just talking mess in informational… So I’m trying not to do it in an emotional way. I’m trying to do it just based on information and just thinking about, you know, two pieces of information in two places, that are both being changed (they start off as copies), they will diverge. And the longer the time, the greater the divergence, therefore the more work to put them together again. That is incontroversably true. And so continuous integration, continuous delivery is about trying to minimize that time, trying to shrink that time down so that you’re taking less risk with the changes. So there’s ideas like that which I enjoy, and I enjoy trying to find a simple way of describing sometimes complex ideas.

I think that’s a very good thought to end on, because it’s a very profound one. I think people need to think about that. The simplicity in complexity, that I think everybody should strive to look for. Martin Thompson - I think he was a bit of an inspiration there as well. So keep improving, be wrong. Start being wrong, and maybe you’ll be right. Who knows…? Nobody knows.

Check out David’s YouTube channel, it’s really good. It will be worth your time, trust me. And Dave, it’s been a pleasure. Thank you very much for making the time, and I’m looking forward to the next one. Thank you.

Great. Thank you very much. It’s been fun.