To build, or to buy, that is the question

Welcome to Go Time! I’m your host, Kris Brandow, and today we’re gonna be doing something a little bit different than what we usually do. This is the first episode in a mini-series that we’re gonna be doing on maintenance. Today’s topic is Build vs. Buy.

I’m joined today by Angelica Hill… Hello, Angelica. How are you?

I’m very well.

Amazing. And I’m also joined by Johnny Boursiquot. How are you doing today, Johnny?

Hello. I’m doing quite well, actually.

Perfect. Alright, so - since no one here needs introductions, we’re all hosts on this show, I think we can just jump right into it. The idea behind this mini-series - I was thinking the other day about how important maintenance is for our lives, for our software, for everything we do, but in our industry we tend to talk a lot about innovation, about new things, about greenfield projects, and we rarely talk about maintenance. So that’s the kind of framing for today’s episode - talk about things that we have talked about before, but within an eye toward maintenance… And we’re starting with that classic Build vs. Buy debate. Let’s just start with setting the scene - what does build vs. buy mean to you? Johnny, how about you start?

Build vs. buy. Well, to me that’s sort of a layered question, because you’re gonna be asking yourself this question almost at every layer of whatever technology stack or software or product, whatever it is you’re building; you’re gonna be asking yourself that all the way down the stack. So we can start with “Well, we are a company that focuses on technology X. Should our focus be only on technology X, and whenever we need technology A, B and C, we go out and buy those, as opposed to building those things, because those are not the things that make us money.” So that’s sort of the question you’re asking yourself right then and there.

And then it can go as low as a developer making a decision on a dependency or a library they wanna bring into their project. We don’t often realize that, but when you bring in a dependency into your project, you are in effect buying that with your time. It might be free and open source and that’s all good, but you are paying for that in some way, because it’s gonna be the maintenance of that thing if for whatever reason it breaks, or maybe the maintainer decides they wanna go do something else and now you’ve inherited the maintenance, the cost of actually having that dependency in your software. So it’s a question you’re gonna be asking yourself, or you should be asking yourself really, throughout the lifecycle of whatever it is you’re working on.

Yeah, I definitely agree with that. Angelica, what does build vs. buy mean to you?

I mean, I might be giving away my product mind here, but as soon as you say “build vs. buy”, I just get dollar signs in my brain… [laughs] But I think Johnny touched on something that is just kind of quintessential to the decision you make when you’re thinking “build vs. buy”, which is cost… Whether that be, as first came to my mind, your dollar signs, or cost in terms of your engineers’ time, whether that be building that solution, whether it be maintaining it, whether it be going back and forth between a third-party vendor.

I think one of the things that I spend a lot of time thinking about when we’re making decisions on a build vs. buy is “Does this third-party solution, does this external dependency, this open open source project (as Johnny chatted about) have maintainers, have people we can go to? Do they have an on-call engineer if something goes wrong? Are they gonna be a partner that we want to be tied to, in terms of having confidence in that solution?”, in the same way we would have confidence in a build solution, where we would be putting our faith in our engineering team and our colleagues being able to help us out if things go wrong.

Yeah, I like that perspective, too. A little bit more on the side of the business, but equally important; we have to consider more than just the technical aspects of building software when we create something. We also have to think about the knock-on effects, because that contributes to maintainability at the end of the day. It’s not just about “Can we maintain the actual code that exists?”, but “Does the organization have the bandwidth from a product perspective or a project management perspective to actually maintain the project from that perspective?” If you don’t have enough product people to maintain the features you’re building - well, that’s a problem, so that has to be brought into the build vs. buy equation.

[08:16] So I kind of wanna talk about the initial stages of creating a new project, because there’s obviously a difference between talking about building vs. buying for something brand new that you’re just starting doing, vs. something you already have around. So I guess from that initial perspective, we have a new project - where do you tend to fall on that line of build vs. buying? What nuance would you also add to that perspective?

For me personally, it all starts with the requirements. The technical requirements, what are we gonna expect from this thing in the short term, in the immediate, but also thinking long-term. Eventually, is it something that we wanna be able to scale up? Is this something that we’re gonna expect to handle a large amount of load? Really making sure that when making the decision and going in from day one, knowing exactly what your expectations are gonna be, because then you’re gonna have a clear shopping list to go through and think “Okay, this buy option - is it gonna fulfill all my use cases?” If no, then that’s a clear – like, that’s not the right solution. If yes - great. That gives you space to talk more about the things that aren’t as essential, that might be added.

There’s a sort of default stance you can take with build vs. buy. Again, speaking from the point of view of someone who has to – depending on what stage you’re at, and really what environment you’re in, and what company you’re working for… Is it a startup, is it a large enterprise? There are some constraints that you’re gonna have to factor in, be they regulatory, or compliance, or [unintelligible 00:09:48.15] or any of those laundry list of things, depending on the industry you’re in. You might have the ability to do certain things, you might be able to use something off the shelf, whether it be something you paid for, or something that’s open source and maintained by a large community.

The default stance is to always buy. You need to kind of be contextual, but I think if you are building a product, you can sort of - depending on what your circumstances are, you can take a default stance and say “You know what - we are in the business of making web applications for, say, shoemakers.” So perhaps you’re not in a position to go build a database specifically tailored for shoemaking apps. So trying to go build a custom, tailored database to solve that particular problem - you’re kind of shooting yourself in the foot, because you’re really wasting your engineering resources and talent and money. You’re kind of wasting your effort and time. Specifically for a startup, time is against you. Maybe you’re running out of money, or whatever it is; you need to get product-market fit quickly and start generating revenue, or whatever the case may be… Then really, your focus is on “Okay, what is the focus that we need to have? What is the discipline that we need to have to get to profitability?”

If you’re a larger organization, then it’s not the money that’s driving that decision, it’s more of – all these other things I’ve mentioned previously; what are the constraints? What are the requirements we have on this organization that perhaps limit our choices on the build vs. buy thing.

You find that in a lot of large corporations, and if you’ve worked for one, you will know that a lot of times you end up having sort of this “not invented here” syndrome. You might know what I’m talking about… Whereby “Yes, we can go pick up this product from this startup, which is solely focused on delivering this particular kind of product. That’s what we need internally. But because of reason X, Y and Z, be it architecture, be it compliance, be it whatever, we can’t simply buy this and integrate it. We would be better off building something close to it, which is never really gonna be as good as the off-the-shelf solution”, but those constraints kind of force you into a “Let’s glue some things together so we can arrive at a solution.” So it’s never cut and dry, but it’s more contextual.

Yeah, I definitely feel like big companies tend to fall on the extremes… Because I’ve worked at some big companies that also do the opposite, where it’s like as soon as you need something, it’s like “Okay, let’s go find a vendor to provide this for us. Let’s go find someone that has already built it. There’s no way that someone hasn’t built this thing before”, which - I do think it’s always a trade-off of like “Is the thing that you’re trying to acquire, or the thing that you want to do - do you have the requirements set out for it?”

[12:17] Now, going back to what Angelica said, because I think that’s a crucial aspect of this conversation as well. So I feel like whenever I’ve been in some of these conversations, especially when it’s about commodity components, as you mentioned, like a database or something like that, either it jumps out as like “Of course we can’t build our own database”, or we start talking about all of the options that are out there without really coming back to what is it that we’re trying to do at the end of the day, what problem are we trying to solve.

And talking about commodity components as well is that the area where this conversation sits most of the time, because I think – there’s obviously the core part of your business is something that you are going to build, at least in the software world, because that’s why we’re here, that’s what we’re doing; we are trying to build a product to sell to people. So we’re not going to, in most cases, buy our main thing. But you know, when it comes to building something that is commodity, there has to be a line, there has to be a point at which it starts making sense.

If we’re talking about a two-person startup - it doesn’t really make sense to build your own database. But then you get to the size of a company like Google or Amazon… They have lots of databases that are built internally and that people use, and no one kind of bats an eye at that sort of thing. In fact, sometimes we expect that sort of thing to happen.

So I would kind of pose to each of you, where do you think that that line is? At what point should an organization start considering building something that is a commodity, that is available out there, but that they could perhaps get more benefit from it if they’ve built it themselves, versus just buying or acquiring it from somewhere else?

My view is you should always be thinking about the decision of build vs. buy, in anything you’re building. To go to your point around kind of if you have something that is [unintelligible 00:13:59.11] is essential, it’s something that isn’t anything specifically unique to your use case, your problem, I would also say in my experience there are instances - your company will get to a certain size (we’ve had this at the New York Times) where there is not a buy option for the scale and the load and the amount of data or uses for example when it comes to breaking news that can support your needs. So you have to build, and you have to build something that is unique to you, although it is a “non-unique” use case.

So I think if you’re clever and you have the time to do that kind of research which is a big point here is if you’re on a deadline, if you’re really pushed against the clock, you might not have the time to really go granular and do a two-week spike investigation on this “What would happen if we build vs. buy?” But if you do, I would strongly encourage to always think through your options before making that decision.

But I would say there are definitely forcing mechanisms, and I would say – to go to your point, Kris, I think you’re right in that big companies often, especially when they’re on a time crunch, will push for buy, because they want their engineers working on new stuff, unique stuff, things that aren’t in their minds not unique, if that makes sense. But Johnny, I’ll pass over to you for your strong opinions…

Well, I say strong because over the years my stance on that has changed. In my earlier days as an engineer I was basically always gung-ho for building, because I always found the problem spaces that I happened to be in to be interesting, and I always wanted to build stuff. And even if there was something out there, I’d look at it and I’m like “Ah, I can do better.” Kind of letting myself be carried away… You know, “Oh, Twitter - that’s just nothing but an ever-scrolling feed. You can refresh things in the background. That’s fine, you can build that in a weekend.”

Obviously, as time goes on and you grow as an engineer, you realize “Okay, so there’s more to a product than just the engineering of it.” There’s always more – a lot of things you don’t know you don’t know. So as you mature as an engineer, you sort of realize “Okay, there are always constraints, there are always trade-offs. What am I trading off?”

[16:06] So to your question, like “When is that threshold?”, for me, I’m gonna borrow a term from the cloud industry that basically says “As long as you have things that are considered undifferentiated, like commodity…” Like, I’m not gonna build a data center or invest in a data center to host my SaaS web app, or whatever. It just doesn’t make sense to do that. But if I’m in the business of actually building highly customized servers and hardware for our cloud-hosting businesses, then yes, maybe I do need my own data center to do these things, because that is in the line of business. That is the business you’re in.

As long as whatever it is you’re evaluating does not lead directly to revenue generation, or in some cases cost reduction for the service that you’re providing, then for me it’s a no-brainer. It’s not even worth considering. Because I can go find something out there that’s gonna do - perhaps not 100% of everything I need, but again, I’m using the 80/20 rule when I’m talking about this here, “What is the thing that gets me close enough?” And if I need to stitch together some extra things and whatever… Maybe there’s some things that it doesn’t do quite well, and maybe I wanna dedicate basically an extra 10% of engineering time and effort to get there… No solution is gonna be just right, and even when we build our own, we find out later on that “Oh, the requirements changed”, as they often do. “The criteria that we were using, that we decided to go build this thing on - they have changed a little bit.” Or the business doesn’t care about this as much, or we don’t need all these nines for reliability. Maybe we can get away with four nines for this thing, because our customers are “business hours only”.

There’s a lot of things you’re gonna factor into that decision, but I think for me, a really cut and dry rule, whatever it is that I’m thinking of investing in, “Does this contribute directly to the revenue, or to the cost reduction of the service I’m trying to render?”

Yeah, I definitely agree with that. But I think, maybe to play a little bit of devil advocate, or to kind of take a slightly different view on this as well - I wonder if a lot of the time when we’re considering the buy option we don’t take into account enough how much it takes to actually vet a solution that we want to buy. Not just like, “Oh, there’s a solution out there that does this thing”, but like “Is this company gonna be around for as long as we’re gonna be using this thing that we buy? Are they gonna actually be able to sustain the type of demand that we have?”

I know there’s a number of SaaS products that I’ve seen in the past and all of a sudden you start looking at the company and it’s just like one dude in his basement that’s maintaining it, and it’s like “Oh, we’ve found something that perfectly does what we want”, but that seems like a major risk for us to actually take that on and use that… Because if that dude decides he doesn’t wanna do this anymore, or something happens to them, now we have to figure out either another solution that we wanna buy, or figure out something that we’re gonna build, and it might happen in a very inopportune time.

So I wonder, is that something that you think a lot of companies are already accounting for, or is this kind of like a little bit of a blind space when it comes to the maintenance of the things that we wind up buying?

I certainly think that if you’re going to one guy in a basement, that should cross your mind as something to check in on. I think that’s a great question; I would like to say everyone thinks that through, but I don’t think they always do.

I think what you’re both touching on - I think Kris you’ve touched on this, in terms of relying on dependencies that might not be there in the long run, and Johnny, you’ve touched on this in terms of people, even if it’s a big company, changing things; maybe they fix a bug that you’re unaware of, that you were relying on, or they changed that ethos, or the way that they want you to use that product… And I think that’s another interesting area to explore, is around the fact that you’re giving up control when you’re relying on these external, bought solutions, in a way that perhaps you should really bring into the equation in a big way, because you are then relying on external people to make sure that your infrastructure is working the way you want to. You’re giving away a facet of your overall technology that you could control and you could have complete ownership of.

[20:12] So I think certainly in my mind - which goes to kind of evaluating what you eventually want it to do - certainly when I’m having conversations around “Should we be building this? Should we be buying it?”, is this something that it’s really important that we have full control over, full ownership of? Are we gonna wanna have the flexibility to tweak it, to change it, to innovate on it at a pace, 1) that is gonna be sustainable? We don’t wanna be pinging a ticket every two seconds to our external dependencies and following up on PRs, following up on “Hey, have you done this feature update which we’ve put in three years ago, that we really need?” Because then you eventually - personal experience - get to a point where you’re like “This was a great solution, but they’re just so slow. They just don’t get back to us, and we actually should just build this ourselves, even though it’s more time at the front.”

When we have full ownership, we’re gonna be able to iterate faster, we’re gonna be able to have more personalization, we’re gonna be able to just completely be in control. I see you smiling, Johnny…

[laughs] Well, first of all, to preface my thought here… Obviously, the different experiences we each have when it comes to this whole build vs. buy thing - half the time it’s completely by luck. So you might be in an organization where they favor a particular approach over another. Or you might be in the same organization where there’s been a changing of the guard and the new leadership feels a different way than the previous leadership.

Over the years, when I see people with strong opinions internally, sometimes I kind of have a chuckle, because I’m like “Ah… You haven’t been through this rodeo yet, have you?” [laughs] Because I see this sea change will happen so often. Again, it’s all about trade-offs. Usually, what I do – and this actually leads right into a technique I like to use when I’m trying to make a decision of build vs. buy. So if I’m managing an engineering team and we’re trying to decide whether “Okay, should we build our own search tool, or should we find an open source tool, or go with some de facto tooling out there, maybe Elasticsearch, maybe some popular open source project that we know is gonna be maintained etc?” So the vetting process to decide one way or the other - we document that. We create a decision record, meaning that “Okay, at this particular point in time, this is the problem we’re solving. These are the constraints we have. These are the trade-offs we’re willing to make. These are the things we’re willing to budge on, these are the things we’re not willing to budge on.”

So once you’ve laid that groundwork and everybody agrees – you can’t 100% get there, but you start to sort of remove the emotion, the feeling out of the decision… Because I might go for it after all, I might have a penchant for using Bleve as an open source search tool. Enumerate a list of the benefits - I don’t need to have massive servers, it’s fast, it’s written in Go, I can understand the code, what’s going on back there… And if I have to, I can reach out to Marty and say “Hey, Marty, we’re having this particular problem. Have you encountered this before? Have you had customers who’ve had this particular issue?”

I have this sort of intimate relationship with that ecosystem, with the maintainers of the project, with the project itself, that I don’t have with, say, Elasticsearch. I may already be biased towards picking a particular solution because of my familiarity with it. But if we all agree, whoever is part of the decision-making, if we all agree “These are the things that we care about”, that foundational thing, “Let’s agree that these are the things that we care about. Now, these are the options.” I can throw Bleve in there, as an example. Somebody else can throw Elasticsearch. Somebody can throw a hosted solution, like Algolia. Somebody can throw – whatever it is.

[23:49] Then we start evaluating these things based on that criteria that we all agreed were the things that we cared about… So that when that decision gets made, and we decide to invest into building our own, or invest into bringing in a third-party dependency and being on the hook for basically keeping it going, should the project be abandoned or should it not be moving as fast enough, as Angelica was saying - we’re forking it and bringing it in-house to do these things… We’re taking on that – we’re going in with both eyes open.

So that’s what I’m talking about, when I’m talking about why you may have a default stance, and that default is going to change, because people change. The same people who care about this thing today are going to be not caring as hard about it tomorrow. New people coming into the picture are gonna have different opinions. They too may have their bias, like “Hey, I wanna use Postgres. Postgres is the cuckoo’s nest, and I wanna use it for everything, including full-text searching.” And there are some pros and cons to that approach as well. Throw that in there. Let’s look at that, too. But at the end, when the decision is made, you have a decision record that says “Okay, these are all the things we’ve evaluated, and based on this evaluation, which we try to make as objective as possible, this is the solution we ended up going with.” So that when the new guard comes in, the new set of developers, or whatever it is, hopefully - that doesn’t always happen, but hopefully, rather than saying “Ugh… Who built this thing?! This was stupid decisions.” You’ve seen it, you’ve heard it - new people come in, they’re like “Oh yeah, I’m gonna fix all the things.” We were probably some of these developers over the years, right? They come in, full of ego, full of bravado, and they’re like “Yeah, I’m gonna fix all the things.” They’re like, “Sigh… Who made this decision? Why are you using this library? Why are you using this language?” But that’s for a different podcast episode.

But yeah, hopefully, with that knowledge in hand, if they look at it, they can say “Okay, these were the constraints then.” Perhaps today we’re working under a different set of constraints. Now, hopefully they do a similar exercise to find what to transition to, whether those be in-housed, off the shelf, whatever it is.” But again, it’s removing that sort of “Oh, I like this thing, because I like that thing” kind of thing. Try to make it as objective as possible. It’s not 100%, but try and get there is really what matters most.

Break: [26:00]

I feel like there’s this impulse - at least with some of the companies I’ve been at - where we’ll be on one side of this spectrum, the pendulum will have swung to one direction, where it’s like “Okay, we’re gonna buy everything, we’re gonna build everything.” And then a new regime does come in, and all of a sudden everything that was there before is crap, we need to do everything different, and we kind of swing the pendulum all the way to the other side, and then it’s kind of just as bad.

I’ve always had the position that you kind of wanna be in the center. You wanna be oscillating in the center. You don’t wanna get stale, you don’t wanna get stagnant and stuck in the same place, but swinging to the edges of like “We must buy everything” versus “We must build everything” never seems to be a good position to be in. I feel like you were gonna say something, Angelica…

[27:50] I was. Actually, two things came to mind as you were chatting… One was - to your point, Johnny, when you’re making this kind of decision tree that’s this record, how do you consider the rest of the company and/or the other dependencies you have? i.e. if you get to a point where you have (I don’t know) six vendors that you’re using to fulfill one use case, because they all give you a little different feature - that feels to me slightly problematic. However, you assess those individual capabilities you required, and you came to the conclusion that this buy solution was the best, and less time consuming, and gave you what you needed.

However, I wonder how you bring in that broader ecosystem, whether it be the company ethos, or just on a team, one team, if they have a broad enough product scope - is there a point where you say “Oh, actually we bought too much. We actually should stop buying, even if there is something that is easy to bring in, because then we just have to keep track of all these different bought solutions.” I wonder if there’s a tipping point in your mind.

The bigger the organization, the more teams, the more layers, the larger the body of people who may have influence over these decisions, the higher the likelihood that you’ll have redundant products, that kind of do similar things, but each one does them differently enough that this particular team was able to justify to their boss, to their chain, that “Oh yeah, making this investment will allow us, our team, team X, to move twice as fast, or to be able to operate X set of services better, because this product X from vendor Y offers us all the -ilities that we’re looking for. It offers us the ability to observe, and to do this and that”, in the case of, say, monitoring, or something… Because there’s tons of monitoring solutions out there; that’s an easy one to sort of pick at.

So you’ll find that eventually. Somebody’s gonna come in. Usually, 2-3 levels above the team manager. Maybe it’s gonna be a director, or a VP, or whatever. Somebody’s gonna come in, probably gonna they’re gonna be brand new to your organization, with no pony in the race, so to speak… They’re gonna come in and they’re gonna be like “Why do we have four different monitoring solutions?” And then somebody, maybe somebody who’s still around from the time all these decisions were being made, they’ll be like “Well, there was a team that did this particular feature set, and they really felt strongly about it, they made a case for it, and we approved it.” And then somebody else is gonna be like “Well, I don’t know; that’s before my time. I can go back to the team and see if anybody remembers why we picked this thing.” And then somebody will be like “I have zero clue. I don’t know. I’ve been asking the same question.” And then all of a sudden, that starts a whole new cycle of people saying “Well, now we must consolidate all these things”, and everybody now starts running around, trying to figure out “Why do we do these things again? [unintelligible 00:30:43.15] Now we need to make a decision on which of these products to pick. We don’t wanna be paying four different vendors for the same thing, more or less.”

So the business starts to drive the technology decisions, as it should have in the first place… But now that starts to bring in all the cowboys running off and being like “Oh, it’s time to lasso these back in, try to bring these things back in.” We need to bring order to the chaos, or so we think.

And then what’s gonna happen at some point - that same VP who was asking the questions, people are gonna be making cases, “Oh yeah, this product is what we need.” And then 4, 5, 6 years down the line you find yourself creeping back towards the same thing, because needs have changed, technology has changed, things, people… You get the idea. Which is why it’s so important to know exactly what problem are you trying to solve. And to have the courage to go back to that decision record or whatever artifact you wanna use to track the things that were influencing the decision you were making, to go back to that at some point and say “Hey, is this product still meeting our needs? Is this approach still meeting our needs? The fact that we’ve been tracking developer hours going into maintaining a home-grown solution - we could have bought this product twice over. Heck, we could have acquired the company had we just looked at the cost of this thing.”

[32:06] Because if you’re not tracking that, you’re kind of – sometimes people don’t track costs and hours and things because you kind of have to reconcile that with reality and be like “Oh, this is… This is kind of embarrassing.” We don’t wanna do that, right?

So again, if we’re being honest with ourselves, eventually we do have to sort of – that reckoning has to come. You’re gonna be like “Okay, this is what we spend, this is all the things that we’re doing. What are we making this decision on?” It’s okay to make the wrong decision, as long as you have the things you were looking at that says “This is the criteria that I’m using to make that decision.” It’s okay to get it wrong. But at least you have some documentation that says “These were the things at play when I made that decision.” And it’s easy to say to the VP “Hey, this is what went into making that decision.” Now, if they wanna change the constraints, they wanna change the parameters, it’s all good. It’s all good. The company has money to throw at it? Fine. It’s all good. But everybody has a common understanding of why we picked all these vendors.

So knowing that there’s all this flexibility, and things are gonna change inevitably, you might pick to do a bought solution, and then 3-4 years later - or honestly maybe a year later - have like “No, there’s this other bought solution that the company wants you to do”, or even decide “We should build this”, I wonder… It’d be actually interesting, Kris, to hear your opinion on this - does it change the way you as developers approach building that solution, i.e. utilizing that dependency? Does it mean that you would approach writing your code differently to be dependency-agnostic, or flexible enough that if it comes to the case where we have to decouple it with some third-party vendor, some dependency in the future, do you go in and make it very closely-coupled, clean, really closely integrated? Or do you in fact, knowing that “Oh, okay… Two years down the line this might change”, write your code in a different way?

There’s no general answer to that question, I think. It’s highly context-dependent. I think a great example of this would be Kubernetes in the current day. Kubernetes is amazingly popular, everybody is using it, it’s sort of becoming a standard of its own… And you kind of have a choice when you start using it. You can either keep your applications as they were before, running in whatever they were before, and just make them run on Kubernetes. Or you can start leaning into Kubernetes, build operators, tap into the API, get all this information, but then you’re tying yourself directly to Kubernetes. And it’s kind of hard to figure out which one of those things you should do, because Kubernetes has only been around since 2014; that’s about seven years… So seven years from now is Kubernetes still gonna be a thing? And if you lean into it very heavily, then you’re making the assertion that “Yes, I think that it’s still going to be a thing in seven years. I think we’re gonna benefit from having done this integration.”

If you don’t, maybe you’re saying “Well, maybe there’s a possibility that we could hook into something else.” But I think usually what winds up happening is that people don’t make a decision either way and you wind up with something in the middle, and in this case, being in the middle is bad, because that means you’re still stuck with Kubernetes at the end of the day, but you can’t actually move to something else, because you’re too tied into some of the things that Kubernetes is doing.

This also happens with cloud platforms. People are like “Oh, we wanna be multi-cloud” or “We wanna be able to switch away from AWS, and be cloud-agnostic”, and then you actually sit down and you’re like “Well, here are all the things that you’re doing that make it so that it’s actually gonna be a giant pain to lift and shift to somewhere else”, as they say.

So I think it’s one of those things where you have to gather as much information as you can, and then make a decision in the moment that makes the most amount of sense. And then realize you’ve made that decision, kind of more or less live with it, and then realize that at some point in the future when you have more information, that you should come back and potentially revisit that situation.

So if you’re building something today, I would say if you’re already using Kubernetes, lean all the way in. Use operators, use all these great features, because that will make your life a bit easier, it’ll make your current way of operating a bit easier. But just remember that some day down the road, maybe 4-5 years from now, those decisions you made won’t be making as much sense.

[36:05] You’ve gotta continually evaluate and reevaluate that decision that you made, which I think is kind of the theme that we’ve had this whole time, around just like “You have to bring in the requirements, you have to bring in the information, you have to gather all of that stuff.” And on that, I wanna raise the prospect of build vs. buy outside of just the code realm, and kind of bring it to our process and project management ethos and the things that we do… Because as we have been saying, we have to gather all of this information, we have to evaluate, we have to track time, we have to do all of this.

In my experience, I’ve very rarely seen people making explicit build vs. buy decision around this; it usually starts out as a buy. Someone’s like “Okay, we’re gonna do Agile. Or we’re gonna do Scrum. We’re gonna do X, Y or Z, and we’re gonna use these methodologies”, and then it morphs into some sort of built thing that doesn’t really look as much like the thing you started with… And new people come in and they’re like “Well, this isn’t how I did X, Y, Z, Agile, whatever, at my last place.” So it seems like quite a mess. So I’d like to hear your opinions on “Do you perceive this as a problem as well?” And where do you fall maybe on how we can start resolving this, or at least making it easier for people to make an explicit build vs. buy decision around this stuff?

I’ll defer to Angelica on that one, because she’s master product manager and all that stuff…

Oh, gosh… That’s a title I would like to one day be worthy of… I mean, in terms of – honestly, I would maybe hope to understand exactly where your viewpoint is coming from, Kris, in terms of like, you take a methodology of project management, a way of working, and then it morphs into something that may not be a copy-paste of what everyone else, other companies are doing… I mean, I personally see that less of a mess, but more of a molding it to a shape that fits your team.

I will say, my team - yes, we practice Agile practice, we do two-week sprints… On the surface - yes, we follow that kind of boilerplate, “This is how Agile teams work”, but within that we have nuances, we have different ways of doing things, different ways of structuring our meetings… We allow flexibility to mold that kind of template and make it into something that works better for us.

Yes, I would say I have seen and heard of places where by throwing out the rulebook and kind of adding too much flexibility around the process has turned into a bit of mess, but I feel like when it comes to how you project-manage and how you run a team, I feel strongly that you should have the flexibility to break the rules sometimes… As long as it’s discussed with the team and it’s not like one person has a different understanding than another.

Management and management tooling is one of those things where it’s so soft and nebulous. It’s not like an engineering discipline where you measure and you cut and you come out with the precise thing. It’s way too malleable for that. That goes for practices such as Agile, capital A Agile… I kind of find that to be a joke these days; that’s lost its way in my opinion. But to have an agile team (lower-case a), to be able to react to change… Because in a business change happens all the time, so you want your engineering team to be able to react to certain things. So that’s what you want… And that agility must come with flexibility. It’s kind of a prerequisite that you have to have enough flexibility in how you do things, be it in tooling, in process… So if you wanna have stand-ups or not, whatever everybody is doing these days, if you wanna have it remotely, people just write in on Slack, check-in on Slack - if you want to do that, that’s fine, too. Whatever works for the team you’re managing right now. Whatever works for the team and the communication lines that you wanna keep open - whatever works for that team, I think that’s what you want to do.

[39:55] I have a tremendous amount of respect for Scrum masters and project managers, product managers, because they have to sort of deal with that amorphous sort of thing. This is something that requires a lot of EQ basically to know what works well in a given environment, because from one job to the next, from one department to the next, from one team to the next, you’re gonna find different things that work… And even you as an individual contributor that joins the team, whatever worked for you personally in a previous team may not work for you in this one.

So you might find, “Okay, most people do things this way. It’s not my favorite, we didn’t like it either. We didn’t like this particular way in the previous team, so we did it differently, but now everybody seems to like this one, so I guess I’m gonna have to put myself in the backseat for a little bit and just see how this team operates.” So these kinds of things, they’re not cut and dry decisions. Sometimes you have to experience a little bit of discomfort for the betterment of the whole team, for the velocity of the whole team as opposed to just your own preferences, and things like that.

So yeah, that’s a very mushy subject, and I wish there was more cut and dry solutions to that, but yeah, this is where experience and empathy and having that EQ is really gonna help you do well in those environments.

I would also say that that flexibility totally works if there is a shared understanding of that baseline “What is Agile process? What is the purpose of a retro? What is the purpose of a stand-up?” I feel like making sure that every member of your team, if you do choose to have that Agile as opposed to waterfall process or whatever it might be - making sure every new member, every current member really understands the baseline principles of that working style, so that you can have informed conversations about where you’re gonna implement this flexibility, where you’re gonna change things up… Because I think – and I’ve had this. I joined a new team, and I’m not familiar with their working style, even just like the boilerplate of what that even means as a process, I’m not gonna be able to really work in whatever this new flexible model they’ve developed… But nor am I gonna be able to have informed conversations if something isn’t working for me. Because if I don’t really understand the purpose of X, Y, Z parts of the process, then I’m not gonna be able to speak to “How do we change that? How do we improve this?” So I think that in these scenarios there is a need for that boilerplate.

I’m just trying to think through, if there was no working style, no methodology to follow at all, and you’re kind of just going in “Oh, let’s just make up a way of working as a team” - I don’t know, it just fills me with anxiety. I like that there’s a structure that I can then play with.

I guess, in a way, I don’t think we would ever just not have anything, but I wonder if it’s sort of like the requirements of sitting down and figuring out, instead of just saying “Okay, we’re gonna do Agile.” Starting from that perspective, how do we wanna work? What do we wanna do? That seems the thing that perhaps we should be buying, instead of more prescriptive, capital A Agile methodology and all of the things, the sprints, the retros, all of these other words about moving very fast that we tend to bucket together. It feels like we do that kind of work we’ve suggested so far in the episode [unintelligible 00:43:04.02] that requirements gathering, that evaluation… It feels like the current way we do things is we do it after the fact. We start off with some structure, then we’re like “Okay, we know that we’ve modified this almost every place we’ve been. We’re gonna start with this and then bumble around for a bit, and then our team will kind of jell, and we’ll figure it out, and we’ll go through that process.

When you have a new member of your team, you absolutely should do team norms and go through the process where you evaluate and make sure they understand the current process, but also they’re empowered to speak up and say “Oh, I did it differently at this other company. Would you be open to trying it?”, which is a key thing here. You have to be open as a team to trying these new things. Maybe we’ll do waterfall for a month. We’ll try it out.

I would say though, just like a slight logistical one, is - on the product side, I love the fact that the whole company operates in one working style, because if I am reliant on X team, who does two-week sprints that are aligned with our two-week sprints in terms of dates, then I know who to be like “Hey, you need to get this ticket in this sprint”, and then I know that we can then take this ticket in this next sprint.

[44:16] So I like that I know that everyone I’m speaking to across the company will understand that baseline structure and bluntly deliverable at the end of each sprint structure, so that I can track through and follow up with people, and make sure that when you’re doing cross-team work, which most of the time you are, you don’t have to faff about as much with like “Oh, well we’re doing a waterfall for the next 12 weeks.” It focuses the project planning more, in my brain.

So this kind of guides us in a really nice direction that I wanted to go as well, around – I guess in-betweens. So not just like a code, but not just a more soft/social skills thing, but like, are we buying at the right level? And I have the example of something like – you know, we have these standards; we have TCP, HTTP… These very old things that have lasted 30, 50+ years, with relatively small amounts of change when you consider it. HTTP - we’re just getting to HTTP 3, and it’s compatible with all the other versions of HTTP… And it’s ubiquitous, it’s used everywhere, and there’s implementations of it that you can buy, but if you decide to swap out that implementation with something else, then there’s not a problem. And I think that would also map onto what you were just talking about, Angelica, where it’s like “Well, does it really matter how the team’s operating, or does it matter – hey, I can say that this ticket is gonna get done by this point in time, and I can just come plug that in.”

So I guess my question around this is “Should we be looking a little bit more at standards as a way of buying things, instead of just looking at the code or looking at a particular esoteric implementation of something?”

You’ll have to give us both some time to think that through… That’s really interesting. What do you think, Kris?

I think from a maintenance perspective, a maintenance side of things, this is an area where it absolutely makes sense to really lean heavily into this idea of standardization and only open standards. I think that there is a rather large amount of upfront cost though, because standards do take a lot of time to develop, you have to be very careful, you have to be able to think and project into the future in a way that people just really aren’t used to.

I think gRPC is a good example of this. gRPC is really nice - you write a protobuf file, you write some APIs… Real easy to get started. But almost every place I’ve been, gRPC becomes this kind of mess where every time you wanna do something, every time you wanna change something, you’re adding just more new methods, and there’s not a lot of documentation infrastructure around it, so you’ve gotta build that up on your own, or you just have this human documentation system where you just go ask people…

[47:44] And when I look at HTTP, I see a system that’s – you don’t care about the version of Facebook or the version of Google that you land on when you navigate to that website. You don’t care about any of those things that we have to care about when it comes to APIs. And the whole system just works. And there’s a beauty to that, because you can kind of swap things in where you like, you don’t have to use a particular thing. You can experiment, you can try new things, you can develop new things. And that is, in my experience, extremely difficult to do with bot solutions, especially when it comes to SaaS solutions. I think a lot of SaaS products have this incentive to make it hard for you to move away from them, and make it hard to interoperate with other solutions. And we do have some standards around things that people offer as SaaS products, but those never feel as robust as something like HTTP, or something as robust as TCP.

And I feel like there’s another side to this, as well. gRPC is built on top of HTTP/2, GraphQL is also built on top of HTTP… But it’s always struck me that there’s these things that we build on top of them that are this buy model. The thing you get with gRPC is a bunch of code. You don’t have to go implement something yourself. But it kind of breaks the model as well. gRPC is not just HTTP. You can’t do a lot of the things you can do in just regular HTTP with gRPC. So it’s not the same amount of flexibility.

So we kind of think from that perspective, it’s like “I would like us to be able to buy more standards. I would like us to be able to just use HTTP by default and layer on the stuff that we want after the fact, knowing that we’ll be able to change and modify it over time.

I’ll give an example from a very real and relevant lived experience for me right now, on some stuff I’m working on… OpenTelemetry - this is sort of the big standard that brings together OpenCensus, OpenTracing… If you looked at anything monitoring, tracing, observability over the last couple years, you probably no doubt have run into OpenTelemetry. And even if you haven’t, you’ve probably heard somebody who was talking about OpenTelemetry, or something.

So you have the OpenCensus crowd, who are trying to set up a standard, and you have the OpenTracing crowd, who also had similar ideas… And thankfully, we’ve sort of netted on OpenTelemetry being the standard that we all want to embrace, and that’s sort of the glorious future that we’re building right now. But that project is still sort of really in its infancy. But a lot of companies are making big bets. The big giants like AWS and big vendors in the monitoring space, the New Relics of the world… If you just search observability and OpenTelemetry, you’ll see things from Datadog, from New Relic, from all these other players. Everybody sees that writing on the wall, so to speak, that says “Okay, up to this point we’ve always had our own formats for transmissioning data to our systems…” Encoding, packing, whatever it is; everybody’s got their own thing, which is efficient for their environment and whatever it is. They provide their own SDKs, their own tooling, and everything else…

And all of a sudden, here comes OpenTelemetry, trying to establish the one way to do things, and now everybody’s trying to create plugins and basically say “Hey, yes, we’ll accept OpenTelemetry-formatted traces and metrics and things.” But there’s so much pain to get here. Something like HTTP and TCP - these are fundamental, core things that really even if you’re a laggard, to use a term from the adoption curve… Even if you’re a skeptic, or even if you don’t want to use it, the moment you get on the web - well, you’re kind of using it. So it’s kind of fundamental to everything else. At some point I think OpenTelemetry is gonna be one of those things where – right now it’s the tech enthusiasts, it’s the early adopters, it’s the people who see that great future that are sort of saying “Hey, this is a standard we see. It’s coming. Let’s jump in” and you’re paying that price.

So to your point, Chris, I can see that standard, but right now because of the kinds of projects I’m working on for my employer, I’m feeling every step of that pain, because a lot of the things we’re doing involves that new hotness, that new standard that everybody’s trying to work towards. We’re an innovative company, we’re ahead of the curve in that way, so we’re making that investment and feeling that pain. The changes to the SDK, to the APIs, and changes to the concept, the constructs… [unintelligible 00:51:58.14] with the OpenTelemetry Go SDK and library, so whenever there’s a change there, I’m feeling that pain, and I have to go to all the projects that are already using our stuff internally and having to update those things… Because we want to help those teams embrace that standard for the future of our platform, of the company, and things like that. So we’re paying that price.

[52:19] So again, it is a price that we deliberately chose to pay, and that’s the major distinction here. We saw what was coming and said “Okay, are we willing to make that investment? And literally, we had to go make a case for adopting the standard, for adopting vendors that support that standard… Basically, a domino. Basically, for everything to fall into place, for somebody to sign up and say “Yup, we are gonna stop doing X, we are now gonna start doing Y.” But we went in with both eyes open. We said “Okay, this is the cost. This is what it’s going to take. It’s going to take a team of N people over a period of Y time, to accomplish result X.”

We went in with all eyes open… So it’s not a surprise. The cost of that adoption is not a surprise to our leadership team, and to the people above them. So that’s the thing. That’s really where for me it’s always gonna come down to “Do we all have a shared expectation of how this is going to roll out, or how we’d like it to unfold? Do we have a shared expectation?” Because that way the accountant is not surprised when I show up with a two million dollar AWS bill, or the leadership team is not surprised when I make a decision technology-wise that’s gonna impact teams across the platform… That way there’s no surprise as to what’s coming, why it’s coming, why we decided to do what we’re doing… Nobody’s surprised. As long as everybody’s on the same page as we’re trying to accomplish, I can go in the weeds and go figure out the how. But the why we’re doing it, what are the major steps we’re gonna have to take to get there - as soon as everybody’s on that same page, I think piggybacking off of the standards is absolutely the right thing to do. You’ve just gotta know you’ll be paying that cost, and that’s fine.

That was great. I love that. And I think that’s a perfect way to transition into the final segment here, of Unpopular Opinions.

Alright… Angelica, do you have an unpopular opinion?

I have many.

Ooh… [laughter]

She came prepared.

No, I don’t – I’ll only share one. I think my unpopular opinion may well just not be that unpopular, but given that recently - shameless plug - I listened to the opinion box being opened, and the list of the most unpopular, and one of the most is around chocolate, I’m gonna make mine also food-themed… My unpopular opinion is that we should switch the foods that we eat for meals and have dinner food for breakfast, and have breakfast food for dinner. I wanna wake up and have a lovely bowl of pasta, and I wanna go to sleep having eaten a lovely bowl of granola.

Okay…

I mean, breakfast for dinner is a thing. I don’t know if dinner for breakfast is a thing…

I always wake up absolutely ravenous, and I’m just like “I want a big meal.” Whereas at night, I’m like “Why am I eating lots of food at dinner, to get to sleep?” I don’t need energy to sleep. I need energy for my day. So just in general, it’s completely made up biologically, which [unintelligible 00:55:38.16] because it’s not probably scientifically accurate. I need more energy for the whole day, so why do I eat my big meal in the morning?

Yeah, you’re not wrong… There’s even a saying, “Breakfast like a king, lunch like a prince, dinner like a pauper”, right?

Yes!

Whoever came up with that - that’s a thing, right? Although I’m having a hard time thinking about – I wake up in the morning and I go have a ribeye, with mashed potatoes and gravy…

I feel like that’s the food that’s gonna put you back to sleep. [laughs]

Start your day with a nice glass of wine with your steak… Now we’re getting into problematic territory. [laughter]

Steak and eggs is a thing though…

It is.

[56:21] I’ll meet you halfway - how about brunch? I could have dinner-like things at brunch.

If I could just snack all day, and just have the biggest brunch ever every day, that would be ideal. The thing is though, brunch is an event. I wanna get dolled up, I wanna go out and have brunch… I want it to be a whole occasion.

Yeah, yeah…

Or we could just do like a continual meal; just always have food available, and just snack continuously…

Brunch is also that excuse to drink in the morning, but not feel bad about it, so…

Have a Mimosa…

Yeah.

Next Go Time. We’ll do Go Time brunch, at 3 PM EST.

Angelica’s like “Yeah, I wanna eat throughout the day.” Yeah, your metabolism hasn’t [unintelligible 00:57:03.19] I don’t know about you, but I’m struggling to keep mine under control.

Alright, Johnny, do you have an unpopular opinion?

I think control is overrated. What we’ve been discussing, around build vs. buy… And I think Angelica mentioned something about basically the control that you might wanna have because perhaps a vendor is not speedy enough with updates, or whatever, or responsive enough, or perhaps you have a cloud hosting provider that you don’t wanna be locked into, and things like that… So you exercise your control by basically building abstractions on top of things, and creating a separation of concerns… Or rather a separation between you and really any third-party dependency; that way, if you wanna move it, you can just have one surface area to re-engineer, and not your whole product. And I think that’s fine. Those are luxuries that you have as a profitable company, with lots of engineering resources and talent and cash. You don’t wanna do multi-cloud – if you’re a startup starting out, unless your business is in multi-cloud, you’re offering something that is supposed to be multi-cloud, you have no business trying to do multi-cloud as a startup company. You should pick something - I’m biased, I’ll say start with Heroku… [laughs] We’ll abstract a lot of things away from you.

If abstraction is what you’re looking for, you can pick Heroku, that’s fine. If you wanna go pick an odd vendor, a GCP, Azure, AWS, whatever - pick one, and focus on making money. Give yourself a break… People talking about control, and you don’t wanna be locked in, all that stuff - that’s fine; personally, I think that’s a luxury. Something that you get to experience eventually, at some point. Yes, there are software packages and open source tooling, things these days that make it easier to start out that way, but man, if startup life is not sexy as people make it out to be. It is a grind. You are struggling to make the next dollar. If you have VC money, they want their money in multiples, in a certain amount of time… I mean, that sexiness we attribute to startup life, and all the things, and the technology selections as a result of it, talking about multi-cloud this - that’s all industry buzz. Focus on making money.

I think personally, one of the ways you narrow that focus is to pick a cloud vendor. Build on that vendor. Integrate tightly if you must. Get to profitability. And then if you wanna create abstractions after you have money in the bank, go crazy. But I think control is overrated in that sense.

I don’t disagree with you there. I think people want the sense of control; they want this sense of freedom, this ability to jump to a new cloud, even though practically they’re probably never going to do that. They want to know that they could if they wanted to, even though they very much know that they are never going to want it.

Right, exactly.

So it’s that sense of control, in a way… I think that point you had about startups is very accurate as well. I think people go into a startup and they’re like “This is going to be fun”, and it’s like, you’re taking out this nice 0% interest loan and you’re buying a giant house, or something… But it’s like, no, those payments are gonna come due each month, and if you get behind on those interest payments, it’s gonna take you a long time.

[01:00:14.20] I feel like that’s what happens - eventually, a company gets acquired, or there’s some exit that happens, and then you look around and you’re like “Wow, there’s a lot of stuff that like, we thought we were doing this in such a great way, but all we were doing was just paying off our old loans with the new loans, and it’s like, all of this has now caught up to us, and we have to actually sit down and figure out how to retool this so that we can actually move forward.

And I just think the hours people work at startups is a lot more than people think… It’s a fun experience, I think, for people that are young and really into what they do, but I think it has a number of downsides as well.

And it’s usually fun for the people who are along for the ride, and to some degree, it can be fun for the founders too, in retrospect, like big picture kind of thing… But man, I’ve had to go without pay in order to pay my people. Think about that for a second.

Yeah…

I’ve had to delay my rent, not pay for certain things, not be able to afford certain things, in order to pay the people that work for me. So you kind of have to – like, that whole sexiness we attribute to startup life, and showing up in Inc. Magazine and all that stuff… Yeah, we ought to have another episode on that, because I’m feeling a soapbox.

I do feel like that could fit really nicely in with this maintenance series of just like maintaining our lives, maintaining ourselves, maintaining our industry. One of the things that we’ll hopefully be able to put an episode together is even just like the way we speak about things. I kind of brought it up a little bit earlier, but the sprints, the velocity, the hackathons… I’ve started saying to some of my friends, it’s like, we develop software as a series of three-day hackathons packaged into two-week sprints. Everything is always like “We’ve gotta solve this right now, we’ve gotta build this in right now, we’ve gotta have this real quick”, and part of the reason why I wanna do this series is because it’s like “But there’s a long cost to a lot of this.” Like, how many times have we come into a company and it’s like “Oh, man. We’ve gotta scrap this entire codebase”, and it’s like, how many human years have been put into building this thing? And now you’re just like “No, we’re just gonna toss it out. We’re gonna try again.” And then three years later, four, five, six years later you’ve gotta do the whole thing again… And we continually do that.

But there are these other projects - I bring up HTTP and TCP for a reason. These are things where we managed to get it right. It’s not perfect; HTTP, by no means, it’s not perfect. Neither is HTML, nor is CSS, nor is TCP… They all have their warts, they all have their problematic areas. But they have remained largely unchanged, and it’s like this phenomena - you look at everything else, and it’s like, ten years ago we certainly weren’t using Kubernetes, because it didn’t exist… So in a ten-year span we’ve completely changed the way that we run all of our software. And we continue to change it in an incompatible way, and it’s like, how much person time, how much knowledge time have we lost just from making all of these changes? How do we reconcile that?

[01:03:20.20] We are an industry that’s always excited about the new and the shiny, and the what can we do that is bigger and better than what we did yesterday? But rarely do people want to pay the long-term cost of that. As you said, with the standards - building standards is a painful, long, slow process. But standards are how we actually work. This industry probably wouldn’t look anything like it does if we didn’t have HTTP, if we didn’t have TCP, if we didn’t have HTML. People harp on HTML and CSS, but it’s like, these are the things – like, HTML killed off basically every single competitor that’s ever come up against it. Flash - gone. Silverlight - gone. Java applets - gone. Everything that someone’s tried to propose against HTML, it defeated soundly. It even had this eight-year period where it just went nowhere, and we’d built up so much stuff, where it’s like “HTML5! Let’s go!”

So I think we need to start having more of these conversations where we can tap into what is the long-term, how do we get to that long-term? Everybody wants to have HTTP, everybody wants to have this nice, ubiquitous thing… It’s so common you don’t even think about it. And it’s so extendable and so robust… I’m still marveled at the HTTP 1.1, to 2, to 3 transition that we’re going at, and how – you know, we talk about versioning, especially a lot in the Go community right now, and it’s like, we have three versions of something that’s 30 years old that are all compatible with each other, and have an interface that’s so cleanly designed that you don’t even have to know which of these three you’re using, because all of the pieces just fit together. And I’m like, “How do we do more of that?” Because that’s where we really push our industry forward. That’s real innovation. I think HTTP is stellar innovation, and I want us to do more of that.

And it’s boring, if I may say so. I think in the next episode in the series we should – although we have touched on Go a little bit throughout this episode, but I’d like to see us talk about Go in this context, about Go being really boring technology that I think is here for the long term… And yes, I think we’re all biased on this show. This is Go Time, after all… But I think there’s merits to Go’s no-flash, non-exciting features and how that plays into really the decisions you make as a startup founder or as a big company knowledge executive to pick the technology… I think there’s a lot of room there to explore, and I’d really like us to center on Go really in that context the next time we bring this up.

Yeah. There was a section on Go that we ran out of time for that was talking about libraries and maintenance, but I think that could also just be open source, and Go open source maintenance could be its own entire episode; there’s a lot to talk about there. So yeah, this is why I’m like, “We need a series on this…”

Indeed.

…because I think it’s a really important topic. Thank you for joining me, Johnny and Angelica. I feel like this was a really great conversation, and I’m definitely looking forward to the next episode in this series.

Awesome. It was a pleasure.