Big shoes to fill

Welcome, welcome, welcome to another wonderful episode of Go Time. On this week’s episode we’re talking about recent news, both in the Go community and in the programmer community at large. This week I’m joined by two wonderful co-hosts. We’ve got Johnny and Angelica. How are you doing today, Johnny?

I’m doing alright.

Not running Linux on the desktop still, I hope, right?

No, I’ve kind of given up on that.

Yeah, that makes sense. And Angelica, how are you doing today?

Splendiferous. Very excited and ready to go.

Ready to go. Love the pun, love the pun.

Any opportunity.

Any and every opportunity. Alright, so let’s just dive right into the news. Our first thing we’re talking about is the pretty substantial news that dropped - was it a week and a half ago or so? …that there’s going to be some changes in the Go leadership team.

We have Russ Cox, I guess more so stepping back than stepping away. He’s going to still be working on Go stuff, moving to working primarily on a project called Oscar. And we have kind of other people that are in the leadership of the Go team, who people might not know as well, who are now stepping into new roles.

So we have Austin Clements, who’s shifting into the Go technical lead role that Russ currently occupies, and then we have Cherry Mui - I believe that’s how you pronounce her last name - who is stepping into what was Austin’s role, or I guess currently is Austin’s role as the leader of the compiler and runtime team. So yeah, it was a pretty long email that went out… But yeah, what are your guys’ thoughts?

I mean, I’m excited… I think I’m someone who – I’m always very excited by change. I think for me, two core thoughts. One is excited slash anticipating what this means… Is there going to be a change of technical strategy? Is this going to be a continuation of the current strategy? For me, it’s more a question mark and an excitement.

I think secondly - and for those who haven’t read the email, I would really encourage you to… But for me, I had a deep amount of respect, in terms of being self-aware, from Russ’es side, of like “Okay, I’ve led this team for a very long time”, and kind of acknowledging like “Okay, I think it’s time to kind of step back, bring in some new–” I mean, as you said, Kris, not step away, but step back… So for me, that was a feeling that I felt reading the email; nice to give away the reins and kind of let others innovate.

Yeah, change is good. It’s been 12 years, over 12 years since Russ has been leading the technical aspects of Go. And to a large extent, all of us in the community have looked to him for leadership, exceptional leadership at that. I don’t think everybody has always agreed with every idea he’s had, every approach, but what I loved about, or continue to love about Russ’es approach is that he’s very thorough. You may not love everything that he proposes or the direction he wants everything to move, but he’s going to have very well-reasoned, very well thought out explanations for the choices we make, or that the language is going to have, not have… He is very thorough in that approach. All you have to do is really go on his blog and check out any of his posts on any topic, be it how we chose to handle dependency management back when we had to handle that, to his thoughts on generics, our approach to that, what we’re doing/not doing, in collaboration with Ian… So I think that kind of leadership, he’s set the pace, or rather he’s demonstrated time and again what it takes to be a programming language technical leader.

[00:08:29.17] And I don’t have a personal experience, first-hand experience with either Austin or Cherry - I hope I’m pronouncing their name correctly - but I’m hoping they will meet and exceed what Russ has been able to do over the last 12+ years.

So definitely, as Angelica puts it, change can be an exciting and good thing. I’m looking forward to what Austin and Cherry bring to the table, but I think Russ is kind of taking a step back, really leaving go stronger than when he found it, honestly. And I think we all benefit from that leadership, and we’ll continue to benefit from that leadership for years to come.

And yeah, with the new folks stepping in, I hope they will be as forthcoming in their decision-making process and thought processes as Russ has demonstrated… But obviously, leaving room for these individuals to shine through, and show us what they’re about. So we’re not going to be comparing them to Russ. What we are going to be looking for is, okay, what is their leadership style? How do they reason about things? How do they make decisions? Can we get behind the way they do that? Again, not expecting that everything we want we’ll get, or not every decision that they make, we will agree with. That’s the burden of leadership, not everybody agrees with you… But you provide the direction. So I wish Austin and Cherry the very best, and I think we all in the community are looking forward to what they bring to the table, and we’ll support them as best as we can, as we always have.

Yeah, yeah. I definitely agree that like Go has, at least from a technical perspective, we’re at this point much better off than we were 10 years ago, I would say. Remember, 10 years ago we were dealing with “Don’t use dependencies, because dependency management is a giant mess.” 10 years ago I don’t even think we had vendoring really working yet. You had to do a lot of really crazy things to make dependencies work. And obviously, we didn’t have generics, and we didn’t have… There’s a lot of things that we didn’t have, that I really appreciate now for like the longevity of the language.

One of the things I hope we get out of this transition is maybe more of a community leader from somewhere. I think that’s the thing that Rob Pike did when he was kind of leading Go, is he was both very much the technical leader and the community leader. And I think when Russ stepped into the role, he was intentionally very focused on the technical aspect, and less so on the community aspect. So I’m hoping that we kind of get more of that community aspect back.

And I also hope there’s a little more space for dissent in the community. I remember when I first joined Go, there were a lot of voices of people that would push back on Rob, push back on Russ, and it was very vibrant. And I feel like perhaps as a factor of the language maturing, and people moving on to other things they find interesting, or because of some of the the past struggles, especially around dependency management we all went through, it doesn’t seem like we have as many of those voices. But I think for the long-term sustainability of a community, you need to have those.

[00:11:52.00] So I feel like now that we’re in a really good technical position in Go, it’d be nice if we could find a way to focus on that community aspect of Go and figuring out how to bring – not bring back the exact same people, but bring back more of that kind of vibrant community of open dissent. I do miss those aspects of it. It feels like the community is a little – we’re getting into a little too much echo chambery territory, and I think that’s never too good, when you don’t have enough diversity of voices.

But yeah, I’m super-excited to see how this transition goes. I’m excited to see what kind of new new ideas that Austin and Cherry bring to the way that Go was led and the technical aspects of Go. Yeah, I think some of Russ’es ideas have been absolutely brilliant. I’m excited for iterators, because that’s been a huge, huge thorn in my side throughout most of my career. So I’m excited we’re getting these things that have been problems for so long. Yeah, super, super-excited about that. But that’s kind of my thoughts on the matter.

Yeah, the one thing I’ll add… I don’t think I share the same experience you’ve had, Kris, with regards to your perception with regards to dissent in the community. My view is one that whenever there’s a proposal, there’s a lot of back and forth, be it on GitHub, or Reddit, or wherever the community discusses; I’ve even seen back and forths on X or Twitter… So I think there is open discussion, but I think, again, going back to the burden of leadership, at some point people can make very passionate and legit arguments for doing one thing or not doing another.

From how we implemented generics, to how we’re doing iterators, to how we did X, Y, and Z, you name it, there’s always going to be somebody who thinks they have a better approach, or that they don’t like the approach that’s been proposed, or the most popular approach and whatnot… And that’s perfectly fine. I personally see that discussion going all the time, be it on Golang Nuts, or be it on LinkedIn, of all places, or on X, I see these discussions. And I think that that dialogue has been maintained under Russ’es leadership. I don’t think anybody’s been shut down, or told not to propose anything, or… I think what I’ve seen time and again is rather, when you take a – perhaps it is an echo of Russ’es approach to decision-making; again, him being very thorough, he researches and proposes and he’s very elaborate in the reasoning behind the choices. For someone like that, you also have to bring that level of rigor, if you’re trying to make a case for one thing or another, in that circle. You can’t just say “Well, I feel like it should be this”, and then leave it at that. That’s not going to get you anywhere with that kind of leadership style. You need to bring data, you need to bring not just anecdotal, but really you need to bring some proof for why you believe something should be done in this way or that way. If you don’t have that, if the only thing you have is anecdotes, be they personal or a very small sample, you’re not going to be able to move the needle very much in those discussions. And I think that is the bar that should exist. For a programming language, I think that is the bar that needs to exist.

There has to be a lot of thorough, rigorous discussions, backed by data, backed by facts, as much of it as can be gathered, to push the language in one direction or another. We all know if the Go team gets it wrong, by some definition of wrong, then people are going to be unhappy, in the same way they were unhappy when generics came, or couldn’t come soon enough, and when it did, you still had people who didn’t like it… I mean, you can’t please everybody, and that’s just the way it is. But whatever the decisions that are made in the language, which is one of the things I really admire about the Go community and the Go team leadership style, is that you are going to have these discussions, and by the time the decision comes down, there’s going to be a very well vetted, very thorough discussion and understanding of why a particular decision was made. And again, you don’t have to love it, but if that is the direction, I can get behind it. I can disagree, but still commit, and still move forward.

[00:16:32.05] I don’t think I particularly agree with that viewpoint, because I think there’s a number of notable things where it’s just been – a) I think appealing to data is not a… I think using data to help back up some of what you want to do is a good idea, but I think heavy reliance on data is not. Because data is not objective. There’s no such thing as objective data. Data has to be interpreted. So when you map objectivity onto data, it allows you to potentially ignore problems that exist, because “Well, your data isn’t showing this.” This has happened, where there’s been people expressing pain with several different features in Go, and the response has been “Well, our data shows that this isn’t a problem.” But once again, there’s millions of people using this programming language. Most of them, the vast majority of them, do not engage with the language discussion process. There’s only a very small number of people engaging in that process.

So any data you collect from the people that are engaging in the process needs to be viewed with a great deal of skepticism. It can help certainly bolster your arguments or your technical views of things, but if we lean too much into it, I think we wind up with, to give an example, semantic import versioning… Which I would not say is working. I would still say it’s frustrating for people, it’s confusing, it’s still a semantic mismatch at the end of the day… Probably one of the not great technical decisions that’s been made, and that has caused people a lot of pain. And I think over time we’ve caused people a lot of pain, which is the – when I was talking about dissent, I was really talking about how is that dissent received. Not just in the technical aspect, but on the human level. A good technical decision, that’s executed not well, or that causes a lot of pain in its execution, I don’t think is a good community solution. And that’s why I said I would like to see more community leadership, I would like to see more compassion, less “This is technically right. I have proven from a technical perspective why this is correct”, and more “Here’s why it’s technically correct, and here’s why it will cause the least pain possible in the community.” And I think we’ve been lacking a little bit of the least pain possible in the community aspect of things. I’d just like to see more of that.

At the end of the day, it’s not like Go is going to disappear if that isn’t added, but it’s definitely one of the things that I miss from when I started using Go. And I think, at the end of the day, programming language communities have their own values, and if that’s not a value we want to have as a community, if we want to strive for technical excellence over everything else, then that’s great. But I also don’t think the foundation of the language was really built for that. It feels like - you know, going back to what Rob Pike said a lot about the language when he was designing it, or when the group of all of them, which includes Russ, were designing it, it was meant to empower engineers, to make them effective and efficient, and help them get things done and just build software, perhaps over technical excellence, in some cases.

[00:19:54.16] I think people don’t like Go’s type system because it’s not as perfect as it could be. But it works, you know? People don’t like that we have nil, but for the most part, it works. It makes people be able to be effective in the software that they want to write. And I feel like if you over-index on data, if you over-index on technical aspects of things, sometimes you can – you just miss the mark on actually being able to help people build the software they want to build. You can start building things that get in the way… Not forever. I think modules is an excellent example of this. Modules today are absolutely fantastic. I really like where we wound up with modules. But in 2018 I don’t think modules were ready to be a forefront, “This is how we’re doing dependency management.” It was a whole lot of pain, a whole lot of frustration, a whole lot of time spent by people in the community trying to get something to work, that was technically a good solution, but community-wise, especially tooling-wise, was just not really there yet. And it’s on, I think, us as a community as a whole to try and find ways to make sure that those things don’t happen as much.

But I would say I very much disagree with the idea that you should just bring a lot of technical rigor and data, and that will be enough. I think that that’s not sufficient for any kind of software project or software community. I think you need to ensure that you’re also taking care of and maintaining the community and making sure that the people who are using the software out in the world and people who are part of the community are aligned with the leadership team at the end of the day. Because if you don’t make sure of those things, you can create fractures.

I’m reminded of a talk that Bryan Cantrill did in 2017 about the Node.js fracture, and what wound up causing that at the end of the day was not so much a bunch of personalities, but it was a misalignment between what the current leaders of that project wanted at Joyent and the other companies that were kind of sponsoring the development and what the broader community wanted. And that eventually wound up with a fork happening… And then there was a whole bunch of stuff that happened that reconciled, and people left the community to go find something, and one of those things was Go, that met more of their expectations at the end of the day.

But I think it’s very important for people to be tapped into the community, understanding what the community wants and where the community wants to go, so that there’s less of that pain, at the end of the day. Because I think we should be trying to both be technically excellent and have a great deal of human compassion, and want to not cause people pain and struggling and exasperation at the end of the day. And I feel like if I’m being 100% honest with myself, I feel like there has been some growing pains over the last decade and a half. I feel like those are always inevitable, but I feel like that’s something that we should learn from, and we should try and get a little bit better at. Not necessarily by decreasing technical rigor - I think it’s very good that we have technical rigor - but I would like to see us be able to do both. I’d like to see us be able to chew gum and walk at the same time. But I’m just one person, this is just my viewpoint on things. I am not the leader of Go. But yeah, I mean, that’s just how I see things.

I mean, as you were talking, I think what came to mind for me is do you think that those growing pains are related to the growth of the Go community? I mean, what I’m thinking of is just logistically, when the language was small, there were less people in the community, so it was easier to hear those voices. Whereas now, we’re a spooling, a huge global community… I would assume it’s – well, in fact, I would know it must be harder to keep your pulse on that community, especially when, as you noted, Kris, there are people who just, they sit, they use Go, they’re not going to engage in Gopher Slack, the Reddit… There are a number of people who want to be active voices in the community, but there are so many others that are not engaging in those conversations, but are really active Go users.

So I wonder if part of the issue, or part of the disconnect is somewhat inevitable, given one team, one leader, say - I know we’re talking about kind of being [unintelligible 00:24:09.27] on the Go team, it must be really hard to keep your pulse on that full community, and making sure everyone’s happy. And I would venture that it’s impossible to make everyone happy. I’m just intrigued.

[00:24:24.19] And then also just going back to the title of “tech lead”, you are the technical lead. That does that not then imply that you are accountable for the technology, its resilience. For me, it feels - and maybe this is just my personal view - more like it’s, I don’t know, a product manager, or someone else’s role; everyone on the team 100% should have that pulse on the broader community, but I just wonder, just thinking on the granular… I have always seen the role of tech lead to be leading the technology. I think they should be, and to be a great tech lead I think you need to have your pulse on the community… But I wonder whether ascribing that to the lead - I think it’s a group effort. I think it’s a community effort. The Go team in general - that should be part of what they do, and I think they do… Looking at the Slack, they engage in conferences, they look at the issue tracker… But also we as a community, as individuals can just encourage people who care deeply, but maybe aren’t looking at decisions, or being part of the conversation. I don’t know, I’m just speaking aloud here.

I think you’ve just described perfectly why we have DevRel, why we have developer relations. We love to give them grief, or people to – we like to get on our high horses and say “Oh, developer relations people - they’re not real programmers, they’re not whatever.” They serve – and that’s just ego speaking. A lot of us make people feel less than, so that we can feel better ourselves. And that’s a thing. That’s called being human. But when we sit down and think deeply about exactly what you’ve just said, there’s the need for that technical leader who’s just going to look at the raw data… But like you say, Kris, it’s not just about the raw data. If you just looked at raw data – I mean, you can look at raw data and you can justify anything good or bad by some lens; you can justify anything with data. But there’s another aspect to this. There’s the “Well, how do people feel when using this particular tool?” Take, for example, the Go tool chain. We’re getting more and more opt-in telemetry that the Go tool chain is going to now start sending. So every time we do a – whenever we use the Go tool chain in the command line, some telemetry is going to be captured and can be sent back to the Go team for analysis, to help really remove the – maybe we’ll say “Well, this thing sucks. Nobody ever uses it.” Well, we can actually go to the data and see “Okay, well, actually, no. It is used pretty heavily, and it’s part of tons of workflows.” So we have the raw data.

Now, one can say “Okay, well, we have the raw data, we know people are using it, but how does it feel to use it? How usable is the tool?” Do they use it in spite, or are they using it because it’s a great experience and things of that nature? And that’s something that a developer relations person, somebody who’s not just looking at the technical aspect of things, somebody who’s not looking at the raw data… Even though that role can also go and get more data to make their point, to validate their point. Basically, they are the ones that are in between, that are basically interfacing with the community, and gathering the sentiment around using a tool or the language, and the features. Are the features working, not working? How do they make people – do they bring people joy in using?
[00:27:51.20] I remember being back in my Ruby days, there was such an emphasis on the joy of writing Ruby. I see a little less of that in Go, and perhaps, who knows, maybe every community has their own quirkiness, if you want to call it that… In Go we’re less about the developer joy and more about the simplicity. Our vernacular doesn’t include things like “the joy of writing Go.” It’s more like the simplicity, and the straightforwardness. We favor certain things in this community.

But you need that person or that set of people that is going to be able to be in underground, with the people using the tool or the technology or the programming language, and then report that back to the Go team. Who’s looking at the raw data, but also didn’t need another facet to look at in order to make decisions. They can’t just be about the data.

So I think you’ve just nailed the whole problem on the head. We need somebody - and hopefully, again, I don’t know Austin and Cherry styles of leadership, but if they can provide that balance between the two, I’m sure there are other people on the Go team, hopefully, that can also step in to help provide that kind of a role. Not sure what we’d call it. Is it developer relations? I don’t know. Maybe it has a different name for it on the Go team, but we need somebody like that who’s going to provide it. I think the closest thing we have now… What’s his name? I think he’s the actual product manager for the Go team.

Cameron?

Cameron, exactly. That’s the closest thing I think we have now to that. I think Cameron is doing a good job. Every interaction I’ve had with him has been quite pleasant. He’s quite knowledgeable. He does a great job of stating the benefits and the case for Go… But I think maybe there’s something else that’s needed. Maybe what we’re all feeling is that there’s a bit of a gap between - maybe it’s Cameron working with somebody else that is more developer-centric… I’m struggling to encapsulate that, but I think it is indeed a relations type of person, who is an engineer, who is in the community, and can report the usage, the usability, the sentiment around using the tools and the language back to the Go team to couple that sentiment analysis with the data, the raw data we’re getting.

Yeah, I feel like, Angelica, you had a very important point about the language has grown. There’s many, many, many, many more engineers. I don’t think when Russ took over and decided that he wanted to focus pretty much exclusively on the technical aspects, I don’t think that was an incorrect decision. I think that implied that we needed to have somebody that would be in that equivalent role to him, to do the community aspect side of things and to have decisions at the end of the day be kind of like a joint thing between them. Like “Is this technically correct and viable, and do we have the data to back it up, and all of that? But also, is it community wise correct and viable, and do we have the data to back that up?” And I think that’s the – I mean, as I said before, I think that’s the role I think we’re missing right now, and I’d really love to see somehow getting that role… Because I think that’s the thing that has caused what I feel is just like some of the strife that we’ve had.

And I feel, I think, Johnny, when you were talking about the data, one of the really important aspects of data is you have to really ask yourself “Are you asking the right questions? Are you asking valid questions?” If you have telemetry, and then you’re getting a lot of input that says “Oh yes, this feature is very used.” Okay, well, then you know that feature is at least used by some large group of people, or some group of people. But you can’t really say the inverse of that. You can’t say “Oh, well, our data is telling us this thing isn’t used”, because your sample size is not random, and it’s much smaller. So it could be used by some other large portion of the community. So you need to use some other type of tool, some other type of heuristic to be like “Okay, no, really, this thing isn’t used.” And I think that that type of data is extremely difficult to get. But I think we’re starting to get closer to getting that type of data.

[00:31:57.08] There’s a team within the Go team called the user experience research team, the UX research team, and they’ve been doing lots of experiments to try and get more high-quality data, which I think has been very helpful. They have the Go survey that they now do twice a year, and that gives them a bunch of data. But I was actually talking to Alice from that team at GopherCon, and she mentioned that they also were able to add plugins to VS Code and to Goland that would just randomly survey people. So an actual just like random pop up, random sampling. And what they’ve found is that way more people are using Go on Windows than has been shown in any of the other data and information that we have. And I think intuitively, as a community, that made sense. That made sense to me immediately. I’m like “Yeah, of course there’s a huge amount of like Go on Windows popularity”, because Go on Windows was developed by the community. There’s a whole group of people that built that target. That was not part of what the original Go team built.

The Go team took it over, but it was originally built by the community. So I would expect that there’s a vibrant number of people, a large number of people using Go on Windows. But the data we had was not indicating that, until we went and we said “Oh, let’s go find another way to collect data”, and then that data started showing, “Oh, okay, there are many, many more people using Go on Windows.” And not that Go was going to drop support for Windows, but if you’d just gone on the survey you would likely come away with a very different conclusion than looking at this wider data.

So I think we have the infrastructure in place to have someone in this role, I just think we need the role itself at the end of the day. Someone that can sit with Austin, sit with Cherry, sit with the rest of the Go team and say, “Hey, I know that this is a really good technical solution, and we should get to it eventually… Let’s figure out what that path looks like, and let’s tap into the community.” We need someone that’s like a community organizer that knows how to build those structures in the community so they have at least a reasonable pulse on what the community wants. Because I think it’s very – this is not an easy task to do by any means, of understanding what a community of… Even a small community of like tens or hundreds of thousands of people is a lot. But millions of people, actually figuring out what that community wants is extremely challenging.

So I guess that’s like my one hope during this transition, is that we find someone to kind of step into that role, or that role gets officially added to the Go team at Google, or another company steps up and volunteers and partners with the Go team at Google for governance at whole, and we have some sort of agreement to make sure that we have this other very crucial aspect of the leadership team filled.

Well, should we talk about happiness, or unhappiness?

Yes. But before we talk about that, next week we’re actually going to be talking with Russ Cox and Austin Clements about this transition. So we will hear what they have to say, hear what their view is on all of this… But yeah, make sure you’re subscribed.

You might also have Cherry, too.

Oh, we might also have Cherry. That’s exciting.

Breaking news.

Breaking news.

That’ll be wonderful.

That’s going to be so exciting.

This is what we do for you, GoTime. News breaks, and then we reach out to the source and we say “Hey, come on Go Time.”

“Look, we already announced it. Don’t make a liar out of me.”

Don’t make a liar out of us, yeah. No, I’m super-excited to hear their viewpoints on all of this, and to see what they see as the future of Go and all of that.

I think my final point on this is almost like a community call-out, because it’s something that was on my mind when I saw this… Russ has been such an instrumental, known person in the Go community for so long… I think my call to the Go community is - Russ’s shoes are big. Support Cherry and Austin as they take on this role. It’s a new role. They’re going to probably make mistakes, they’re going to find their groove. Let’s give them grace. Let’s show what’s so amazing about the Go community and support them in this new, exciting role for them. So I put it out there… Let’s give them grace, let’s give them time, and let’s show what the Go community is about in terms of supporting people.

[00:36:13.02] Hear, hear.

Yeah, absolutely. Now, for a completely different discussion… Developers. Unhappy group of people right now. We have some results from - was it Stack Overflow that did a survey?

Yeah.

And it just shows that developers are having a real tough time. And it’s not because of AI, it’s not coding in general… Which is kind of interesting, because I think a lot of people might have assumed that a) AI is coming, it’s going to take our jobs. Not really what people are worried about. So to not bury the lead, it looks like the reason is people don’t like their jobs. A lot to do with technical debt, a lot to do with burnout, and layoffs, and culture, and kind of the move fast, move fast, move fast culture of things… And also, as a little side note in AI, people think “Great at solving small problems, but not so great at solving complex things.” So yeah, what are your guys thoughts on these results from this rather large survey?

Wait, so you’re telling me developers aren’t happy because of technical debt? Okay… Is it that the technical debt is unmanageable, not being addressed? Like, what about – I mean, any sufficiently healthy business that uses software to make money is going to have some amount of technical debt. That’s just unavoidable. Because you can’t fix everything. Business pressures, timelines, product roadmap, release cycles… There’s so many things outside of writing the software itself that influences the pace at which you deliver said software. The understanding is that you’re always going to have technical debt, no matter how much you slice it and dice it.

So in my experience, you accept that technical debt is, and you figure out how to pay it down as you go. Now, if you tell me that culturally there is no effort being put into paying down that debt as you go, then that’s a separate conversation. Yes, I’ll be like “Okay, maybe this is a team dynamics thing, a leadership thing. They don’t understand the value of paying down technical debt”, things of that nature. But having technical debt on its own - that is a sign of a healthy business. As long as you’re around to have technical debt, you’re doing okay. Am I getting this wrong?

I mean, having just scanned the article, I think it’s less that there even is - and I will emphasize on “scanned” the article… I think, from scanning it, there is an emphasis on technical debt causing unhappiness, but less because there is technical debt, and more that it’s not prioritized, it keeps on building up… They’re having to build features upon inefficient, broken systems… Management is giving unrealistic deadlines and prioritizing new functionality, as opposed to fixing, improving, iterating on the existing underlying infrastructure. And that’s causing frustration around “We know that we have – we might start with a little bit of tech debt, but it gets to a point where it’s so much that it feels like a mountain of tech debt, and any new functionality is going to be crippled if we don’t address it.”

And then you get this vicious cycle of like “There’s so much tech debt. How are we going to address it?” It’s not being prioritized, and no product manager is – well, some might, but I don’t think many managers or product managers would prioritize two months of tech debt; bug bashing, for example. That was my sense, which I can empathize with.

[00:40:11.02] I think that the tech debt is a symptom, because I think it’s not intentionally taken on tech debt. I think it’s because of the assumptions of the software stack that you’re supposed to be using. I think a lot of projects would have very manageable technical debt if they were built as a monolith, running on a few VMs, used a very simple database like Postgres, or something.

I feel like if there wasn’t so much inherent complexity in how we assume you need to build an application, the actual technical debt load would be much lower. But when we’re in an era where if you’re at any company, the assumption is that you’re going to roll out Kubernetes, and you’re going to build everything as a microservice… That on its own is likely to add an unmanageable amount of technical debt to any project, regardless of what at the end of the day you’re actually building. Because there’s a lot of concepts and a lot of things that you have to understand in order to actually use that tooling efficiently. But where is the time for people to actually learn how to use that tooling efficiently?

I’ve been at several companies where we’ve adopted Kubernetes, and I always asked “Where is the time to train every developer we have on how all of these aspects work, so they can properly design and build their software for running on Kubernetes?” And that time is never there. We have to use this thing, but there’s no time to actually teach our engineers how to use it. And I feel like that’s true of so many different layers of things, whether it’s just like Kubernetes, or GraphQL, or gRPC, or any of these types of technologies we have. We just kind of throw them at people. “You’ve got to use this. Have fun learning it.” And then as they’re kind of learning it, as they’re building, they make mistakes, they don’t do things right, they don’t have the right idioms… And then there’s this pile of stuff, and then they just have to keep going. There’s no time to go back and be like “Now that we’ve learned everything, we’ve got to go back and rebuild it.” There’s rarely any time to actually go do that. There’s rarely time to sit down and evaluate “Oh, these are all the things that we did. These are the mistakes that we’ve made. Let’s go back and change it.”

And then on top of that, you have the fast-paced moving of “Okay, we’re going to –” Somebody in the org doesn’t like gRPC. They want to use GraphQL now, for whatever. So they’re like “Okay, we’ve got to spin up, we’ve got to ramp up on using GraphQL.” Or maybe it’s using gRPC. And now we’ve also got to use GraphQL. And we’ve also got to build an HTTP API. And now you’ve got these three different things that are effectively doing the same thing, but for different parts of the stack, and you’re like “Okay, well, now I’ve got to support all of these, but I barely had any time to do before, and now I’ve also got to do that on top of Kubernetes… And things are catching on fire, and things are crashing constantly, and my ops team is yelling at me…” That is enough to make anybody miserable. And I think for a lot of people, that’s going to manifest as technical debt. They’re going to say that’s technical debt. They’re going to say “This application is absolutely rotten. It’s tech debt. We need to throw it out and we need to rebuild it.”

But we as a community, as a culture, have just not given a whole lot of space to do that learning. And I think AI has pushed people to actually do the opposite in a lot of ways. There’s a lot of “Oh, I don’t know how to use this API.

I’m just going to go to ChatGPT. I’m just going to–” Well, probably not ChatGPT. “I’m going to go to Copilot”, or one of these other AI assistants, and “It’s going to write this code for me.” And that can work to get you started, but I think it was The Primeagen who pointed out in one of his videos, the bulk of code in the world is not good. There’s a lot of really subtle bugs in the openly available code. I wouldn’t go out there and say that the majority of code that’s available for this type of scraping is high-quality. I think the majority of the code is going to be average. Sort of by definition, but just in general. It’s going to be mediocre, average code. Which means that AI is not going to be spitting out the best code. It’s going to be spitting out average code. But average code that you don’t understand. So if there’s subtle bugs in there, you’re not going to be able to find those subtle bugs. And that might manifest in production. And now you’ve got to sit down and actually spend time learning about “How does this thing actually work? How does this API that I didn’t take the time to learn in the first place - how do I debug it?”

[00:44:14.24] So I think AI is only exacerbating this issue at the end of the day, because it’s tricking people into thinking they can build things that they don’t know how to build yet. And I think that also winds up manifesting itself as technical debt. So I think it’s less technical debt as we usually envision it, and technical debt as a symptom of a greater problem that our industry is having, of not taking the time to really, at the end of the day, sit down and let people actually learn how to do things. Learn the deep, fundamental things that you need to learn in order to be a software engineer. Because I think we’re a weird industry, in that – our professional degree program, computer science and software engineering degrees, they don’t equip you for actually building software in the real world. They equip you to kind of know how computers work. And you learn a little bit about programming, but you’re – I think we can all kind of agree that most people that come out of CS programs are not ready to be the full-force professional developer. There’s a lot of training you’ve got to go through on the job to learn how all of this works.

There’s nuance there. I disagree, to some degree. So when we think of technical debt, we tend to assume it’s because the code is bad. The examples we’ve used is AI generates code, and the code by definition is going to be average, because it takes in all the – it ingests a bunch of code from the world around it, and all that code is mostly not good. I’d argue that whatever code has been fed into these models, certainly that code has gotten the world to where it is now.

And we are a technology-advanced world. I would rather live in this world, with its, by some definition, bad code, than to live in a world without it.

So I would say that we should give ourselves as humanity a pat on the back for having written code that gets us to this point. I mean, we can fly planes, send rockets to the moon, and do all kinds of crazy things. So I’m going to be a bit more gracious towards us, engineers, software developers as a whole, and say rather than it’s the code being bad, I think it’s the fact that it’s not often – you take any developer at most companies… It’s not that they’re writing bad code. It’s not that they don’t want to write good code. Technical debt is not inherently a software problem. It is a business problem… Meaning that you are going to be learning. The business is learning. The reason why the business keeps coming to you and asks you to change a feature, or add a new feature, or fix something that they thought was going to work one way and now it doesn’t work, is because they too are learning.

So when the business comes to you and says “Hey, I need you to build this software. And here’s the requirements doc”, however little, however much they give you… And you as an engineer, you take that spec, with your team, or your manager, or whatever it is, and you go build software. You’re building that software on certain assumptions, and then you come to find out later on that, okay, some of those assumptions were incorrect, because either we didn’t have enough data, or whoever was supposed to give us the correct answers made some assumptions based on bad data… Whatever the case may be. It’s supposed to be a learning process for every party involved, the stakeholder and the people building the software. So that learning experience is going to generate some artifacts. It’s going to generate some learnings. That is inherently what technical debt is. It is some assumptions that were made in delivering a piece of software, but now that we know better, how do we address this?

So to me it’s not that it’s bad code. It’s bad assumptions, leading us to say, “Okay, well, because we had to create these classes, or we had to build this functionality, or write these services that are now no longer valid, or now need to change, because now the business knows better”, then we’re like “Okay, well, crap, we have to deliver this new set of features that they’ve just brought us, we have to do that next week… We don’t have enough time to go back and fix all the things, all the assumptions that have been made.” Now we have to say “Okay, well, how can we patch these six other things just well enough to allow us to deliver on this new set of things that has just been brought to our doorstep?”

[00:48:23.09] So you do that enough times, of course the technical debt is going to mount, and it’s going to mount, and it’s going to mount, to the point where teams can’t move fast anymore. And then at that point you’re like “The business doesn’t really want to give us time…” So every time you hear the myth of “Oh, I wish the business could just give us a couple of months for us to go fix some things.” That is an expression of frustration from the engineering team basically saying “Ah, We never get time to fix those things we know we ought to be able to fix, in order to move faster. We never get that time from the business.” That’s a point of frustration.

No developer realistically believes that they’re going to get six weeks, eight weeks to go just fix things. But what they are saying – it’s a cry for help. What they are saying is “Ah, I wish addressing our technical debt as we proceed, as we go was part of the culture.” And to me, that is the biggest issue I’ve seen time and again, be it very large companies, to very small startups, that are designed, that should be accumulating technical debt, because they are learning a lot faster than the larger organizations, based on my experience… They are generating technical debt faster. The problem always comes, big or small, when you don’t address technical debt. You just keep on building, building, building, building, patching things, making just enough so you can build the next set of things, never addressing and going back and fixing the old things. To me, that’s the biggest issue.

I don’t know if I agree with that first premise you had. I think, a) there’s a little bit of a fallacy in there, of “Oh, I’d rather live in this timeline, or this time period.” This is the only time period we live in, so this is what we’re used to. I think that that makes a mistake of assuming that this technology has been unilaterally good, or even net good, and I don’t know if that’s the case. I think that there have been a lot of very, very bad things that have happened to society as a whole as a result of the technology and software that we have, that were much less severe in the past because we didn’t have the software enabling it. But I also think that - sure, we have all of the software, it has built the modern world, but the question is not “Is the modern world good?” or “Are we doing good?”, but “Are we where we should be given the technology and advancements we have?” And I think the answer to that is unequivocally no.

I think the advancements we’ve had in hardware kind of prove out that software has not improved nearly enough over time. And in fact, it has regressed some. Our hardware has gotten tens of billions of times better over the last 60 years.

And I would not say that our software has gotten – maybe it’s gotten a thousand times better over the same time period, but we’re orders of magnitude off there. And I think the reason we’re orders of magnitude off is because we – I think we look too much, we focus too much on the micro issues, at the end of the day. Like the kind of “How did we accumulate all of this technical debt?”, instead of asking “Why is it that we have technical debt at all? Why is it that we assume that you have to have this thing as part of the process? Why is that a base assumption?” Why can’t it be that we know how to build things because we’ve been building them for a while, and for the most part, for most systems, you build it, and it works, and you move on to the next thing. And of course there’s going to be problems. It’s not to say that it’ll be perfect every time, it’s not to say there will be no issues, but that the amount of problems is small enough that we don’t constantly feel the need to have to spend months redoing things and reworking things.

[00:51:55.24] I think it’s a failing of us as an industry that we get ourselves in this position over and over and over and over again, and I think it’s from those same base level things. People pick up technology off the shelf. They don’t actually understand what the core parts of it are. They think that they need to be using it, and they quickly find out that no, we don’t know how to operate this, and now we’ve got to go back and learn, but “Oops, there’s no time to learn.” And so we get stuck in this cycle of just being trapped with this software, and trapped with these things that we can’t really pay down over time as much as we want. But that’s not necessarily because we made those choices. It’s because somebody, somewhere made those choices and said “No, you must do these things. No, we must operate this way.”

We as an industry say “No, of course you should be using Kubernetes. Of course you should be using gRPC and GraphQL and OpenAPI, and all of these things. If you don’t want to use these things, you’re just wasting your time.

You’re just trying to reinvent the wheel. You’re not really doing it.” But at the end of the day, how do you actually learn how any of this stuff works if you never actually build it yourself, if you never actually sit down and learn about it or understand it? So I think that’s what we’re really facing.

Once again, I do not believe that the complaint about technical debt is actually about technical debt. I think that’s where we’re probably differing, Johnny, is that it really feels to me like this technical debt thing is a symptom of a much larger cultural problem that needs to be resolved, which aligns with also the dislike of the hustle culture, the dislike of this capitalistic layoffs, and lack of job security.

So it just feels to me like that’s the root of it, and that’s where we have to start addressing the problem. It’s like, how do we make it so that at the end of the day we don’t actually wind up with much, if any, technical debt? I want to believe that we as an industry can get to a point where we can successfully build a software project, and have it run for more than a couple years, and not want to burn the whole thing down and rebuild it. I feel like we should be able to get there. I feel like that’s something we should be able to do.

I disagree. I just feel like the idea of building something that will last, and will not need change over a period of years, the pace of needs, how we operate technology, innovating - I just don’t think that’s realistic. You’re always going to have to iterate. You build a platform that does X thing… We need to then have some other thing needs to be added to it, some other thing… The way that people want to interact with it changes; the world changes and we should be flexible and change with it.

The way I think about it is you launch something and you should be ready for that to become tech debt, old, pretty fast, because you want to keep iterating and innovating. I think there is an issue with giving the time to innovate and iterate… An example being a couple of years ago my team built a email platform that was for email. And the requirements at the time was like “The New York Times is never going to use this platform to send push notifications ever. We’re just not. This is the email platform for emails. These are the requirements.” You know what’s coming, Johnny…

[laughs]

You best believe, a year later they were like “Oh, actually, the push platform that we’re using right now for breaking news is not working. It’s not sending fast enough. So we need you to use your platform to build it and to send it, and you have three months to do it, because our contract with this other third-party vendor is expiring.”

So yeah, we have to shove a push platform into email infrastructure. And yes, as soon as they got pushed via the platform, they wanted targeting. They wanted fun images. They wanted gifs. And could I get investment to give my team a year to actually build a push platform that worked? No, not until last year, when nothing worked, and we had so much tech debt that the only thing we could do was do a complete greenfield rewrite, which we did… But it had to get to that point, which was the frustration, because there is such a pressure to keep iterating. “Oh, it works good enough. Can we build on top of it?” “Yes.” “Should we?” “Maybe not.” “But can we?” “Yeah, I guess we can.”

Where I think the problem lies is finding that balance. I don’t think we’ve found the balance between how do we continually innovate and pay off tech debt so it doesn’t become this whole big thing, while also continuing to push business value, continue to iterate, continue to be flexible and change, and be okay putting software that sucks? That’s why we have MVPs. MVP, you get it out there, see if it works… You might have to completely rebuild it.

[00:56:30.15] I don’t think tech debt is bad. I think it’s good, in a way, but only if you make sure that you don’t get too much, to the point where it becomes such an issue, and it becomes this negative thing that is pulling you down as an engineering team.

But I feel like even in the example you just gave, I think we all – when you said, “Oh, this will never be used for push notifications”, we were all like “Of course it’s going to be used for push notifications.” We all knew that. So I feel like that is part of the risk calculation. We didn’t do – I mean, I’m abstracting. Not you particularly; there’s company dynamics and all of this. But in a hypothetical version of this, I feel like that’s a mistake of risk calculation, of saying “We’re building this email platform. They’re telling us it’s not going to be used for push, but it probably will be. It’s probably going to want to have all these features that we’re going to have at the end of the day. Is there a way we can design it such that we can add that functionality, we can turn that functionality on when they need it?”

See, but then when we do that, you call it premature optimization. So…

I mean, this is the thing. I think we’re ping-ponging between two extremes –

We can’t have it both ways.

…and we need to be in the middle. This is what risk estimation and calculation is all about. It’s like “Okay, how much do we actually believe that they’re going to come back and say this to us?” Is it a little bit? Okay, then you do a little bit of optimization. Is it “They’re most likely going to do this”? Okay, we do a lot of optimization. Is it “They’re probably telling us the truth and they’re never going to do it? Okay, then we will eat that risk.” But I think at the end of the day what we have is unmitigated risk. And we as an industry have not gained the ability to sit down and do like actual risk calculations. We do it all based on our guts, and I think our guts are wrong a lot.

That’s an oversimplification. I’ve been in large enough companies where some executive in some completely unrelated part of the business hears that there’s some tech somewhere, built by some team for a completely different purpose. They’re like “Hey, you know what? How about we repurpose that technology to do this thing it was never intended to do?” Now here you are, all of a sudden, you have a team that is taking some other project, and now has to re-engineer and refactor it, do whatever they need to do in a very short amount of time to make something work and fulfill a business purpose it was never intended to do. And guess what? They’re taking whatever that technical debt that came from the original project, that it accrued over years - all of that stuff is being brought into this new thing. And then now you have a new team, that is responsible for tech they didn’t write, that they have to change and refactor somehow to fulfill a brand new business use case that it was never intended for.

So it’s very easy to say that we don’t think far enough ahead, that we don’t do enough analysis of risk ahead of time… But when we try to do that, we get penalized because we’re doing premature optimization, or we’re doing the YAGNI thing… “You ain’t gonna need it, so don’t build ahead of time.” There’s so much “best practice” that is shoved at us in order to try to keep things simple. So we can’t both anticipate future needs, and try to keep it simple at the same time. And I think we’re trying to make this an engineering problem. We’re trying to make this a developer problem. It is not a developer problem. It is not a programming language problem, it is not how good the code is problem… It is a business problem. And as long as the business doesn’t work with engineering to move things forward as we both learn, and allow for time to actually pay back some of the technical debt that is naturally going to be accrued; that both parties must understand, that is a natural artifact of writing software. It’s called soft ware for a reason.

[01:00:15.07] When you have hardware, you build it, you have a board, you put all the circuits on it… The next time you need to iterate, you create a new board. You don’t go and change the existing board. You create a new one. You ship a new product, a new version. You think the board you have on your iPhone 15 is the same one that was in the iPhone 10? Of course not.

So it’s easy to say the hardware has come a long way… But when that iteration is happening, you’re not getting the same thing. You’re getting a brand new, redesigned product. Software - the same software that was built for iOS 1, I’m sure there’s a large portion of it still is in existence, running in the latest iOS now, unless it’s been rebuilt in between now and then… But it’s the same software that has been evolving over time. It’s not the same board that has been evolving over time. It’s not the same physical board we ship to you. We iterate on the design of it and we ship a new one. So to me in my mind it’s a very different iteration and maturation cycle… But again, I think this is way more of a business problem than it is a technical problem.

I’ll just say that – I mean, that’s not really what I meant by like the whole hardware versus software thing. And also, most hardware today is actually really software. Yeah, sure, the board – like, you program hardware; there’s not a big difference between the two, to that degree. But I do feel like yes, in this current era we are, I think that we are stuck in this position where the business is kind of erratic, we’re kind of erratic… But I want a future where neither of us needs to be – where we’ve optimized for the right set of values. And I think that that’s where we have this mismatch. Like, do we want things to be simple, or do we want to be able to execute quickly? I think we’ve shown over the past 20+ years that we cannot do both of those things. You can’t have “You ain’t gonna need it” premature optimization mentality, and also be able to build software without technical debt. I don’t – I think that’s where we are as an industry. We’ve reached the point where we are at an impasse. We have to sit down and decide what are the things we’re gonna prioritize, and how are we gonna prioritize them, so that we’re not doing this anymore, so that we’re not coming back every year saying “Oh, we’ve gotta add this, oh we’ve gotta add this, this is so much pain… We’re gonna have to like burn it all down and rebuild it.”

I think that at the end of the day what I hope for our industry is that we can shift away from that; that we can make software engineering - perhaps “boring” is the right word. A boring endeavor. Something that is just part of life, and it works for the most part, and it’s challenging, yes, but it’s not stressful. I want challenging, but not stressful for us building software, no matter what software you’re building. And I think right now it is both challenging and stressful. And I think that’s what these survey results are saying, and I think that’s where we as an industry need to sit down and say “How do we fix this?” Because I don’t think saying – like, yes, it might be a business problem. Okay, then how do we get businesses to understand or change what they’re doing so this isn’t a problem? If it’s an industry problem, we have to sit down and say, how do we change as an industry what is happening here?

But clearly, I mean, I think we can all agree - I hope we can all agree - that there’s a problem that needs fixing here. If 80% of people are unhappy, that is not a long-term tenable, healthy environment for our industry, and we need to fix that, and we need to find some solution. Is that something we can all agree on? Is there disagreement there?

Agreed. We can agree to agree on this.

Okay. We’ve agreed to agree, even though we’ve been disagreeing for the past half hour. [laughter]

It makes for lively discussions…

I feel like we could have a whole other episode about this… As someone on the business side, I have a lot of views.

Yeah, let’s queue it up. This feels like another hour talk.

It does, yeah.

But for now, I think that’s it. I think we’re good, we’re done… Sorry, folks, we don’t really have time for unpopular opinions… You’ll just have to wait for the next episode.

I think we’ve had enough spice, and we don’t need unpopular opinions. We’ve got quite enough of spice on this one, I think.

This was a spicy episode.

Indeed.

Well, thank you, Johnny and Angelica, for joining for this wonderful, as you said, spicy discussion… And thank you, listener, for listening as always, and make sure you tune in next week for that episode with the Go leadership team. Bye for now.

Sweet.

See ya!