How things get done on the Go Team

Hello, and welcome to Go Time. Today we have a very special episode for you. I’m joined by Cameron Balahan, Sameer Ajmani and Russ Cox from the Go team at Google, who are going to be talking to us a little bit about how things get done on the Go team. So how do they decide what to improve? How do they decide in what order to improve these things? As well as deciding what to work on, when… Are they going to work on it now? Are they going to work on it next week, next year? And then we’re going to hear a little bit about how they individually feel about the future of Go. As they look to the future, what do they see? I am so, so excited to have you on the show. Just to give a little bit of an intro before we dive in… We have, as I mentioned, Cameron Balahan, who is the product lead for the Go programming language at Google. We have Sameer Ajmani, who is the engineering director at Google leading the Go programming language team, and then finally, we have Russ Cox, who is a Go primary programming language tech lead at Google. Hello!

Hello.

How are you all?

Hello, Angelica.

Good to be here.

How are we doing?

Doing really well.

Excited to be here.

Glad to be here.

Awesome. So we’re going to start with a very basic question… For those who may not be familiar with you all in the Go space, or perhaps might not have seen some of your many wonderful talks that you’ve given, I would love to hear a little bit about how you all ended up where you are today. How did you get into Go? And perhaps we can start with you, Russ.

I got into Go because I had worked with Rob Pike and Ken Thompson before at Bell Labs. I was a summer intern in college there. And so when Rob went to Google, I kept in touch, and when he was starting to work on Go, I was finishing up grad school, and he said “You should come join us.” And I did. And I’ve been working on Go ever since. So it’s been quite a few years at this point. It’s hard to do the math. 16 years.

Awesome. And then Sameer, what was your Go journey?

So I started at Google in 2004, mostly building systems in C++. And rounda bout, I want to say 2010, Rob Pike came to give a tutorial on this new programming language that he and his team were developing called Go… And I got really into it, I thought it was really cool, and in particular I could see how it would simplify a lot the programming I was doing in distributed systems in C++.

So I started contributing on the side. Google’s famous 20% time - I was using that to contribute to the Go libraries inside Google. And eventually, Russ set up a meeting. We knew each other from MIT, from grad school… He was like “Hey, do you just want to work on Go full-time? We’re looking for someone to work on building out our library suite inside Google.” So I joined the team in 2012.

And then Cameron, how did you come to hear about Go?

So I’m the relative newbie here… I’ve only been at Google for four years. And I’ve had a rather unfocused, maybe, career path up until this point, which probably wasn’t optimal in hindsight… But I actually was a lawyer for a little while, and did something totally unrelated to any of this. And I then worked in high frequency trading for almost a decade, where I worked a lot in C++ and a bit of Java, sort of writing low-latency trading infrastructure… And I always had an appreciation for the stuff that’s close to the metal, and knowing how things really work deep down… And so when I joined Google and Go was an option, it was like “Wow, what an opportunity.” So I guess I joined Go because I got lucky.

So did I.

I think we all feel very lucky to work on this project.

That’s awesome. So my assumption is - and a clarification for those who might know this - that Russ, as you joined Go, it was very much like at the beginning, a few people working on it… And then it grew, and then Sameer, you joined, and then a fully-formed team at Google when Cameron you joined. Can you tell me a little bit about how the Go team is made up now? Kind of what that journey was from kind of someone had an idea, started working on it, and then Sameer, you used your 20% time on it… And then what is now a very fully-formed, fully functional team.

Sure, I can speak to this. So in the beginning, it was three people. Rob Pike, Robert Griesemer and Ken Thompson. They were soon joined by Russ Cox and Ian Lance Taylor. And those five I think of as the initial Go Team. By the time I joined in 2012, I think there was about a dozen people, and that was right around when they were launching Go 1.0. So you can say I joined just as the team was shipping 1.0.

I started managing a subteam on Go focused on Go inside Google, and then by 2016 I shifted into managing the collective Go team. We were over 20 people at that point. The Go team subsequent to that continued to grow, particularly as we expanded into the cloud area, and aligned with Google’s cloud business. That was a substantial part of our growth. And that’s also around when we started building out our cross-functional team, the people beyond our core suites, where we started having a product manager, and UX research help, dev rel partners, program managers and things like that. So we really started learning how to run as a team within the broader Google Enterprise, and moving beyond sort of more of a skunkworks project. The team has shifted and changed a lot in the years since. Exactly how we organize internally depends on where we’re focused, and we’ll speak more about that later in the podcast.

And then having this internal team that is the Go team, but then also having this very rich open source community… Go is a very active community, we love to see each other, we love to talk to each other… But I’m interested to hear a little bit more about how those two communities interact, when you have a formal team, but you then you also have just developers out there contributing, and you have this really rich open source community.

Yeah, I mean, there’s lots of different levels of interaction. The most informal is probably like Slack, and the mailing lists, where the people on the Go team are just participating… And maybe they have a little bit more knowledge to share, but it’s sort of equal participants there. And then you have the issue tracker, where a lot of our work gets done, and any contributors who want to come and hang out on the issue tracker are always welcome. And there are people who it’s their day job to hang out on the issue tracker, and so you see them a lot, and then there’s people who are there because there’s one issue they care about, or there’s people who are there because they just like doing it, but they’re still not getting paid to do it… And so you get varying levels of sort of activity, and that’s great. It takes sort of all the different levels to really make things work.

And then the most formal way that we have that we interact is the proposal process, which we’ve started quite a few years ago now. I wrote a whole series of blog posts about thinking about how you manage an open source project, and how you make decisions, and we created the proposal process to have kind of a formal way to involve everyone in making the decisions about something as little as what new function we’re going to add to the strings package, or as big as “Should we add telemetry to the Go tool chain?” So it really just sort of fits all of them. But the goal of the proposal process is to have an explicit place where anyone who wants to contribute and be part of that can be part of that.

So typically, in terms of how the Go team works, I know some of you mentioned that the makeup of the team shifts based on the focus of the team, of where you think that the innovation in the language lies… Russ, you mentioned the proposal process… I’m interested in kind of understanding - and also Cameron, from your perspective as the product manager - how do you think about all these different kind of streams of ideas? You have your proposal, you have kind of your obviously personal opinions on where to innovate within the language from your various different roles… And then you have the actual Go team of engineers, who I’m sure have an abundance of ideas. How do things really get done? How do you decide “Okay, this is what this team is what backing on?” And maybe because you mentioned it, Sameer, if the team is changing, if you’re changing the makeup of the teams, how do you make those decisions?

That’s a great question. So the current Go Team is largely organized into three broad subteams. We have what we call the core sub team that owns the compiler, runtime, linker, and runs the core release processes. We have the tool subteam that owns our build system, the Go Command, as well as our IDE plugin and the gopls language server backend, that is separate from the core release process. And that’s been a more recent area of investment, really. The last, I think, five, six years we’ve really focused on building our IDE experience.

And then even more recently than that, we’ve built out a specific subteam focused on security. So about several years ago, there were several big supply chain security attacks, open source supply chain security attacks in the news. Codecov, SolarWinds… It was around the same time we were also developing the Go module system… And we recognize an opportunity for Go to really try and solve some of the supply chain security concerns within our module system and build system, and to some in the language itself. And so we embraced that and said “We’re going to focus a subset of our team really on Go security, how we handle security reports, how we do vulnerability scanning, how we think about the composition of Go programs…” And that has worked out very well. And this is a case also where we need to align with Google’s interests in Go, and the concerns of Google’s customers who are using Go, as well as our internal systems using Go.

Just to sort of add to that, I think one other way to think about this is the two primary stakeholders of Go, one being sort of the Go users and the Go community, and what value they’re looking for from Go… And then Google. And so Google’s obviously interested in making sure that its internal systems work well, that its developers are happy, that its systems are reliable, secure, things like that. And then it turns out that Go developers want that as well, external Go developers. And another thing that Google kind of wants is for those external Go developers to be successful and happy, not just out of the goodness of its heart, but because actually then those developers, they’re able to be more successful themselves, get faster time to value, or build more reliable software, more secure software… And all those things sort of contribute to Google’s developer platforms, and then also just its bigger businesses in general; it’s sort of in its interest for the internet to be a well functioning, good place where companies are able to build their products and get them out and get users using them.

So for that value for users, I think the thing that we’ve honed in on is that we know there’s this productivity thing, where - yeah, there are a lot of productive languages, but Go has got its focus on keeping things simple, and then sort of a whole platform where everything connects together to make sure you have kind of end-to-end solutions without having to figure things out with third party tools and whatever else.

And then it also simultaneously makes sure that that software that you create productively is some of the most reliable, secure, performant software as well. Because that’s really important, to make sure you’re not spending your time updating things, or fixing stuff. You can focus your resources on building new stuff.

So that’s actually provided a pretty good lens to think about everything. Like, “Well, if we build something new, how does that further that mission of making sure that our developers everywhere are being more productive, and they’re building production-grade stuff that will make them more successful?” That makes Google happy, that makes our users happy, that makes the community happy, and then the Go team gets really happy about that, too.

Break: [13:53]

So Russ, what is it like being the tech lead on the Go team? Like, what is your day to day? Are you just spending all day talking to the teams, and thinking about your own interesting ideas? I’m very intrigued to understand what your interaction with these various teams are, and what does it entail to be the tech lead of this huge language that you have such invested users of? I mean, people who – you know, which we’ve seen somewhat in the past, you make a change and some people love it, some people are in uproars. So I’m interested to understand how you think about your role, and just what’s it like.

[laughs] I don’t know, it’s hard to say what it’s like. I’ve been doing it for so long, it’s like a fish in the water at this point. But part of my job is to try to reduce the number of uproars. I think that that’s definitely – that’s what we learned with the proposal process, was we needed more ways to keep people involved, and have a meaningful way for them to contribute and make sure that they were heard. And so that’s really an important part. And I think since we really put the proposal process in, we haven’t had those kinds of problems.

Telemetry, for example… We’ve only heard positive things about telemetry. It’s really been amazing, and largely because we made it opt-in, and in response to feedback from the original discussions. And the fact that it’s opt-in I think really just makes everyone very, very comfortable with it, and the fact that we’re sharing the data back out, so that it’s not some sort of proprietary dataset that only we have access to I think is the other important part.

But as far as like day to day - it depends. Sometimes we’re working on something where I get to write some code, and I can spend a week or two writing new code, and that’s always fun… A lot of the time it’s kind of prototypes for things, where I expect that some other part of the team will take it and maybe completely replace what I’ve written, but I sort of prove out that “This is possible. This structure can work”, but then let the team do the real version.

And sometimes it’s just a lot of meeting with people, and reviewing designs, and talking to them about how things are going and what they’re working on, and trying to kind of steer things in a useful way. Even just something as simple as someone beating their head against the wall for some problem, and I can say “Oh, we have this thing over here that we haven’t talked about in a few years. That would probably help.” Just sort of being that resource is also incredibly helpful. So it varies day to day quite a bit, but… The days that I get to help other people are good, the days that I get to write some code are good… In general, it’s mostly good days, so it’s good.

One of the things I’d add to that is the Go team has weirdly a high number of really talented people on it. I mean, I think there’s a lot of that everywhere, and maybe at Google as well, but the Go team just sort of a little bit more than that, I think… And then Russ just kind of sits on top of all of that and is like the paradigmatic example of a weirdly talented and rounded person… [laughter] I’m always sort of trying to find the place where he’s not talented, but so far pretty unsuccessful.

Russ, you’ve gotta put “weirdly rounded” in your tagline now.

If I get the pleasure of introducing you at GopherCon, I’m gonna say that… “This weirdly rounded individual…” [laughter] Okay, so you have all these wonderful ways that you crowdsource ideas, proposals, of course you have your own ideas and proposals… But when it comes to “Okay, what are we working on?”, we have all of these interesting avenues, we have all these interesting things that we could improve… How do you – I mean, do you sit down and discuss all the ideas that are coming out? Do you sprint-plan together? What is the actual logistical process for saying “We have all these awesome ideas, but what are we actually going to do with our time?”

The planning is more sort of distributed, in that the three different subteams are really kind of in charge of their own planning. And so there are things that come out of the proposal process and things like that, where we’ll talk to people and say “Hey, this seems important. We should try to figure out how to work this in”, but it’s not sort of a [unintelligible 00:23:07.26] mandate where I’m deciding what each person is going to do next week, or anything like that. I don’t think that would be productive.

But in general, the important thing is to set goals so that we’re sort of all pushing in the same direction. And at the very beginning of Go, the goals that we set were that we wanted it to work for the two different types of scale that I’ve talked about in the past, which are sort of production scale of scaling for lots of machines, and lots of data, and all that sort of computer scale… And then also the people scale of scaling to large projects, with lots of different people working on them… And even if you are a small five-person team, if you’re using open source packages, now you’re also working with hundreds or thousands of other people. And so we want Go to work well both for people scale and machine scale.

And at this point in Go’s sort of evolution in life, it’s pretty difficult to make changes. Any change, especially like language changes - you start in the compiler, but then you have to fix all the tooling around the compiler, and gopls, and the documentation, and all that sort of stuff. And even library changes - we want to make sure that this is an API that we want to support for the next decade before it’ll make sense to revisit it.

So we’re really trying to make decisions where we’re going to be happy with this for 10 years. That’s sort of the bar that I’m aiming for. And so, a lot of the question of what do we do is based on “Does it serve the goals of kind of production scale and people scale?” And “Is it something that we’re happy with today?” Because no is temporary, and yes is approximately forever. And so it’s really important that we say yes to the things that we’re really sure about.

And then for planning, sometimes things come up where there’s an external or an internal use for something, and we give that some more priority… Things like the recent work on compatibility is an example of that, where the Kubernetes team came to us and said “Look, you keep breaking us in subtle ways, that are not actually violations of compatibility policy…” You know, you fix a bug. For example, the sort of canonical version was that we had IP addresses with leading zeros. So if you say 18.26.014 - like, is that decimal 14, or is it like a 12 and octal? And it turns out all the BSD IP address parsers treat that as octal. And it’s like “Wow, okay, I didn’t know that when I wrote the IP address parser in Go”, and we treat them as decimal. And so if different tools are reading them, they actually disagree on what it means. So we changed it to just disallow leading zeros.

We can’t change what we used to return, but we can say “We’re not going to parse that anymore.” And now at least if we accept an IP address, it won’t mean something different to someone else. But the Kubernetes team was rightly worried that there might be configs out there that have written leading zeros and [unintelligible 00:25:54.00] decimal, because that’s what Go gets you… And it’ll break people. And so that’s an example of the sort of change we used to make, where it kept them from updating. And so we did a whole lot of work around compatibility that I talked about at GopherCon a couple years ago, to make compatibility even more compatible. So those kinds of changes now, they don’t trigger until you also update the Go version in your go.mod file. So Kubernetes can move to a newer tool chain, but leave their old Go version, and then still have the sort of old bug for bug compatibility of the older versions. And so that’s an example of something where we bumped the priority of that because there was a team that came to us and said “Look, we can’t get you security updates, because we can’t update to new Go versions, because there’s these subtle breakages.” And so a lot of what we do is there’s some specific impact we’re going for. It’s not just in isolation.

I’ll build on that and say that there’s two planning cycles that are sort of interlocking and interweaving in the Go team. There’s our release planning cycle, which is public, and everyone sees like “Hey, what’s going into the next release?” And the tools team has their own ones around IDE releases, and the security team has to deal with security releases, and things like that. And then there’s – we’re all Google employees, so we’re part of Google’s planning cycle. And there tends to be an annual planning cycle where we do OKRs, objectives and key results… So Cameron and I in particular are dealing with our Google stakeholders and saying “Well, this is what we’re working on and why.”

And so to build on Russ’es compatibility example, Kubernetes is a public open source project used by every major cloud provider and more besides, but there’s also the Google Kubernetes Engine team, which is a major, huge revenue product for Google. And we could say “Look, here’s a partner team. In order for them to serve their customers - which is a major revenue generator for Google - we need to address this problem and how Go changes itself, and how we deal with compatibility.” So we’re able to sort of line up “Here is Google’s stake or Google’s interest in this, and here’s why this actually requires change on our end”, and we’re able to get that additional stakeholder from Kubernetes saying “Yeah, that’s actually really important to us.” But we try to do such changes in a way that’s not bespoke to Google or bespoke to Kubernetes, but is a general lift for the Go ecosystem.

So I’m particularly really thrilled with the compatibility work, because now we can make changes to improve Go. So the recent change that we made to loop variable, range variable scoping and shadowing fixed one of the most painful gotchas in the Go language, and it really would not have been possible without some of that compatibly work. So we look for these win/wins whenever we have a specific customer, or a specific urgent need that we need to satisfy and say “Okay, how do we solve this problem in a way that generalizes and brings value to the whole ecosystem?”

One more thing I’ll add is – and maybe this is implicit in what Russ and Sameer said, but we have these three teams, and thinking about these end-to-end solutions is another part of the planning process. So it’s not just the compiler and the runtime, but there’s also, for example, the IDE team that has gopls, and the Visual Studio Code plugin, and other things across everything. Some vulnerability work we did with the security team… So last year, when we introduced vulnerability management, or [unintelligible 00:29:01.01] there were several different surfaces to think about. One is the command line, with the govulncheck tool, another is the IDE, where vulnerabilities kind of surface in the code as you’re writing them, another is pkg.go.dev, where maybe you want to get information about the known vulnerabilities and the dependencies you’re thinking about using… So we try to create these – we’ll use all of these different things that we’re building to sort of create a coherence across what we release. That’s extra powerful. And we’ve actually found that the Go community will pick that stuff up pretty quick. They will all use – and say in another ecosystem maybe there’s a variety of tools you could use for the same job. In Go, most users will use our stuff, and so we’re able to kind of move them along into these coherent end-to-end solutions that I think makes it easier for everyone to sort of adopt these things that they see value in, like better sort of vulnerability detection and remediation.

And I want to add on to that that – even though these three subteams, they work closely together where you don’t really feel that on a day to day basis… And Cameron explained for vulnerability checking obviously security was involved and tools were involved… The compiler runtime were also involved with the core team, because there were standard library changes around getting the information out of the binaries, there’s linker changes and compiler changes to embed the information about which modules were used to build this particular binary… And so that was really a whole team effort. And on a day to day basis, we don’t hear – you’re not thinking about “Well, that person is on a different subteam, so I can’t talk to them”, or anything like that. It really feels like one coherent team. And that’s a real strength that we have in Go.

I think of Go as serving two different, but complementary purposes. One is to actually be an end to end system that we control, and that we can – if we decide “Hey, [unintelligible 00:30:54.21] vulnerability management is important”, then we can work from the compiler all the way up to the IDE and make any changes we want, because we have that whole stack in our domain.

And then the other reason that Go is important is that it serves as kind of an example for other systems, because we can prove something out in Go, because we control the whole thing, and it’s easy for us to make whole system changes. And then we can say “Oh look, we did this whole thing. This is how well it worked. These are the things that worked really well. This is what might apply to other systems”, and other systems can take advantage of that.

When we did goroutines, the C++ team inside Google did fibers, and I think that we’ve published some of that work, that look a lot like goroutines… More recently, the checksum database that we did for the Go module proxy, other package management systems are starting to adopt checksum databases too… And so it’s good that Go is a production system that we can all use. And it’s also good that we can set an example where we can show that “Hey, this actually works.” We can use it as a testing ground, a proving ground for these things.

And that leadership element of Go, if anything, is accelerating. So for example when we did vulnerabilities, Russ worked with another team inside Google that was standardizing OSV, the open source vulnerability standard. So one of the things that differentiates Go’s approach to vulnerabilities is we include information not only about specific packages and versions that the vulnerability affects, but even the symbol information, like which functions. If you call these functions, then you trigger the vulnerability, and if you don’t call those functions, then you’re not affected. And we use this - plus static analysis in Go - to filter vulnerability reports and prioritize them. We can eliminate 40% of reports that are not relevant to your program, because you do not call an affected function. By incorporating that into OSV, this is now available to other ecosystems. And if I recall correctly, it’s also the new CSV – sorry, CVE 5.0 standard. So this is a case where Go’s leadership at a particular domain of security and vulnerability checking is very rapidly influencing the industry to follow suit.

And is that from your perspective one of the core benefits of having like a dedicated team, as you mentioned, Russ, true end-to-end, that is constantly able to collaborate and make sure that if there is a change in one place, it’s not going to have a negative knock-on effect, and it really is going to continue to maintain that kind of end-to-end, fluid cohesiveness?

Yeah, definitely. I mean, having Go feel like it’s one system is really, really important to us. And it’s not that doing something in one piece would have a negative effect. We don’t want that, obviously… But the absence of that is not really the bar. Doing something in one place - it needs to actually percolate through and be well-supported everywhere. It needs to feel like it was a whole thing. It shouldn’t be that there’s like a language feature that for example gopls doesn’t support yet. So if gopls support isn’t there yet, or isn’t ready, or whatever, then we’ll hold the language feature back. We don’t want to ship something where it only works in kind of part of the ecosystem. We really want it to be a whole thing.

And so as Go has grown – like, in the initial release of Go in 2009, it was just a compiler. There wasn’t even a go command. You just ran like [unintelligible 00:34:09.05] It was very, very primitive. And now we built up from that. First we had the Go command, and then eventually we got all the cert tooling around it, things like that, and generate, and test… And then now we have the IDE, and all the other – like, we have VS Code Go and gopls, but then all the other IDEs now, or editors, they all use gopls… And so we really keep building up to what Go involves. And now we have the proxy, and the checksum database… It just keeps growing and growing, and we really want to have one coherent system.

So I’m interested to understand a little bit - either with a specific example or in general - how you think about instances in which there is conflict between either the open source community and the internal Go team, within the Go team, within the Go community… Because you have, as we’ve mentioned a number of times, very talented, very invested engineers, both on the Go team and outside the Go team, who just love Go… And I’m sure there’s a number of times where the right way to implement X, Y, Z thing, or whether or not to introduce X, Y, Z feature is up for debate. It’s not an easy yes or no. And I’m wondering how you deal with those kind of decision points.

Yeah, I mean, it doesn’t come up as much as you would think… And when it does, I think that it’s because people have different views that haven’t been put on the table yet. So when I was doing the proposal process reading - and I read a ton of stuff about how other people run processes, and those sorts of things… And I found this great document by John Ousterhout, titled “Open decision-making”. You can just google for it. And he says about consensus-driven processes, he says that “If you manage it correctly, it’s almost always possible to achieve consensus directly.” And the quote that he says is “If a collection of smart people all look at the same problem with the same information, and if they have the same goals, then they’re likely to reach the same conclusion.” So if we’re disagreeing about the conclusion, what we need to do is go back to “Do we have the same information, and do we have the same goals?” And if we disagree, what goal are you going at, and what goal am I going at? And if we can work backwards from that, usually we can talk about the goals. Or maybe there’s information that I have, that you don’t have, that I can share, and now we have the same information… We try to work at that level, and then the consensus tends to follow from that.

So I think that what we learned many years ago - it might have been like 2014 even - when we tried to do aliases the first time, and it really didn’t go well, part of the problem is we didn’t actually present all the information that we had, and we didn’t explain why it was an important feature. And so we backed it out, and then that November at Gotham Go I gave a talk about – I call it “Gradual code repair”, and about just large-scale refactoring of programs one package at a time. And why aliases in particular, type aliases were really, really important to enabling that. You can’t do it without type aliases.

And with that background then, we did type aliases, and there was basically no objections at that point. And the problem was that we hadn’t shared enough about “Why is this important?” Because in a very large codebase, it’s super-important. In a small codebase, where you control the whole thing, and you can just make one big commit, it’s very different. So again, I think all the conflict ultimately comes back to “Do we have the same goals?” and “Do we have the same information?” And we can fix information by just sharing more information. And goals - we can say, “Okay, well, we have different goals… But these are the Go project’s or goals, and so that’s where our disagreement is.” And people tend to be okay with that, at that point. It’s like “Okay, I understand, I have a different goal than you. That’s why we disagree. It’s not like you’re not hearing me.” So I think those have been really important things for us to have learned over the past decade.

And does that translate directly also - and this is a question perhaps for you, Sameer, into how you think about like internal conflicts, or maybe disagreements of what either an approach to be taken to solve whatever problem that’s been kind of prioritized, or what problem are we solving? Is this even a problem, perhaps?

So internal to the Go team – it’s a fantastic team, and I think that the culture we’ve built working through problems is very, very good. The main place I’ve seen conflict is more about prioritization, which is people disagree about what is important to do first. That one person may have a set of goals that they’re working towards, or project they’re working towards, and they’re under some time constraints, or they have other people depending on them… And then another team member depends on that first person to get something done, and they feel like they’re not getting the cycles they need. And this is just a case of limited resources, and oftentimes we need to protect people’s time and focus to let them finish what they’re doing, and then they can shift attention… Rather than – we’ve made the mistake in the past of asking people to try and do too many things at once, and it can be very exhausting, and it slows everyone and everything down.
So we try to have more of the team working on collective, shared goals… And so often, these bigger launches like generics, like vulnerabilities, that do tend to pull the team together, we do our best to try and organize everyone around that, so at least we have a near-term shared goal and we can prioritize towards that.

The other place where we deal with goals and conflicts is when you go beyond the Go team and you’re thinking about broader Google, and how does it Go team rest within Google, and how do we prioritize there. The good news is that Google has been a fantastic steward and sponsor of Go through my experience as a leader… And generally, we the Go team are extremely well trusted to do what’s right for Go and the Go users. And I’ve never felt pressured at Google to do anything that would be bad for Go or Go users. We have at times disagreed about what’s necessary, or what to prioritize, but this is natural in a large company… And generally, the art of what Cameron and Russ and I do is finding that win/win scenario where what’s right for Go and what’s right for Google are well aligned.

I think that leads on to my next question, which is for you, Cameron, because we’ve heard so much kind of from you, Sameer, on like the difficulty of prioritization… How do you product manage Go as a software engineering language? So I’m a technical product manager, I’m product-manage a product… But it’s a platform, it’s backend data infrastructure… But how do you product-manage or how do you think about Go as a product?

I know. It’s weird, right? I think you’re not the only one who wonders that. I think product management is a very role-specific and company-specific and product-specific thing generally. So maybe it’s always a bit nebulous… But the way that I try to sort of figure out what’s best for, say, the things from the prioritization roadmap, vision perspective, and also from what is the value that we’re providing to our users and to Google, is sort of thinking about that thing I mentioned at the start, which is this productive platform for production-grade software.

So we know that that sort of formula is really effective, and that both Go users and Google really get a lot of value out of that. And we look at the market generally in like a traditional product management kind of way, where we think about the product option lifecycle, we think about what’s the angle that we’re trying to solve problems here… It turns out that Go is very successful at solving cloud stuff. A lot of the cloud’s infrastructure is written in Go. The majority of the cloud’s infrastructure is written in Go, and Go has done really well there. That also means that in this new era of like a lot of different sort of supply chain attacks, that Go plays this critical role in all of this infrastructure, so security is really important. And we also know that our users are going to want that, too. So I can think about how that fits into that production-grade aspect of what I’m looking to solve.

So I think of that framework and I think “Alright, what is it that we’re considering? How does it further the goal of productivity and production-grade? How do we make sure that we create these coherent solutions out of that?” And also, does it add the value then that we want it to add, which is that our developers are able to get faster time to value, that they’re able to lower their sort of total cost of ownership over time, that their software is more reliable and secure… And these things that contribute to our users being more successful.

So knowing that sort of path between the mission, the work and the success of the customers is sort of this fundamental guideline that I can use to think about everything else. And then sort of adding to something Sameer just said, I think I’m very fortunate that the product is Go, because - well, one, there’s the users and the community who really love it. And so the thing just continues to grow on its own. The community adds to it, builds libraries around it, finds new uses for it… It just picks it up and uses it in new sort of paradigms of computing; you just see like new stuff, “Oh, it’s written in Go. That’s awesome.” So that just happens on its own.

And then internally to Google, there is the fact that Google itself really respects what Go is. I mean, Google does a lot of cool stuff, and a lot of its stuff has a lot of users, and things like that exist… But Go is really special. I think Google recognizes how special Go is, and how Go users are really happy with it. I don’t know if you’ve seen in our latest developer survey, our CSAT was like 93%, which I think is unheard of in the industry. So that’s cool, and I think everyone recognizes that everywhere. And that sort of makes things easier for me.

For sure. Gophers love Go. So I’d love – we’ve talked a little bit about kind of the past, we’ve talked a little bit about the process and the present… I would love to hear from each of you what you’re excited about, whether it be like what’s coming up, when you’re thinking about the future of Go what are you excited about, what are you thinking about, and kind of what is making you wonder? What is making you think about interesting questions? Maybe we’ll start with you, Russ.

Sure. I’m actually really excited for Go 1.23. I’ve been getting to write a whole lot of code recently, and getting to use the new iterators and new range loops is just really amazing. I mean, being able to write custom iterators that fit into range statements - it’s really kind of transformed the way I write a lot of my code. And being able to abstract things like scans over databases, and those sorts of things is really, really helpful. So I can’t wait for all that to be live, so that other people can write those sort of things, too.

In the sort of longer term, something that I’m sort of wondering about - I think that’s the right framing - is just the sort of overall sustainability of open source. Because we saw with things like the xz attack that – we’ve known for years that critical things are just dramatically underfunded, and it creates this really easy opportunity for bad actors to come in, and really just sort of do arbitrary damage at these low levels.

I asked a roomful of senior-ish people from a variety of companies how much money they thought the xz attack cost. And the general consensus was “Maybe $1 or $2 million.” And maybe $1 or $2 million to get an SSH shell on every machine in the internet almost - it’s pretty good. That’s pretty good ROI. So we need to figure out ways to make those attacks way harder. We can’t keep doing what we’re doing in open source. So that’s the sort of interesting wonder that we have to figure out as an industry in the next decade.

Two things… One is we talked a bit about supply chain security and vulnerabilities - I actually think there’s still quite a lot that we can do there. One of the differentiating features of the Go ecosystem is how uniform it is. Everyone, for the most part, is using the go command to build, and go modules to manage dependencies… That is an enormous amount of leverage. That means that we can do things like run tests when someone is making a change to an upstream package; you can run some sample of the dependent tests to see if the change breaks anything before the change gets committed or marked as the next version. And you can imagine if we could do this at scale, we could dramatically drive down the rate of breaking changes across the ecosystem, which would lower costs for everyone. Not only would it lower costs, but then if you have fewer breaking changes happening, now updating to the latest version gets a lot easier and a lot cheaper. That makes it easier not only to keep up with changes for features, but also to remediate vulnerabilities. So if you have a dependency that’s vulnerable, and you know that updating is unlikely to break you, because the ecosystem has fewer breaking changes, now the persistence of vulnerabilities is much less likely. And so if there’s a supply chain attack, like Russ just mentioned, we have to be able to remediate that at scale much more rapidly. And these are things that the uniformity of the Go ecosystem and the build system and the module system in particular enable at a scale which I don’t think is true of most other ecosystems. So I find that exciting.

The other thing - and I’m sure Cameron will talk about this, too - is AI is eating all our lives, right? Everyone’s talking about AI, asking about AI… The good news is we’ve been thinking about Go and AI for a while now. And even as far back as beginning of last year or earlier in the year before, we’re thinking about “Well, Python users really do like Go, and a lot of them do adopt Go, and Python’s used for AI… So is there something there?” And our conclusion was “Not really”, in the sense that data scientists love Python because it is a fantastic and very ergonomic language, very productive, it works great in your notebooks, and there’s this enormous ecosystem of very rich and good libraries. And we have no desire to try and compete with that from Go. But there’s a moment now, as more and more large companies and enterprises and startups want to build systems on top of AI models, where to Cameron’s point, they want to build production-grade, trustworthy, reliable systems on AI. And now we’re in Go’s house, right?

So the question in my mind, that we’ve all been discussing, is “What is that transition moment?”, where you go from, say, prototyping in Python, and proof of concept and idea, and you’re like “Okay, now we’ve got to get serious. We want to build a product, we want to build a system. We want to build a service, we wanna build infrastructure that uses AI, and has the right checks and balances to sort of deal with AI’s unreliability and hallucinations.” If we want that to be in Go, there’s a transition moment, where “Okay, now [unintelligible 00:49:15.26] And what does that look like, and how do we facilitate that? How do we make Go the language for building production AI systems? I think that’s the next big frontier, and we’ll see a lot of demand for that, and I think Go is a fantastic fit.

So yeah, Sameer sort of stole a bit of what I was gonna say, but the way I would have framed it is that I’m excited for Go’s growth. Programming language adoption is just this fundamentally slow thing. Nobody goes back and rewrites what already works, and we don’t want that either… And if you already have a ton of servers written in one language, you’ll probably make the next one written in that language as well… So introducing a programming language and then having it be adopted - it’s kind of a big deal, and it takes a long time. Just seeing Go’s sort of organic growth, especially in recent years, and - like I was saying, the community loves it, and it just keeps self-perpetuating… I think we’ve reached a threshold now where I see it growing even more rapidly, potentially, or at least finding it in more and more places. And I think part of that is maybe in a specific way related to the AI frontier, but it’s really just more generally like just new paradigms of computing; as those as those occur, those are opportunities where sort of new code is written. Maybe new companies come up and they have to choose a stack, and they’re probably going to use the cloud, and they’re probably going to be concerned about the productivity and how quickly they can get their product out, and they’re probably going to be concerned about the total cost of ownership in the long term, and the security and reliability of the products, and they’re going to find that Go is such a great fit for all those things, and the other languages that they maybe learned a while ago, or seen in production in an earlier time don’t fit quite as well. And that’s pretty exciting, because Go really does then fit the bill. And so I’m really excited to see what happens with that. The sort of growth, generally. So AI as a contributor to growth.

Then you asked about kind of a worry, or something like that… I’m maybe not worried, but just sort of wondering about AI. I wonder what this all looks like. It’s obviously very interesting, and I think that especially in the software development lifecycle, there’s opportunities to cut inefficiencies, and make work more productive… Sometimes though it seems like some of that is oversold, and I don’t know where the actual truth is, but I’m interested to see how that plays out. What the impact of having sort of more robust code completion will actually have on software engineering more generally.

So as Sameer said, we keep our tabs on that, because it’s important to understand the market in which we’re in, and what our users want. We want to be adaptable. We want to provide for all those things for our users that sort of keeps them happy, like the productivity and the other things. So I’d say growth generally, but wondering what AI will kind of make of our whole profession in the future.

For sure. I have one final question, which is more about how you’re thinking about Go and its progression. I know, Sameer, for example, you referenced Python’s role in this space, and that’s not Go’s house, but then this is… I’m interested a little bit about in your minds - and anyone can take this question - is it about making Go the best in the space it is currently in, and really making that, as you said, Sameer, that house the strongest house that you could ever build, with the best materials? Or is it about building a house and a boat and a bungalow, and an airplane…? [laughter] Or is it more – and forgive me if I lost you with the explanation… Or is it more about “Okay, this house is good enough. Let’s now use Go to build this plane”, and really move into a space that Go is not in right now?

So I’ll ground this in just what are the opportunities for the existing house, versus the other opportunities. Let’s look at cloud, for example. Go is a fantastic language for building cloud applications, multi-threaded servers, doing a lot of communication. All the estimates I hear of point at the majority of workloads that could be on the cloud still aren’t. So there’s a huge still future for building scalable software and infrastructure that needs to be reliable and production ready and efficient, and all the things that Go is. So I think our existing sort of base for Go is still really important, and it still requires a lot of investment to do really well.

At the same time, as Cameron pointed out, software engineering and software development is transforming. And we need to pay attention to how that is changing. So I don’t think we know – I remember seeing in your notes, you were asking like “Well, what about ML? What about frontends?” We even looked at Go mobile years ago… We try not to chase next new opportunities just based on what’s hot. We try to be selective. For example, when it was all the crypto and mining stuff, some of the crypto people adopted Go, and people at Google asked “Oh, we see [unintelligible 00:54:56.19] and we basically said “No, we’re good. We don’t need to chase that ball.” But with supply chain security we saw the opportunity to do something that was truly differentiated and was well-aligned with what our existing users would want.

And so with AI - again, we’re not trying to chase the data scientists and the Python users who are iterating on model development. What we’re looking at are people who build in production at scale - they’re going to have to deal with AI, and deal with incorporating AI into their existing systems, and they’re going to deal with models and security and observability. So how do we serve them? So there’s a way to take our existing user base, with their existing concerns, and also think “Well, how are their needs expanding? How is the Go user’s job going to change?” And so it’s both building systems that use models, and also using AI themselves to generate Go code, or understand their Go programs. So I guess that’s the way I think about how our jobs change, is if you are an average Go user, how is your world changing, and how can we, the Go team, facilitate that change?

So if gophers get shiny, new ML Ferraris, you’re gonna have to make sure your Go house has a garage fit for that ML Ferrari?

I love it. That’s a perfect metaphor.

I was just going to add - and don’t forget that there’s the whole Go community that builds on top of Go. The Go team itself can’t build all of those bungalows and those cars and all those things… And just ensuring that they’re enabled to build what they need to build in order to continue – keep Go on that sort of frontier of what people need in order to build their businesses, build software, build what they need to build productively, and at a production grade… That’s an important piece of the thinking, is there’s a whole community on top of it.

Any final thoughts? We’re gonna turn over to unpopular opinions, but I want to make sure if there’s any final thoughts or things top of mind that you’re really excited to share with our Go Time listeners, that I give you that opportunity.

Sure, I have a thought. Go to telemetry.go.dev and enable telemetry.

Love it. Okay.

Jingle: [57:03]

So years ago – well, probably my biggest contribution to Go is actually the context data type, that everyone has to deal with invading their function signatures… Russ and I did the original design together, but I did most of the authorship and popularization of it. And we’ve gotten a lot of feedback about “Aargh…! Wasn’t there a better way?” My unpopular opinion is context is fine. It actually does its job pretty well, and what it does is very subtle and tricky, considering that you have to deal with multi-threaded request handlers, and Goroutines, and deadlines and timeouts, and request [unintelligible 00:57:54.01] values, and… It pretty much just works. And the implementation is simple enough that we’ve been able to optimize it over time, and… Yeah. The sort of risks of context abuse have not become a giant, pervasive problem that we’re really worried about… And for the most part, Go developers seem to be holding it right, and I’m thankful for that.

That’s a great unpopular opinion. Next up, Cameron or Russ, do you have an unpopular opinion that you’d like to share?

Sure. So I was thinking about this a bit… One of them, I think, I’ve shared before in some context, is that I really like Go’s error handling. I really like that. So I’m always confused when people are disappointed in that. But I was also thinking, what about non-Go unpopular opinions, and I think the one that I’ll go with is that I think those sour gummy worms and gummy bears and stuff are like a lot better than chocolate, or things of that sort. And I don’t know, you’re making a face, and I’m thinking you probably think this is a terrible opinion, but that’s the one I’d really like to go with.

Why…?

I’m not sure how to answer why. It just seems like it’s better.

Okay. Is it the consistency?

It’s just the perfect combination of the sweet and the sour together.

Yeah, you know? Like, it’s… Russ, do you agree?

I totally agree. And now that you’ve mentioned those in this context, I feel like we need to get someone to make the little gummy gophers.

That’s a great idea.

We need to figure out how to make that happen.

Oh, my gosh… Calling any gophers who work at food, or sweet processing companies… Slack us on Gopher Slack. Reach out to Go Time. We’ll try and make it happen. That’d be awesome.

I would love that.

Russ, do you have another unpopular opinion?

Sure. Every couple of months I see something on Reddit or somewhere asking “Why hasn’t Go gotten rid of null pointers yet?” And so my unpopular opinion is that null pointers are fine. They’re kind of a fundamental fact of computers, is that memory can be zeroed. And the languages that have tried to hide this, in general - there’s always some clever way you can get to a nil, or worse, uninitialized memory, because they were so sure it could never be viewed… And nil pointer bugs are pretty much the easiest possible bugs to take care of, because at the point where it crashes, it prints an entire stack trace of what the program was doing… And you go to that line, or you look at the parent frames and you go “Oh, that’s how you can get a nil there”, and then you’re basically done. So they’re like my favorite bugs, because they just sit there and they show you exactly what was going on, and then you fix it.

I’m intrigued to see whether that’s unpopular or popular, given your very coherent explanation as to why they are helpful, and we must keep them. [laughs]

Thank you very much for your unpopular opinions. I want to close with just a quick kind of - if people are interested, if they want to either get more involved in Go, if they are new to Go, and they’re like “Okay, I heard this podcast. This Go thing sounds fun”, what would your advice be to them? How do they get involved? What should they do?

I would say Gopher Slack, the mailing list, the issue tracker… If you go to go.dev and you scroll to the bottom, there’s a whole bunch of contact links, and any of those are fine.

Great. And I will link the links that you’ve mentioned in the description to this episode, so wherever you’re listening to it, you can go into the description of this episode and you’ll see a lovely list of links for you to click on.

And there’s certainly the meetups, the conferences, the rest of the community stuff… There’s a lot of stuff out there, and it’s all really exciting, and a really good community. So those are also options for folks.

And if you have any questions, Gopher Slack is where we’re all at, so please hop on that. It’s a lot of fun. And feel free to join the great Go community. Thank you both. I really appreciate your time, and for joining us today. Have a great rest of your day.