Frederic Branczyk: Yeah. Let’s start with Tyler, actually. So I’ve known Tyler for six years, seven years almost… He actually rented a desk from us at CoreOS times, in the CoreOS Berlin office… And he was already – he had some history at Mesosphere, working on Zookeeper as well… And yeah, just any crazy distributed system or high-performance databases that you can think of, he’s had his hands in somehow.

[52:12] I’m also friends with Tyler, I like to go for a coffee with him or something, and we just have common interests. I was talking to him that we’re thinking about building this new database, with these kinds of characteristics, and I’m not sure about our model for transactions. So we just spent kind of several hours together, discussing various isolation and consistency mechanisms… And ultimately, what we ended up implementing is 100% his idea.

Andrea Luzzardi: Yeah, I think they were one of the biggest users of… Twitter also were using Mesos. So it was a time when there were a bunch of tools for different needs. Mesos for enterprises, and Fleet, if you were into the CoreOS ecosystem it maybe made sense… And Nomad for HashiCorp made sense… And then Swarm - it’s this built-in thing, simple to use, and then Kubernetes takes a different approach, where it’s an API… But at the end of the day, the ecosystem standardized so much that it kind of overshadowed all the tools. But back in the day there were five or six different tools, with many users in each, and to each their own, depending on the use case.

Kelsey Hightower: Well, you know, there’s this phrase, “It’s easy to predict the future when you’re working on it.” A lot of us at Google Cloud are working on that replacement. When I first joined Google Cloud from CoreOS, lots of people were running their own Kubernetes clusters across a bunch of virtual machines. This was the norm. And then we had a managed service, Google Kubernetes engine; you can now click a button, and then we would automate all the cluster provisioning, and the upgrades, and so forth. But the nodes were still there. You still saw the machines, and it looked like the cluster you would manage yourself.

And then we came out with tools like Cloud Run, that was based on the Knative spec. And for those that are unfamiliar with Knative, there’s this idea that if you looked at just running a very simple application inside of Kubernetes, you need a deployment object, you need some secrets, some configs, you need a load balancer, ingress service, pod disruption budget, a horizontal pod autoscaler just in case you need to scale, and you still didn’t have things like scale to zero. So we looked at that and said “Okay, what is the pattern that the average person running applications in Kubernetes needs?” So things like Knative was born.

[16:04] Now, if you look at Knative and say “Well, if you think that this is the right abstraction, then you may not need the rest of Kubernetes to run applications in that style.” So now what we can do is have a managed service that’s a serverless platform, cloud-run, we take the same containers, we can take the same Knative spec, and just make Kubernetes disappear. There’s this myth that we are running Knative or Cloud Run on Kubernetes. We are not. There’s no Kubernetes cluster involved, at all. All we do is we allow you to give us the specification at the API layer, and we translate that into our native API internally, so there’s no need for a Kubernetes cluster.

And even the people that need maybe more of the Kubernetes API – so if you just think about APIs and not implementation, then things like DaemonSet, and jobs, and all these other things, you can get that via GKE Autopilot. Again, we make the cluster disappear, but we give you more of the complete Kubernetes control plane.

If you fast-forward this over the next ten years, a lot of these patterns will be available in other platforms. For example, Cloud Run has in beta right now (or alpha) the ability to run Kubernetes jobs.

So I think that people have proven that Kubernetes has been a great prototyping ground for workload definitions and APIs. And if you really believe in promise theory, then we know that those APIs and specs are just the promise that you’re making, and any system should be able to converge or keep them. So this is what I mean by something in the future will replace Kubernetes, because we are actually building the ability to.

Rich Burroughs: Yeah. Kelsey is actually the person who brought me into the community, really, in essence. I saw him speak at a very small event here in Portland. This was in 2015. He was still working at CoreOS at the time. So this was before he was at Google. And he did this talk that – I don’t think the whole thing was recorded at that event, but he did it again, and there is a recording out there. If you google “Kelsey Hightower Tetris”, you’ll probably find it. But he actually played Tetris and he used that as a metaphor for this idea that now we have these clusters, right? And these clusters are made up of a bunch of compute nodes, and they’re really just resources. They’re memory and they’re CPU and storage. And we don’t have to worry anymore about what host an app is running on.

I’m an old school ops guy, and I used to be that person who knew, “Oh, the frontend app runs on these servers”, right? And I could tell you what host everything was running on. And it really appealed to me, this idea that you have this cluster and it puts things on the nodes and balances the workloads and restarts things that die, which is another thing I used to do when PagerDuty woke me up at 3:00 AM… I would go and restart the app that had died. So it was really exciting to me. I was still in operations at that time. It’s been a few years now since I had that kind of role, but I went through a lot of pain in the old days, and I was very interested in the fact that the folks who designed Kubernetes clearly had thought about those same things.

Zac Smith: Yeah. And so we thought it was a good time to sell. I think in retrospect it was maybe a little early. You never know, but it was the right time for us… And I vowed - and this is kind of the turn in the story - I would never get back… I told my wife, I’m like, “That’s it. I’m done with this whole internet thing. It’s too hard. This is gonna be very competitive, it’s a scale business… There’s no way.” And two years later I was sitting at a local bar here in lower Manhattan, and my brother Jacob, he’s like – he got so sick of me bitching about “Well, you know how it should be… You know why it could be this way… You know why we should do it this way…” He’s like “Dude. Zac. We’ve just gotta start another hosting company.” [laughs] And the idea was really that software was just eating the world, and why wouldn’t it also eat the clouds? And the concept of portable software – I’d become friends with a guy named Alex Polvi; he had started CoreOS.

Zac Smith: I mean, I used to in the past. Now it’s a little bit different. I’ve gone from being – let’s call it a CEO of a startup, a company called Packet, which I ran for many years… To now being a busy executive at a Fortune 500 company, which - you know, I have a little bit different set of responsibilities, and part of that is with customers in the field, but a lot of that is also internal and part of our strategy… And you know, I’m gonna call it corporate functionality.

[04:02] So I haven’t been to any conferences over the past few years, but I used to go regularly. I was on the road 2-3 weeks a month, including conferences. To me, conferences have always been – especially things like KubeCon, and earlier I remember being at DockerCon in Barcelona in 2015… And so the best part about these conferences to me is the hallway track; I just love seeing and meeting and hearing from people – you just get that pulse on what’s going on when you can go around that hallway track and see what people are talking about. So to me, that was always my favorite part of going to conferences.

What was my other favorite one…? CoreOS Summit, Monitorama, that was a good one… KubeCon, yes… You know, I didn’t ever go to like the Gartner IT Summits those things weren’t my gig.

Kris Brandow: [01:00:21.04] Yeah, there’s all sorts of… I think I’ve heard one for CoreOS, some called it cOREOS, because if you take the C off, it’s OREOS, so they called it cOREOS… I was like “Oh, okay…” We just do this to ourselves, we really do.

Frederic Branczyk: Yeah. Thank you for making that point. I think I know where you’re going. So we started the company, and a really good friend from CoreOS times, he many years ago at a GopherCon he told me “If you ever start a company, I wanna be the first person to work with you. And he kept his word. In November 2020 he joined Polar Signals. Since then, a couple more people have joined, and in February of this year we launched a private invite-only beta of our product for continuous profiling… And I guess we should talk a little bit about what continuous profiling is.

[27:56] Essentially, profiling itself has been around ever since programming has. When we did our research, we found it had gone back at least to the ’60s and ’70s, because everybody, as soon as they started programming, needed to understand what was happening with the code that they had been writing. What was using the CPU time. And especially in the ’60s and ‘70s it was so much more precious to have CPU time. So profiling has been around for a while. There had been two problems with it - one was for the longest time profiling was incredibly expensive to do in production. You would only do it to specific processes, on-demand, because you didn’t wanna create too much additional overhead.

There was one thing that kind of led to us being able to do this in production and always on, and one of those things is what we call sampling profiling. So instead of tracing exactly absolutely everything a process does, we only a hundred times per second look at what the program does at that particular moment in time, and capture the stack trace of what it does… Because essentially, the stack trace represents what the program is doing, right?

So for some hyperscalers this was already enough to build continuous profiling tools for them to consume internally, because they could do it always-on in production now.

Now, as it goes with so much cloud-native technology and developments, that wasn’t necessarily accessible to everyone… And one of the really amazing things that also have happened somewhat recently has been eBPF. eBPF allows us to capture this data at an even lower overhead, because we can already capture it in the form that we are going to consume it afterwards. We don’t need to use some pre-baked format that may have a ton of information that we don’t need, a ton of detail we don’t need. We can produce exactly the data that we want, and make that exportable to user space, and then ingest it into our storage.

So that was definitely also a really big part of what created a movement… But this doesn’t really have to do with overhead. There’s also another aspect, which is just kind of Kubernetes unifying the observability space, in a way. And I think we might have talked about this in our last session, actually… The way that Prometheus also, and Kubernetes have kind of standardized a lot of terms in our industry. It just makes us all speak the same language.

This is super-powerful, because all of a sudden, when I say pod, and you say pod, we immediately know what we’re talking about. So this is much more cultural than it is technological, but it means that our knowledge is transferable. So this is incredibly powerful… And then the last piece is putting all of this together, eBPF with Kubernetes now allows us to automatically discover all of the containers that are running in our infrastructure, and be able to look at all the CPU time that is being consumed in our infrastructure at once. And the reason why this is so powerful is because all of a sudden we can now say “This stack trace in this binary is what’s causing 20% of our CPU time.” If we optimize this stack trace away, we’re now saving 20% of CPU time in our infrastructure. That’s huge. Think of the banks, automotive companies, any company that has a large cloud bill - they can save millions of dollars with these kinds of measurements. It’s just, the reality is they can do these measurements today.

Adam Stacoviak: On Go Time, Kelsey Hightower - well-known in the cloud space, well-known in the Kubernetes space… Came from CoreOS, did a bunch of cool stuff there, has been really well-known in the Kubernetes space, doing a lot of cool stuff around containers, a lot of things around Docker, a lot of things around Go… He’s been an MC many times, at many conferences, [unintelligible 00:50:36.01] I’ve seen him recently - or at least a couple years ago - at GopherCon… But he wrote a post for us called “Monoliths of the future.” Technically, he didn’t write it, but technically, he did… Maybe you can give a peek behind the veil to the process there… But on Go Time, Kelsey shared an unpopular opinion called “Monoliths are the future.” And he laid it all out there. And we turned that into a post via the transcript, and shared that…

Because hey, what happens often is content will get stuck in a podcast. And we’ve had Alex doing great transcripts for us since episode 200 of The Changelog. We’re now at 400-and-something now? I don’t know what number we’re at. 430-something, I think, if I can recall correctly… So for many years now, basically. So this post from Kelsey - I mean, I think it got 150,000 uniques… What was the number? 200,000?

Kelsey Hightower: [55:58] When I was a CoreOS, when I started to learn about the Raft protocol, Etcd, the implementation of using that to do a key-value store, the biggest thing that I’ve seen was people not really understanding the system. Most people were trying to add more nodes for performance, even though Etcd is the single writer, single leader system… And then people didn’t understand that the mechanism it chose for consistency could also get in the way of availability. We’d make that trade-off of saying “In order to keep the data store consistent, I will take an outage to figure out the consistency”, have a leader election of which one should be the follower based on the data that we have committed in our logs.

People not understanding how that system makes the trade-offs - you tend to get into these issues where, for example, I had people put Etcd in an autoscaling group. So the way that works is three nodes, you have a quorum, everything is healthy; one node crashes, another node comes up. And it looks like three nodes still, based on all the things you can see and observe, but inside of the actual cluster membership API you would see four nodes. And what customers would do is they would be getting lucky for about a year. They would have a machine crash, and six months later now you have five nodes inside of the cluster API, but you actually only have three nodes running. Luckily for you, that’s enough for a quorum.

It’s when you get to that sixth node of this kind of crash/automatic repair that you now have six nodes in the API, and you can no longer reach quorum, because the three other nodes can no longer vote, and now you’re hard down, and the system is down forever, and there’s nothing you can do to recover until you understand that you’ve gotta go remove those three members. But guess what - you can no longer do it online, because it requires a quorum in some systems to actually remove the three dead members from all the other three.

So those are kind of things where things go wrong - people may not understand the trade-offs of the things they’re doing at the consistency layer or the availability layer, and they find themselves in a situation where they can’t troubleshoot… So guess what people are doing? They’re blowing away their entire Etcd setups and starting from scratch because that seemed to fix the problem.

Gerhard Lazu: I think that’s a really good place to start… Because last year, as exciting as it was to roll out that infrastructure for 2019, we were using Docker Swarm, and the big difference was that we didn’t have to install Docker, we didn’t have to do any of that management, because it came with the operating system. We were using CoreOS at the time, and CoreOS out of the box just had Docker, so we didn’t have to install it.

So there were fewer things for our scripts, our Ansible to do, and we could switch to something like Terraform, and we could worry about managing not just the VM, but also integrating with the load balancer - NodeBalancer, in Linode speak - and it was a much simpler configuration. But it still meant that we had a single VM. And some might frown upon that, like “Why single VM?”, but looking at our availability for the entire year, it wasn’t that bad, and any problems that we had were fixed relatively quickly, except one; we may go into that later.

For the entire year, we had downtime less than four hours. That was pretty good for a single VM. So it just goes to show that some simple things can work, and you can push them really far. And I know that Jerod is a big fan of simple things, because they’re easy to understand, when something goes wrong it’s easy to fix it…

I know that Adam was really excited about us going to Kubernetes. We wanted to do that for a while, but the time wasn’t right… And it wasn’t right because Linode didn’t have a simple, one-click Kubernetes story. You had to do a bunch of things… You could do it if you really wanted to, but it wasn’t easy. And then, in 2019, at the end, November, the magic happened. Linode Kubernetes Engine entered beta; I was at KubeCon, I met with Hillary Wilmoth and Mike Quatrani from Linode, we gained access to Linode Kubernetes Engine, it was in beta…

[08:13] And with one command later, we had a three-node Kubernetes cluster. That was really simple; that was the experience that we wanted and we were waiting for. And once we had that, things kind of flowed from there. It was really simple to add all these other components.

Now, compared to what we had before, we had to worry about, I suppose, the migration from CoreOS to Flatcar, because CoreOS became end-of-lifed with the acquisition of CoreOS by Red Hat… So we had to do that migration, and we were approaching – we knew that the end of life would come… So rather than doing that and continuing with a single VM/Docker Swarm complications, we went to something simpler, which was Kubernetes… Because we had this one API, and we could provision everything… Which meant less Terraforming. We didn’t have to provision NodeBalancers, we didn’t have to create volumes and then attach them to VMs using Terraform… We didn’t have to do any of that. This Kubernetes API would do all of those things for us, which meant that it was a much simpler system to work with.

Aaron Schlesinger: [19:44] The last job I had, we were building a platform as a service… And it was all these containers. We started on CoreOS, with their Fleet system. I think that’s deprecated now. It was sort of “We will take a container and we’ll put it on X number of machines”, and that was what Fleet did. And that was pretty powerful at the time, because you had these sort of beginnings of that abstraction, of “I don’t need to care about VMs anymore. I can give an API a container name, and it’ll do its thing.”

And once we got to that point, we then had to start breaking things apart, because a platform as a service has lots of different logical components that don’t necessarily fit together. And this is right along the lines of what you said, Johnny. It has a Git SSH server, it’s got a logging component, it’s got an administrative interface, and a control plane… And the list goes on.

So once we hit that point where we said “We just can’t have a monolith with all of that stuff in it at once, because managing that thing, opening all the different ports and managing certificates and all that - that’s just not feasible for us.” So once we got all of our stuff running on Fleet, we then had to reinvent the wheel and figure out how to do secrets, and distributed locking, and all that stuff. And then Kubernetes came out and then we just adopted all the primitives that Kubernetes gave you… But stripping away the Kubernetes part, even though that was great, and stripping away Fleet as well - the idea that we could have implemented it ourselves would have been painful, but we probably could have - I’m not gonna say definitely; we probably could have… It’s the fact that - yes, we had a technical requirement that stuff was split up, while at the same time stuff could interact with the other stuff.

Service A could interact with services B and C in a way that was manageable and that didn’t require two different operations and release management teams to manage services A, B and C. And for me, right at that moment - and I remember this - I was dreading having to build those systems, to manage all the things and route network traffic and all that stuff. And once we’ve found Fleet, that was when – we went down this road of starting to think about an abstraction, and starting to think about independently scaling, and starting to think about how to organize the team around all these different services, and manage the sort of organizational aspect of this… I started thinking about a lot more things, too… But right then and there was the seed that got planted in my mind, that started me down this whole cloud native road.

Brad Fitzpatrick: So now I’m looking at like “Do I wanna use Flatcar Linux, like the CoreOS continuation project, or do I wanna use K3S, or maybe I just wanna use Podman…?” So I’m kind of like debating all my options now to build a more simple thing to run containers at my house.

Jon Calhoun: And the hard part there is I think almost every open source project has been mostly wrong when trying to figure out how to build a business around it. Even ones – I’m thinking of like CoreOS… I don’t know how well they did, but they had to be acquired, and I assume that if they had a better alternative, they wouldn’t have done that… So you see ones like that, and I’m like “CoreOS seemed like it was doing very well, and… Unfortunately, no.”

Kelsey Hightower: Now we’re talking about the problem with merge conflicts. At least my experience has been “How do we avoid merge conflicts? I know… Let’s start another repository. Let’s have a better API contract.” We’ve been so bad at language-level API contracts, we decided to leverage things like JSON and RESTful interfaces to give us a much harder contract. They’re very hard to violate, because they’re so rigid. You can’t reach behind a class and call a method, because you can’t do that with REST. It’s not easy.

So I think what happens in the team aspect is – I like the idea of modules. When I saw the way Go did modules - you can actually have separate teams building modules, but that’s independent of how you compile the modules into the final deployable.

At CoreOS I remember we used to do this a lot, we used to have a lot of individual modules, and then package main is where the collaboration happened. I would bring in my module and maybe add a route to it, or something. But once you touch that file, it’s only because you’re saying “Hey, I’m now part of the contract. Here’s my route”, but then I would just go do the rest of my work in the module, and allow the build system to take all of our work and combine it together. And if you’re using tools like Bazel, it could be build one big binary, build three little binaries, with flags… But either way, that’s a separate concern, the way you layout your source trees and how you develop code collaboratively, versus how you deploy the results of that effort.

Natasha Woods: I would just go a step further with the documentation… A little story of working on the release team… When we were working on release 1.7 and 1.8, more from the communications side - and I mean communicating with the different stakeholders and contributors, and people that are really giving back, but also the companies that are seeing what’s coming down the line for the new releases, and things like that… We didn’t have a ton of documentation on that. So for 1.9, [unintelligible 00:31:09.01] really championed documenting the process, and documenting everybody’s roles.

So you can go and you can see all this documentation now, but what are each release team member’s roles and responsibilities, what is the previous experience that they should have before they are elected to these roles, because it changes each time for the release… And what are the key steps that need to happen from a timeline perspective. I mean, they’ve always had a timeline, but it was a little bit more detailed. And then we’ve just iterated on it for several releases. So Jace came to me and he said “We need to write down something for marketing.” And marketing is always a very last thought, but there’s so much more into it than just maybe tweeting something out, or calling a reporter to get an article.

[32:03] So I sat down and I documented everything I did for that release, and we implemented it, and then we implemented it for 1.10. Then I was lucky enough to have my second child, and I was on maternity leave, and so I wasn’t able to really update the next person and help them through if they had questions, because I wasn’t available. So they were able to take everything that I documented and implement it from a marketing perspective for the next few releases, which was really great and really helpful…

So some of those things were, you know, learning what is coming down the pipeline from the SIGs, and what is going to make it into the release and what’s not going to make it in the release, and why is this relevant to the audience, and who is the audience; why is it relevant to a vendor, why is it relevant to a customer, why is it relevant to another developer, and then making sure that that is communicated not only within the blog post, but it’s also communicated to any press that are covering it, because you don’t want misinformation out there and confusion to happen. And then how is this being communicated to, say, the companies, the Red Hats, and CoreOS back then, and those types of companies who are following along with the releases, how are you communicating that to them. So we actually created this really great, detailed process out of it.

Obviously, Kubernetes is in a different league, the same league as Linux. But for the smaller projects, you can take pieces of this and see the importance in communication across everything, and also documenting.

Kris Nova: Yeah, and I think it’s important to call out - there’s tooling in this space as well. We’re starting with a prototype; there’s a solution, the operator framework that came out of the folks at Red Hat and Core OS. We have Kubebuilder, which is an open source upstream effort… So we are starting to look at ways of building out frameworks for us to start developing controllers and operators, but again, it’s a lifetime of iterating and working on it, and we’re just not there yet, I don’t think.

Joseph Jacks: Yeah, that is a really good question. I think there’s lots of ways to answer that question, but one potential way of looking at it is 2018 is a really huge year for commercial open source. I guess open source software overall is about 20-25 years old, and just in the last five(ish) years we’ve seen a huge amount of growth in this kind of emerging category of companies that you could classify as commercial open source software companies. Your mention of the index that we’ve been managing/maintaining for several years kind of indicates that obviously there’s quite a lot of activity in companies that have formed, that have reached huge levels of scale.

100 million in revenue or 25 million revenue/quarter was chosen as a metric just based on that revenue number being pretty relevant to companies that can kind of go public, or have large outcomes.

So “Why now?” I think is really a function of 2018. I’m actually just at the GitHub Universe event right now, so GitHub’s a good one to mention, but… So far, in aggregate, we’ve had over 30 billion dollars in either IPOs, private equity events, or mergers and acquisitions of commercial open source software companies. Most of those are companies that have been in existence for several years, at least 5, 6 or 7 years, in some cases 10 years… So it takes a sort of similar or same amount of time roughly for commercial open source software companies. I believe it’s possible to do this in a shorter period, but roughly between 8 to 10 years to become large, sustainable, public, IPO-able companies…

[08:11] And I think 2018 has been sort of this tipping point year for public markets, and then seeing lots of these large outcomes occur, where most of these companies are venture-funded. But as we’ve talked about in the opening, we haven’t really seen a focused firm I think mostly because there just hasn’t been the synthesis of appreciating these companies are fundamentally different as compared to proprietary, closed source enterprise software companies or just software companies in general.

We very strongly believe that commercial open source software companies are fundamentally different functionally in almost every way, as compared to proprietary closed source software companies. And that’s kind of another motivator for starting OSS Capital - the founders need to be served differently, the support structure is very different along lots of different dimensions, and the companies kind of just grow and evolve and go to market and build products for businesses also quite differently.

But just to answer your question, I think 2018 has been a really remarkable year for large open source outcomes, obviously GitHub being bought by Microsoft, MuleSoft having their second exit to SalesForce after IPO-ing last year, Magento getting acquired by Adobe, SUSE getting acquired by a private equity firm, Elastic’s IPO, CoreOS getting acquired by Red Hat, Alfresco getting acquired by a private equity firm… We think there’s quite a lot of IPO dominoes (if you will) that are gonna fall over the next several months even. We think there’s probably a few more between now and the end of the year… So that’s one of the other factors of 2018.

I guess maybe one last comment - in 2018 almost on a monthly basis, like January, February, March, April, May, every single month we’ve actually seen a large commercial open source outcome… Every single month. So it’s been pretty amazing so far, as a year.

Carmen Andoh: [20:06] Yeah, so the one thing that I always kind of latch onto is this inclusion question. And I think about inclusivity because I have been a person who has maybe internalized that maybe I don’t belong. It’s really more relevant to people who maybe don’t feel like they belong, whether that is true or only perceived… But one of the things I noticed in the Go – so I was asked by a part of the Go team, because I’m one of the Working Group through Golang… “Hey, can you take a look at the survey and see if there’s anything we should modify for the upcoming year?” So I looked at it, and everything looked okay. I didn’t at the time see anything that I would change about it.
But when the survey results came out, I kind of took a closer look at the inclusivity question and I was like “Oh, these are interestingly-worded things… Maybe we could discuss and see if we could reword some of the questions.”

I’ll give you a little back-story… My son was diagnosed with autism when he was two, and I’ve talked about this at my GothamGo keynote, but one of the things that I did to try to understand that was to get a masters in it… And my masters did a lot of work with survey design and analysis. So one of the things that’s in surveys is designer bias, whether we like it or not. The perspective of inclusivity in a survey design is by definition you have to decide what is default and what is not default, and why you would identify as an under-represented group, because that was the nature of the question, right?
So last year’s survey, or the results of 2016, there were a lot of write-ins. One of the things that surprised me was “objection to the question” as a write-in… And I would love to speak to those 150 individuals and say “Well, what is it that you object to? Do you object to the way that the question was worded? Do you object to the fact that the question was asked at all? Do you object to the fact that maybe there isn’t a category, or maybe the way that the multiple-choice was put in?” I would love to speak at length further with them.

The other thing that I thought was interesting was that about 33% to 37% in both years did not answer the question at all. They just skipped it. I guess it wasn’t a required thing that you had to choose, right? So I would also love to know - well, that’s interesting… One in three gophers didn’t wanna answer this question; now, what kind of assumptions can I make there? Can I make assumptions that they just didn’t think that this was something worthwhile to answer, and why that is? One of the things is “I don’t consider that I belong to any under-represented group”, whatever that might be; I’m not gonna make any assumptions.

So anyway… Those are the two things in the survey in terms of inclusion that I talk about. Because we always see about diversity and inclusion, and I always focus on inclusion, and it’s something that’s sometimes hard to scale because belonging and inclusion can be intensely personal. I’ve had talks with people who maybe don’t feel included because they feel that they’re of older age; I’ve had conversations about that with some gophers.

Some also are a part of a religious group that cannot drink alcohol or coffee, and so some of our conferences and meetups and events - they feel a little bit left out or there’s no alternate for them. So it’s things like that that we maybe could include, but then again, there’s this whole idea of “Okay, well then do we have to – maybe just allow that write-in, but…”

[23:58] Anyway, so I went and I kind of looked at other tech communities. There was the Stack Overflow User Developer Survey. That, of course, gets far more submissions, but one of the things they did that kind of created a stir on Twitter was that they asked the gender question and they didn’t have any options for trans, non-binary or other gender minorities… So then the next year they included that, and that helped those that identify with the non-binary gender feel included. I felt like we didn’t have anything like that, and lo and behold, I went and I looked at the Rust survey, because gosh, how many times over time we think of Go, sometimes we think of other languages, and there’s Rust… And for Rust, that was an option to check.

I have some non-binary friends and co-workers, and I kind of said [unintelligible 00:24:49.13] I kind of scrubbed the communities as these and I said, “Which survey design or question do you like better?” and they were like “Well, I don’t feel erased in the second one”, meaning Rust.

These are just things to think about, and I don’t know what the answers are, but I do know that – the thing that I’ve talked about is I remember walking into GopherCon in 2015 to the CoreOS pre-party… Do you remember that, Brian and Erik?

Erik St. Martin: Yeah, and even if you look at Raft - Raft itself kind of explains how the consensus works, but you take an implementation of that, like etcd or Consul or something like that, and there’s so much more that goes into it… Especially when you talk about having a service that can stand the test of time; you need to be able to back-up the data and compact logs and all these things. There’s so much more work than just that; that’s kind of the root of it, but like you said, there’s so much more engineering and problems to solve around it.

[32:03] Moving on from performance, another topic you are very into is fuzzing. For anybody who may not be familiar with what fuzzing is, do you wanna kind of give a rundown on what fuzzing is, and then we can get into why you think everybody should be doing it?

Erik St. Martin: So I didn’t do a lot of development this week. Well, a little bit… So I’ve been in New York City at this Open Hack thing that Microsoft’s been hosting, which is like a cool little hacking challenge conference, and I’ll write up something, a little bit more about that, but it’s been super fun. As part of the thing, we had to deploy a Kubernetes cluster with metrics and stuff, and the Prometheus operator by CoreOS is badass… It’s the first time I’ve used it, because all the Kubernetes clusters I’ve set up and administered was prior to this. But the pattern is really awesome, because it actually uses custom resource definitions, and I kind of hinted at this in our conversation about building abstractions using the operator pattern…

So literally, to get our service monitored by Prometheus in our Grafana graphs sidecar process that scraped this stuff over a custom protocol, use the Go library, which automatically gives you an HTTP listener for Prometheus with a slash metrics endpoints to expose the gauges… So boom, that part’s done. And then in order to get Prometheus to find it, it was just a – because they use CRD’s, they have a custom Kubernetes resource called the Service Monitor, where basically I told it to label to look for my custom service, and it automatically knew how to find all instances of that service and to scrape its metrics. That was it.

[01:00:18.27] That’s so useful, not having to custom-configure Prometheus every time you launch a new app, and then reload the configuration for Prometheus, and stuff… I just thought that was really cool.

Jerod Santo: That’s interesting… I wanna talk about that architecture a little bit, because from the outside looking at it, even if you go to CNCF.io and you look at the platinum members and you see AWS and CoreOS and Google and Docker and all these large corporations, and then you see some of the membership fees that they’re paying, which for the platinum is like 370k a year - there’s like this aura of this as a pay-to-play type of a situation, and it’s so interesting that the structure you all put in place is specifically to fight against that happening… Is that what you’re saying?

Ivan Porto Carrero: Yes, it does… NSX-T. It includes NSX-T, which is VMware’s overlay network; it’s the second generation of it. What this does over any of the other solutions that are out there - because most people will typically go with flannel originally, and then maybe look at something like Calico for the policies. It actually gives every pod a container interface that can be managed outside of just the environment. You can have a network administrator who sets up a bunch of global policies in some other system - the NSX management plan - and that will then translate into rules for Kubernetes, for example.

There is more stuff to it, because NSX-T is quite an extensive piece of work. So it’s pretty optimized in how it deals with sending traffic and doing the routing rules and so on, but those are implementation details of NSX-T itself. What is unique I think is that it has a centralized management plan for all types of container interfaces, and that is where Kubernetes also takes advantage of it. The NSX team has an integration for Kubernetes that also works with some of the other Kubernetes distributions, so yes, it’s a very important piece of it, the security aspects that NSX-T brings to bear.

Erik St. Martin: Being you have a networking background, what about something networking related? I think there’s a lot of stuff going on in the cloud networking space now; you know, CNI, and you’ve got things like Flannel and Calico and all of these things that create these mesh networks and things. Understanding that a little bit might be interesting.