Tim Banks: What do you think powered the first Bitcoin miners for old servers? People’s old desktops that [unintelligible 00:43:01.17] stuff like that. The notion of buying it means that when I’m done with it, I can sell it and get some of that money back. Is there a secondary market for it? Absolutely. There’s already a burgeoning secondary market for buying older or used GPUs, and servers, and stuff like that. “Perfect. Let me make some money off of this.” That’s going to be far more economical than giving the money to AWS, because I’m never going to see that again. You know what I’m saying? This is the difference between investing – this is the difference between “I’m going to do this thing as an investment and I’m going to get some money back on it”, versus just giving it away to somebody else. But this is not a day one decision. Day one decision is growth, and it makes perfect sense to be in whatever cloud environments can allow you to grow, right?

Ganeshkumar Ashokavardhanan: Yeah, definitely. So for me it was quite a new paradigm for thinking about software as well, and even thinking about how code is run, and infrastructure in the cloud - it’s very different from what you do in college. Most of my courses were focused on running code on your machine, and when I was doing some training of ML models and so on it was using cloud computing and running on different machines, but not at the scale at which Azure operates, or a lot of production-grade software operates. So even a lot of the analogies that Brendan used and comparisons he did with previous technologies - I did not have that as a reference. So Kubernetes for me was also the first time in which I was learning about many of these concepts, around deployments and resilience, and so on.

So there were two parts, I guess. One part was even before Kubernetes, in my first internship at Microsoft, I got exposure to how rollouts are done, and so on, without using Kubernetes. And then, in my second internship, I had exposure to AKS and Kubernetes when I was interning in the team.

So for me, learning about Kubernetes I think has definitely been something where I explicitly spent time learning from docs, and actually even Brendan’s videos on YouTube, on Kubernetes, I watched them. For me it was also cool to actually meet the person behind the videos, too. So that was nice.

And I think also working on different projects within AKS has helped me learn about various Kubernetes concepts. And one of the things that I worked on is Kubernetes versioning for AKS, which is supporting new versions of Kubernetes on AKS. That process, especially for minor versions, involves many API deprecations, and flag changes, like Kubelet flag changes and so on that we would need to handle… So as part of debugging Kubernetes code there, looking at all these different components within AKS and so on, actually helped me become better at getting a framework of Kubernetes, and models of Kubernetes in my mind. And I think it’s still an ongoing process; there’s so much in this space, so it’s been a good journey.

Brian Leroux: No, no, no. I was there from the very beginning, for sure. That was created by a dude named Austin, and he’s got a company around that. I work on one called OpenJS Architect, which would be I guess technically a competitor, but I don’t really believe in that concept in open source. They take a different approach than we do, and they do good stuff.

But yeah, I always end up – this is an affliction of mine… I get interested in something probably prematurely, and then I go all-in. When mobile was happening, I was really excited about that. That was a big deal. Steve Jobs was like “Hey, you’re gonna build websites and they’re gonna run on your phone.” That blew my mind, completely. The idea of even seeing maps on your phone was novel at that point. And getting directions was a pretty weird idea. So we’ve absolutely come a long way in ten years, and in a sense I actually predicated the whole SPA thing.

And then yeah, when I saw serverless happen, I had been doing a bunch of cloud stuff for PhoneGap actually, up to that point, and I saw Lambda get demo-ed with API Gateway in 2015, and it’s like something snapped. I was like “Yeah, that’s how it’s gonna work. We’re not gonna build balanced containers. That’s a ridiculous proposition. Why would you do that? Let somebody else do that and outsource it to them.”

So for me, that was just immediately obvious in 2015. Here we are in 2021 and that’s still news to some people, but that’s kind of my curse and my blessing. I’ll see these things as they’re happening and get excited about them. I kind of [unintelligible 00:07:16.26] to it maybe a little earlier than I should…

Erik St. Martin: It’s always a juggle, right? Because on one hand you have to have your IP, you have products and you want to evolve those, and you wanna kind of keep stuff to yourself, so that you have these value-adds over competitors. From a business perspective it’s totally understandable, but I think on the other side of it, all of the cloud giants and things like that see the value of working together to evolve the space.

[32:00] From my perspective - and don’t know if this is Microsoft’s view, but this is definitely mine… Competing for customers is kind of a losing game. I don’t think if we offered Netflix free services forever that we could ever get them to convert over, right? So the idea of trying to compete directly and steal customers - I think that you’re putting in a lot more effort for little reward. Now, building abstractions, Virtual Kubelet, building things like Helm and Brigade and things like that that help make the cloud, and things like Kubernetes and containers more approachable to a broader audience - now you’re creating more customers for everybody, right? Because there’s more people that have not adopted the cloud than there are people there, and it makes far more sense for us to keep helping make it more approachable than it does to sit here and try to compete feature for feature, or hoard our knowledge in projects and stuff like that.

Chris Benson: So this is a particularly interesting episode, I would argue, for me, and probably for quite a few of our listeners that listen regularly… Because we’re talking about something - and Erik, we have the privilege of you as the person who’s created this, but we also have Daniel, whom I’ve been working closely with, and our listeners have been listening to… And hearing Daniel’s passion and him building his own business on your platform… And we talked to lots of different companies… And so it definitely has intrigued me in a way that not every different company owner, if you would, has. I’m kind of curious - I’m thinking about it from a slightly different perspective from Daniel, but you’ve really got me wondering how to make this happen. I work at a big company, as you know, we have big investments in kind of the big cloud providers, as all large companies do… What are good strategies for companies to say, “Okay, we have so much in these other big names and stuff that are out there…” How do we start using Modal effectively? What are the kinds of things you’ve seen your larger customers do in terms of migration over, or things that you might recommend, that enable something of a migration to be more seamless, less painful…

[32:18] Because normally, when you think of large company migrations, they are almost always fraught with pain, and misery, and challenges for the IT crews. So how do people get to this thing that we’re hearing about today, and mitigate all of those problems?

Daniel Whitenack: Yeah… I think there are - with everything that we’ve talked about before, and I guess everything related to this - always trade-offs, right? I was talking to a friend of mine who is at a startup, and part of their startup IP and really the secret sauce of what they’re doing is in their machine learning model, right? But then if you take that model and then you push it out to someone’s client device and run it in their browser, of course there’s always the opportunity for – you’re releasing that model out into the wild and people can maybe just take it and look at View Source in the browser and figure out how to get your model and utilize it, and all of that…

[28:25] I know that he was concerned about those risks, but it’s probably – I don’t know, in my mind maybe the benefits outweigh the costs, because in the same way, there have been a lot of papers that have shown even for doing inferencing in the cloud, if you’re exposing some service that does inferencing for image recognition or something like that, it only takes a certain number of requests to that API to be able to mock or spoof that machine learning model, and actually create a duplicate of it.

So I guess there will always be those trade-offs, but there is kind of this transfer of the model to the client’s device, which it probably has some trade-offs there, but also these models aren’t super-small, and if you wanna update them over time, maybe there are some storage, or battery, or other sorts of issues going on there. I’ll be interested to see how people deal with those tradeoffs and what ends up becoming the driving force there.

Dan Kohn: [01:01:43.10] Yeah. [laughs] That’s by a leader in our community, I might add. But I’ll leave him anonymous for now. I think the simplest solution for it is, especially if you’re a startup, if you’re a small organization, if you’re coming in from a greenfield perspective, it’s very likely that you’re gonna use a vendor for these things, and that vendor might be a hosted cloud provider, it might be an enterprise software company that helps you run things on your own bare metal… And one of the great roles that those vendors take is that they pick the projects and the offerings that they think are ideal for their customers, and it’s very likely, as you get started in this space, that whatever the vendor puts together for you - maybe your favorite cloud provider - is gonna be perfectly fine… That to get up to speed on it, to get your logging working, and monitoring, and these other kinds of things, that you can dive in and work with almost any of them.

And then, if you just find that your monitoring solution - maybe it’s a proprietary monitoring solution that you’re paying too much money for, or maybe it’s the vendor’s proprietary monitoring solution on their cloud, and you don’t feel like it gives you enough flexibility, or doesn’t let you hook into all of your services the way you’d like it to, then it’s a relatively natural step to look at that CNCF trail map and say “Oh, I guess I could be looking at Prometheus.” And maybe your vendor is already just offering a hosted version of Prometheus, and you’re either using it or it’s easy to switch to it. But I really want to avoid this implication that you have to be familiar with all 570 options.

There is this concept by actually a former professor of mine at Swarthmore named Barry Schwartz, “the tyranny of choice”, that having more options does not necessarily make you happier, and it certainly makes it just overwhelming to try and make any kinds of decisions.

So the CNCF projects in general somewhat represent a happy path, where we can confidently say, hey, if you choose our graduated incubating projects, we know they all work, we know that there’s real end-users adopting them, we know that there’s vendors out there who are eager to support them, your issues are gonna get responded to; you know, maybe not that your pull requests will get accepted - we can’t go that far - but that it’s a pretty safe bet to engage and get invested in those communities.

And then that’s the other way to look at it, which is, okay, if you wanna just go entirely open source – one of my favorite examples here is the company Bloomberg here in New York, who they just don’t like working with vendors. They wanna have all their expertise internally, they download 100% open source, and they make it work for themselves… So CNCF represents a great spotlight and indicator for them of a set of projects that they can have confidence in the communities behind them.

Jason McGee: Yeah, so there’s kind of three key features, if you will, or capabilities that Istio tries to provide… And remember, it’s always in the context of kind of multiple services talking to each other.

The first is traffic management - providing a way to programmably control how different services talk to each other. I’ll give you a really simple example - how do you roll out a new version of an application? When you roll out a new version of an application, you might want to test that new version with 5% of your traffic, to see if everything’s working okay… Or you might want to have a subset of your users try the new version. Maybe everyone on the East Coast is gonna try the new version, while everyone else in the world stays on the original version.

You need the ability to kind of control the roll-out, so one of the goals of Istio is to give you that kind of programmability of routing, so that you can decide who calls who, what versions get routed to, how the traffic gets split… And that’s something that everyone has to do on some levels, but everyone has kind of solved in their own ways. Kubernetes doesn’t solve that problem directly… Lots of people have built custom solutions to do that. Istio tries to kind of bake that into the service mesh. So that was one key - advanced traffic management for DevOps, for resiliency testing and things like that.

The second key thing Istio is trying to do is security… And this is actually turning out to be a really interesting space with Istio; we could probably dive deeper here, but historically, the way security was often implemented was down at the network. You would define firewalls and network rules and segment your networks to control who could talk to who, and which services were visible to each other.

That model doesn’t work very well in the era of public cloud and in the era of very distributed architectures, where maybe part of your app is running on IBM Cloud, and a part is running in a data center, and parts are running on Amazon… You don’t have one network that spans all of that, so with Istio what we tried to do is say “Can we move all of that security up closer to the application? ..so it’s not defined in terms of the network, but is defined in terms of the application itself”, and automatically set up secure connections between services, and allow you to specify policies about who can talk to who, but independent of the network. That’s turning out to be incredibly powerful for a lot of the early users of Istio, because it gives them a different way to secure their applications.

[10:34] Then the final thing that Istio was after was observability or telemetry - how do you actually see what’s going on? One of my favorite case studies in the microservices space was some material I read around the Gilt Groupe (online shopping) site. They’re not a customer of mine, they’re not an IBM Cloud customer, they’re just people who have talked about their own journey to transform to a microservices architecture… And the kind of tidbit that always stood out to me with them is, you know, their original architecture was a very classical kind of web and database application.

You know, maybe there were ten instances running in a cluster to handle the load, a single monolithic app… And over a period of years, they refactored that into a bunch of microservices, and when they got done, they had like – I forget the exact number… It was like 300 or 400 microservices; each of those microservices, you have to imagine, there’s probably at least 3 instances running, so you go from running like 10 things to running 1,200 things, right? So you have this much more complex environment to manage… How the heck do you actually see what’s going on? If there’s a performance problem, where does it exist, where are the bottlenecks?

You need a totally different way to kind of observe behavior in a much more complex network of services like that… So part of the goal of Istio was to provide ways to automatically gather all of that data about who’s talking to who and how it’s performing, and tracing, and log collection, and everything, in a transparent way, and to make it so every developer didn’t have to figure out how to solve that problem themselves.

Adam Stacoviak: Yeah. Well, because it was – to give some context to those who might be catching up (we’ll link it up in the show notes, of course), I asked Kurt how his life is, and he recently went through a divorce that sort of changed his entire life. And he was happy to share that. And so in the moment, I was like – in my brain, I’m thinking “How deep should we go here? Because this is a deviation from the topic, obviously…” We’d been talking about reinventing the cloud, this new public cloud kind of idea. And now here we are talking about divorce, and I don’t want to get melodramatic, and get sad, but I also want to examine this… Because I feel like part of what we do here at Changelog is just – it’s peel back the layers of the realness of what is what. Why did you not go and sip Mai Tais on the beach? Why did you go back to work? Why do you care so deeply about interviewing, or sitting down with every person who comes in? Why are you so passionate about application developers? And it’s beyond the technology. It’s beyond even your drive as a human being. It’s like “What makes you tick?” That shared decision between you and your wife to say “You know what? Let’s do this.” And it’s not just “You go do the work.” It’s us making the choice of allowing one part of the whole to go and do one more - maybe more - adventures; one big adventure, one more big adventure. And I think that’s what it’s all about, really. You can’t just simply say, “Tell me why you think Kubernetes sucks. Let’s talk about that for 40 minutes.” Or it doesn’t suck. Or let’s just say, why do you think it sucks for application developers, and it doesn’t suck for DevOps. I’ve got a great friend of mine who is a DevOps, I would just say borderline genius… He loves Kubernetes.

David Hernandez: It’s basically everything local. The Go binary, it serves also the static files in local. For the data store we’re using Firestore; it’s the database from Google Cloud Services. They ship an emulator, so in local, for testing, we can spin up the emulator, run whatever test, and tear the emulator down. It’s quite a quick process right now; it’s less than a couple of minutes, or less than a minute, I guess. It may grow with time and it becomes a problem, but right now it’s not containerized; we just run it in local. Mat runs it on his machine and I run it on mine.

[31:55] At some point, at the beginning of the project, we put a continuous integration thing also in G Cloud, but we didn’t use it at all. We were just doing it more in local than we tried to do it in continuous integration. Probably with two people you don’t have the problem. If the thing grows more, we’re probably gonna be back to that situation that we need something in continuous integration… But right now it’s just not necessary. Today was the first time that I was deploying in production, and Mat was deploying also in production, and I got an error like “Oh, someone else is deploying.” Who else could it be…? [laughter]

Chris Benson: It does. That sounds fascinating. And as you know from our pre-chat, I knew a little bit about Omniverse before coming into the conversation, but I know that there is a lot of confusion about how this fits in with all the other – you know, we were joking in the beginning about the various verses that people are hearing. There’s a lot of lingo out there. And as recently as yesterday, a friend of mine named Kevin texted me and I haven’t replied to him yet, but I will have by the time this is aired… He texted me saying, “I don’t understand this verse thing, and I know that you’re involved in this. Can you explain it?” And I think Kevin represents a lot of people in that way. And so could you – we’ve heard multiverse, we’ve heard Metaverse, we’ve now definitely heard Omniverse… Can you give us some context for how this whole industry fits together, so that as we dive into Omniverse, back into it in just a moment, we kind of have a sense of where it fits within – and some of the other companies. We know you’re with NVIDIA and you’re doing this great work… But we’ve heard things from other big companies, the usual array of social media and cloud companies. So can you kind of set the stage for us a bit on it?

Steve Tuck: Yeah, running in the Oxide rack. And one important aspect of any kind of cloud infrastructure is the ability for you to move workloads uninterrupted. So you need to be able to tolerate live migrating things around. So we’re watching this demo, and they’re small-talking, and just like giddy to give the final reveal… And at the end of this Minecraft banter, they had been demonstrating our live migration. They’d been migrating stuff all over the place, with no blips in gameplay.

Kendall Miller: It’s a rye whiskey. We also make a gin called Force Push Gin, focused on the tech world. Anyways, it’s a guy that I hired eight years ago, because I met him at a meetup, I needed a person who did exactly what he did… We worked together great for a little while, and then years later we ended up starting a business together. So networking has network effects. So when I’m out meeting people and I’m genuinely interested in getting to know them - which I am, because I find people fascinating - I find, “Hey, you need a job and I know somebody who’s hiring for you. And if I refer you to that person”, and I don’t take… A lot of people have asked me, because I refer a lot of people for jobs. So a lot of people have asked me, “Why don’t you take a recruiter fee?” I would probably make a lot more money if I did. But as soon as I monetize the network directly, the network loses all of its value.

So because I’ve referred a person over here, that director of engineering now owes me a favor. And so now I take a job at this new company, and I think that this product might be relevant to that director of engineering. And I don’t mean owes me like – you know, it’s not quid pro quo. But when I call him and say “Hey, would you take a look at this?”, he’s going to say yes, because I helped him hire the last five people he’s hired.

Actually – I mean, a good friend of mine in San Francisco, who got fired from a very big company… I don’t actually think he got fired. I think he left. I don’t know. I don’t know what happened. Very large company. And he’s one of these guys that’s too big of a big cheese. Like VP at very large organization. And nobody in my network is the CEO of Microsoft, right? Like, I know a lot of people, but I don’t know those people. So I only know one very large cloud company’s execs, and I wrote to them and said “Hey, this person came available on the market. He’s more expensive than almost any network that I know, so you’re the only place I can refer them to.” Well, they just hired him, I found out yesterday.

[00:24:08.04] So now there’s a C-level executive at one of the largest cloud companies in the world that doesn’t – I’m not saying owes me one, but will definitely pick up the phone when I call. And so those kinds of things just add up, and so what ends up happening is somebody pings me, has a question about “Hey, I’m building XYZ. Here’s why it works. Here’s why it doesn’t. Can I get some feedback?” I look at it… 70% to 80% of the time I just s**t all over their ideas. Sorry, I’m not supposed to curse here, because you’re going to bleep them out. I say bad things all about – you know, I shut them down. I tell them it’s a terrible product.

Danielle Lancashire: I’ve worked on a big mix of both our cloud, and also the bringing Web Assembly to Kubernetes stuff through SpinKube . A lot of that is sort of mine and people I work with domain, which is a mix of – you know, for Kubernetes things we built out a Containerd shim. So Containerd has pluggable runtime environments; like execution things called shims. And there’s a project called Runwasi, that exists to let you write shims that run Web Assembly code. And so we have one of those for running Spin applications, which is Fermyon’s WebAssembly - I guess framework is kind of the right word… It’s like kind of a runtime, but also kind of a framework, and the community hasn’t really standardized on a language for that yet. And then sort of an [unintelligible 00:17:53.25]

Matthew Boyle: So I think [unintelligible 00:55:00.06] good or bad, but I guess just kind off a brief story of how we’ve been using it at CloudFlare, which has been interesting… So firstly, I did mention that we’ve been using Protobuf for our schemas. The support for Protobuf and gRPC in Go is excellent. It’s first-class. So that was a good fit and a good choice, and I would make that choice again. So for schema management, Protobuf’s definitely worth looking at, especially if you are a predominantly Go place.
Something else we did is we created a Go library that we call Message Bus. Very creative. And effectively, what we do in this is we use – it used to be run by Shopify; it’s a Go library called Sarama. Sarama is a Go library that was created by Shopify, that basically allows you to do pretty much everything with Kafka. And I actually credit that library with an awful lot of the adoption of Kafka, both at Cloudflare and elsewhere where Go was involved, because it enabled you to basically do everything that the Java libraries were doing.

One really hard thing about Kafka as we talked about is configuring it can be hard. So one good choice that we made as a company, I think, is we made it an opinionated library, that kind of set up a very good set of default settings and constraints for how we think you should interact with Kafka at CloudFlare, and made it as easy as possible for you to do it. It’s got a bunch of power user settings, if you will, where you can override what we deem to be the best settings, but that was a pretty good choice, I think. And we added a bunch of Prometheus metrics within that library as well, so it means that everybody who pulls in our library gets this dashboard for free of how their Kafka service is performing… Which was very, very helpful, and again, is another thing I’d recommend doing. It’s not Go-specific, you can do it in any language, but we were able to do it with Sarama.

Slight tangent, but Sarama actually got picked up by IBM. So IBM is now responsible for the maintenance of Sarama, because it turns out that Shopify aren’t using it too much anymore. So IBM have taken over stewardship of it. So that was a really cool thing to do… I haven’t checked in on the project in a while, to see how it’s progressing, but it was excellent that they put their hand up to carry on stewardship.

And then the final thing that we have been using Go in Kafka for is we built this thing called – we call them connectors, and they’re built on… There’s a framework called Kafka Connectors, which effectively allows you to plug some code into your database, [unintelligible 00:57:09.24] into Kafka, and then it just like moves the data between the two. So when people are trying to take things out of a database and push it to Kafka, Connectors are a pretty common way to do that.

We built our own framework that we also call Connectors; it’s all written in Go, and effectively with a very small configuration file you write in YAML, we allow you to specify a reader, some transformations to apply, and then a writer. And so what this means is teams can deploy very simple code that reads from a database, applies some transformation to a Protobuf format, and writes it to a Kafka topic. They can do it without actually writing any code; they just create some environment variables and deploy it. And same thing - we’ve got Prometheus metrics, you get a dashboard for free, and you get some alerts around it for free, and stuff.

[00:57:49.17] So all of these things have really helped with Kafka adoption. And I think if you’ve got the resource to deploy Kafka at your company, I would really consider having a team like mine, a platform team that provides tools and services that makes it easy for other teams to do the right thing, and to teach them, too. I think a huge part of our team’s job is just teaching as well, and just making sure people are following the right patterns when using some of these things. And it can help overcome some of these barriers to entry, but obviously, it’s a large cost investment.

One of the reasons Cloudflare picked Go in the first place, and we continue to use it, is it just scales so well. We’ve had a couple of issues with Kafka consumers not being able to keep up with the amount of messages that are being passed through, but after some small tweaks that you would have to make in any language, we’ve been very easily able to scale a bunch of our services to tens of thousands of messages being read a second, without too much heartache. We haven’t had to do anything clever, we haven’t had to write any sort of crazy code to do so. It’s just kind of standard optimizations that – you know, a linter will probably help you with most of them, if I’m honest. So Go has been fantastic for that… And even people who join CloudFlare who haven’t got experience with Kafka and haven’t got experience with Go, we’re usually able to get them productive and writing decent Go in a week or two just because 1) how easy Go is to learn, 2) how easy it is to read, which I think is really important and often overlooked with Go. Being able to read it… I can pick up pretty much any Go service and I can follow it through, and I can roughly figure out what it’s trying to do. I can’t promise you the same thing for Java or PHP, where there’s lots of like auto-wiring and magic, and you have to understand the framework a little bit.

So that’s been really, really powerful and useful for us in terms of adoption, too. And generally, the performance of the Go app. We are a cloud, so we pay a lot of attention to resource utilization, and containerized Go services… I think they are so tiny in comparison to some of the other things that we have to run. And even quite complicated applications that we have running, that are processing a lot of data. Their footprint is tiny. So if we were to do this all again, maybe [unintelligible 00:59:44.25] some other languages, and to be completely clear, especially in the context of the conversation we’ve been having, there has been more and more adoption of Rust at Cloudflare. More teams are definitely starting to dip their toe in and figure out if that’s a good fit for them. And TypeScript, too. There has been a lot of TypeScript usage, especially in Cloudflare Workers, because it’s natively supported and it’s fantastic. But Go isn’t going anywhere. We see new Go services deployed every day, because it just does what we need to do incredibly well.

I think one of the huge drawbacks of picking go and taking this approach that we had - and it’s something that we’re sort of still reevaluating - is, as you probably infer from what I’ve said, we’ve invested a lot of time in tooling for teams that write Go. If you don’t like Go, we haven’t got actually a whole bunch to help you right now. We haven’t rolled the same libraries in Python, we haven’t rolled them in Rust. So we’re actually kind of making it harder for teams to blaze a trail and potentially do what’s right for them, because using Go is easy for them… Which is kind of by design. We want them to stick with Go until maybe it doesn’t make sense for them anymore, because we’ve got all this great tooling that’s got production-battled experience and it works.

But one thing we’d love to support in the future is the same sort of patterns, ideas for other languages. And so we’ve been exploring some interesting things like could we put gRPC in front of Kafka, and therefore generate bindings for further languages, and therefore we could benefit from the same tooling which was set just behind our gRPC server, but the teams who interact with Kafka need to know even less about Kafka, because we’ll handle the hard configuration for them? And then we can support these other languages, too. And the thing that keeps causing me to pause is exactly what Kris was talking about, is if we do this, we’re going to remove the need for teams to understand Kafka at all, if we do it well. And that sounds like a great thing, and it maybe is in the short term, but I just feel like it will bite us a lot in the long term if we don’t – this fundamental piece of infrastructure, if it becomes one team, like mine, who knows how everything’s configured and connected to it, and another team’s kind of passing through it without ever really truly understanding it, I don’t know if that’s actually optimal in the long term. So we’re still trying to battle that and figure that out, but… As of right now, it’s the only path I can see to scale all this great tooling [unintelligible 01:01:50.17] to a way that we can support all these other languages that are starting to appear in the Cloudflare ecosystem.

Bryan Cantrill: First of all, what kid is thinking about breweries? What’s going on? [laughter] Anyway… So now a defunct computer company. So I spent 14 years at Sun; Sun definitely did believe in that hardware/software co-design, for sure. [unintelligible 00:26:42.12] After Sun was invaded by the Nazis, I fled. So Sun was purchased by Oracle in 2010. I went to a cloud computing company, joined, and spent another nine years there. So it was kind of interesting to be on both sides of this. And Joyent was acquired by Samsung in 2016, and I and a colleague of mine that I had come to know at Joyent, Steve Tuck - Steve and I decided, like, we knew we wanted to start a company together… And we were kicking around some very bad ideas for things that we’d go do. And as we were kind of kicking around bad ideas, I was thinking, “Well, you know, maybe it’s time to start talking, start lighting up some venture capitalists…” I wanted to go back to – I had known VCs over the years, but needed to go kind of remind myself of them and maybe get some lunches or whatever. And I went back to the mail from a VC - actually, a pretty famous VC, and he and I would get lunch together with some regularity… And the last email that he had sent me was, “Hey, I really enjoyed lunch. Just wanted to remind you that I will fund literally anything you put in front of me.”

Mikeal Rogers: [01:16:04.26] First of all, coming back to that, differentiating between problems that are current technical limitations versus things that we actually have an incentive to solve… Everyone has an incentive to reduce the energy consumption. It is only a cost center. And if you look at what’s happening broadly in blockchains, we have figured out how to get the energy cost down. Those algorithms exist, they’re implemented in several chains… Ethereum is planning on moving to one of them, so Ethereum will be on one of these that uses significantly less energy…

It is totally unclear to me how and when Bitcoin would ever do something like this… So Bitcoin may be just always using this much energy, and maybe that’s the downfall of the protocol. Maybe that’s why people stop using it at some point, if they can’t make that migration… But I think if you’re looking at the ecosystem overall and the technology overall, we’re already moving past that particular problem, and we’re getting the energy consumption much closer to just what you use in cloud computing and mobile and stuff like that.

I don’t know when your first computer was, but mine had a power supply the size of about 20 iPhones, and it consumed about as much power I think as 20 iPhones maybe, for a fraction of the compute. That went down, and this is all just kind of coming down over time.

John Myers: Yeah, loosely based on reality. I didn’t meet Will Smith or anything, so that was a bummer, but… It was over in Maryland. I’m from Philly originally. I was like “It’s close to home. I can get out there and just experience something new.” When I got there, I got immersed into the intelligence community, and working at NSA, and I still wore the Air Force uniform, but I was kind of in an offshoot program there; I got to work on really cool stuff, working on low-level operating system engineering, building exploits, stuff like that.

Then I kind of pivoted into doing big data analysis, which was a kind of up-and-coming field at that time. Then I left there, did one more stint with the Air Force, doing a different set of things out of Las Vegas, and at that point I was at a critical point of if I wanted to stay in full-time or do something else. At that point I was so hooked on building… I just wanted to build, and engineer, and building engineering teams, that I decided to leave active duty; I joined the reserves, and then when I got out, I did the complete opposite thing you can do when you’re part of a 300,000-person organization, and I have launched a startup in cyber-security with three other people.

[08:21] Then we did an enterprise security startup, and we did that for three years (it was called Efflux Systems), and then we were acquired by a company called NetScout, that kind of is one of the leaders in network performance monitoring, and wanted to utilize some of what we had in cyber-security for some of their upcoming products.

When I got there, I was a principal architect, and among a lot of projects I worked on, I worked a lot on a lot of their cloud infrastructure. They build capabilities that also help enterprises and service providers detect and stop DDOS attacks. A lot of those devices collect a ton of telemetry from the customer environment and send it up to a cloud repository where it’s securely kept… And we wanted to look at how we can do analysis of that data. And you start [unintelligible 00:09:12.15] through that data and you start to realize there’s a lot of sensitive information in this data, and we should probably pre-process it, so we can kind of work on it safely. I think I spent a lot of time doing that; probably more time than I wanted. That was kind of like this pivotal transition point where I got into doing engineering to enable me to do engineering.

At the same time, back of my head it’s kind of like one of those Shark Tank pitches; I’m like “Surely, I’m not the only person, but this problem - what can we do about it?” with some other close colleagues of mine, as we started talking about these stories and we all kind of shared very similar, but different pain points; we kind of orbited around this idea of “What if we could just make data anonymization, and being able to make data safe to use, kind of just like a general-purpose thing that engineers everywhere can use?” and yadda-yadda-yadda. We launched Gretel.

So we launched Gretel in the fall of 2019, with myself and our CEO and my other co-founder, Alex Watson, who has a very heavy machine learning and data science background, and was previously the general manager at AWS Macie, which is another product that is very successful at AWS, for detecting sensitive information in S3 buckets.

So he has a whole different slew of stories - some will call them nightmares - about data anonymization. And yeah, we’ve been doing it ever since. And really, our mission is to make data anonymization and creating safe data generally available to engineers everywhere, not just the resourced organizations like the Facebooks and the Googles, who have massive resources to experiment with all the techniques to do it.

Johnny Boursiquot: I’m interested in understanding how we surface the right cut of information to the right people. We’ve been talking about – if we’re able to solve the first problem, which is collect all the things, and allow the customer to come get what they need, when they need it… If we use that as the baseline; if we now turn inward and say “Well, I have some internal customers. I’m an operator of some container orchestration tooling”, whether it’s hosted by a cloud provider, or whether you wanna roll on your own data center, whatever you wanna do.

Now, you know you’re collecting a lot of information, and you make it available in some sort of a data lake, or pool of data, whatever it is; come get it, for internal folks. Is it incumbent upon you or your team or your department to then teach people how to mine insight out of this pool of data? Is it the job of the people collecting it to tell you how to do that, or do you hand it over and say “You do what you will?”

Natalie Pistunovich: [01:03:52.14] Then… It has a longer preparation. So diversity needs to be improved on many different fronts. Some fronts have improved, some fronts have improved less… It’s hard to say there is any front or any type of diversity that is fixed, and the problem is no longer there. And I’m putting a lot of work into gender diversity, Ronna is putting a lot of work into general diversity… But for me, gender diversity is not the main goal. I don’t have a main goal when I’m working on diversity.

One goal that is very close to my heart is bringing people from different countries, like we spoke today. And I prepared some numbers. GopherCon Europe speakers who are born, raised, and live in an African country in 2018 was 10%. It was supposed to be higher, but too many people had their visa either rejected, or just never responded to.

So they applied all the documents, we prepared everything, we helped the speaker gather all the documents they need, and they just did not hear back. We ended up with having 10%, so one out of the ten speakers is actually coming from an African country. And this still was a way bigger effort than reaching 50/50 speakers of men and not men. That was GopherCon 2018.

In 2019 it was slightly better. Two out of the twelve came from an African country. In 2020, the current line-up is 4 out of the 28, which is still pretty much the same. Cloud 9 it’s also one of eight, and BSides, which is a small local conference about security that I’m doing, is one of six. I have yet to break from the limit of 20% of the speakers coming from an African country, which is a personal mission of mine… And it’s really hard.

Now, this was all the preparation. Here goes the unpopular opinion… A conference that has a line-up of speakers who are diverse, but still all of them are coming either from Europe or from the U.S, is not the most diverse thing. What I am looking to evaluate the conference whether they are really diverse on my criteria - I’m looking at which countries the speakers come from. If it’s not an easy country, where the visa is not an issue, for example, or giving such an opportunity is a big deal versus a huge deal, this is what makes the difference for me. It’s not a good recommendation to give to conference organizers to focus on this, because it’s basically recommending people to do a lot of extra work.

Working on somebody’s visa requires your legal preparation with probably five different types of documents. On top of that - this is just from Europe. I’m sure the U.S. is harder, at least in these times.

So on top of those types of legal documents, you have to prove all kinds of financial support, you have to prepare things like flights, even though they don’t even have a visa yet; they already need to present tickets. Accommodation, which is maybe slightly easier… Plenty of documents to show that those people will go back, so a rental contract, health insurance, connections, like if you’re married or you have parents, a job, last salaries… So basically you’re spending many hours with each person you want to bring, whether that’s a speaker or an attendee, just making the legal parts of it. Still, I find this incredibly important as an aspect of diversity, and I think this has not been discussed enough.

Jared Watts: Yes, and I’d be happy to take that one, since I’ve been working on Rook for just over three years now. I believe that where Rook really shines is its focus being on an orchestrator for storage. If you think about the roots of the Rook project, when we started it more than three years ago, something that we saw as Kubernetes was still in very early days is that you would ask people that are using Kubernetes “Oh, okay, so what are you doing for persistent storage?” and almost nobody had a good answer to that. That was a very commonly unanswered question, because they were just running [unintelligible 01:54:06.17] workloads in Kubernetes.

So we started seeing the value of, okay, if we can use these primitives and these patterns that are in Kubernetes, and these best practices that are starting to form around how do you manage an application’s lifecycle, how do you maintain reliability of a distributed system - all these things, these problems were being solved, and then being able to build on top of that with “Okay, let’s do the same thing for storage.” Let’s reuse the Kubernetes best practices and patterns to stop relying on external storage, or storage that’s outside of the cluster; maybe it’s in a NAS device, or a SAN, or maybe a cloud provider’s block storage service, or whatever it may be… But being able to bring those into the cluster and orchestrate them to be able to take advantage of the resources that are already in the cluster - available hard drives, or different classes of service, a regular spinning platter disk, or an SSD, or an NVMe, or whatever it may be… But being able to provide storaged applications in a cloud-native type of way; you’re not going with the full stack there. That’s something that we found that got a lot of traction pretty quickly. And it wasn’t too long - it was only a few early minor releases - before we started getting production usage of it… Which is always very surprising, because it was an alpha-level project, and we were very clear about “This isn’t intended to be used in production yet”, but we got production adoption pretty early on right away, which helped drive the maturity of the project as well.

Daniel Whitenack: [27:59] Yeah, I think your experiences are almost on a total different side of the spectrum from mine probably, only because right now I’m working with a non-profit, so… I think this is a good contrast. Obviously, the companies that you’ve worked for do have embedded AI research teams; maybe they invest in some of this NVIDIA hardware. But for me, doing machine learning and AI data-related work with a non-profit, I would basically be left out of every room I was in if I tried to get anyone to buy a 200k NVIDIA box.

I pretty much relied on everything in the cloud when I’ve needed it, and I think that wherever you fall on that spectrum for your AI team, there is a route forward… So it’s great if you’re able to afford that kind of dedicated hardware, and you have that commitment level within your organization, but maybe you’re not at that point yet, or you’re a startup trying to get into this space, or another organization that doesn’t have a huge AI team - you can do very similar things in the cloud.

Every cloud provider has instances that are available with specialized hardware, they also have a lot of services that will allow you to spin up clusters to do distributed computing, like Kubernetes clusters, and there’s frameworks on top of that that we can talk about in a bit… So I think wherever you fall on that spectrum, there is a route forward. But in either case you’re going to have some number of compute nodes on-premise or in the cloud that maybe some have just regular CPU’s, maybe some have GPU’s, and then you’re gonna have some storage that is storing the data that you’re working with for your training datasets, and that sort of thing.

Now let’s say that’s the base - so you’ve figured out whether you’re gonna be in the cloud or on-premise, whether you have dedicated hardware or you’re using the cloud stuff - what do AI people actually run on top of this compute and storage infrastructure?

Maybe let’s first think about “What do people run on this for model development and experimentation?” What’s your experience there, Chris? You mentioned notebook environments like Jupyter - do you run those off of your laptop? Do you have experience hosting those within your infrastructure for model development, or how does that work for you?

Dustin Kirkland: Well, so they are working hard on the underlying system call that it depends on. It needs to be able to allocate a TTY, basically, it’s just part of the underlying terminal technology, but it’s one of the top bugs, top priority bugs that the kernel team is working on right now. So it doesn’t yet work, but it is absolutely on the roadmap and certainly planned. It’s one of the most asked for features missing right now. So yeah, Byobu is a passion of mine. I am surprised and delighted that you asked about it, actually. It’s kind of a part time project, it’s certainly not part of my day job, and for the most part it’s implemented technologies; it’s really just bug fixes I work on now. But Byobu started out as a better version of Screen - if you are familiar with Screen. Screen is a beautiful program, but it’s kind of old and a little bit neglected, and then eventually tmux came along to be a better implementation of Screen. On top of both of those, I created this project called Byobu, which ships a set of opinionated configuration defaults and key bindings that turns either old school screen or the much newer tmux into a really rich command line windowing environment, essentially. It’s all command line, but it has a number of advantages. You can run commands and then detach and come back later, which is really important when you’re doing some work in the cloud and maybe you’re on a low connection or your connection drops, but you wanna make sure your long running command continues, or maybe you wanna kick off this job and then drive home and come back and check on it later. So that’s all very natively baked in, as well as the ability to split the screen - horizontal, vertical split, chop it up, run multiple windows… Sharing the sessions - that’s actually really useful. We use it for pair programming across the world. I mentioned that Canonical is very widely distributed around the world. It’s a very common mean for us to share a screen, so we’ll have an instance running somewhere in Amazon or Azure or open stack, and then we’ll run SSH import ID and we’ll give someone’s GitHub or Launchpad user ID, we’ll pull down their SSH key and then that person can SSH into that system and then with Byobu, we are both looking at the exact same terminal. [40:09] We are both typing and we can see what the other is doing. It’s great sharing a screen, at much higher resolution than say Skype of Google Hangouts screen share. You are literally in the same session.

Natalie Pistunovich: It’s not a Swedish company, as people think, apparently… No longer. It was, and still branded as one… But a friend of mine who is a cloud consultant was helping IKEA to migrate to the cloud, and to also adopt Go…

Adam Stacoviak: I think I’m with Jerod on the Sequoia preview. I mean, there’s a lot of stuff in here that I can just see that – one, big fan of 1Password, by the way. They are a sponsor; they may sponsor this episode, I have no idea… But Jerod, you may be excited about this integration. I’m less excited, because now I might be sort of divided between two password apps… But I’m happy that they finally have credentials all in one place, and an application that lets you command it. Not this sort of hidden – I could never really figure out, and that’s why I’ve never adopted keychain in the cloud kind of thing… Because it just seemed really hard to access, where 1Password has solved that in so many ways… And by the way, this is not a sponsored mention. I’m just a big fan, as I’ve said before a decade, pry it from my cold dead hands kind of person… And Apple may be at least peeling back one finger of mine…

Andrew Rynhard: I wanna touch on that, because I actually think that that’s really important to point out, and that’s actually a huge motivating factor around Talos, was because I was managing Kubernetes clusters, and the first place that I was doing this we were debating, “Should we do this with bare metal? Can we run CoreOS?” Well, typically we ran CentOS, but we’re also running this up in AWS… And I wanted this consistency story. And then we also had our developers that were saying “Hey, I wanna be able to actually spin this up on my local laptop and not depend on anything that you guys have set up.” Even though we went to great length to give them testing environments, they still ended up just creating their own.

[52:02] So Talos is really beautiful in that sense, because it’s literally the same image. The same image that runs right there on your laptop can be rolled out to anywhere - Raspberry Pi’s, the cloud, bare metal… Anywhere that you can imagine. And the experience is going to be consistent, more or less. Obviously, when you’re running in containers, you have the element of a kernel being the host operating system’s kernel, and networking, and stuff like that… But that’s minor. Those are things that you can kind of craft after the fact.

Dave Blakey: Yeah, you bring up a second unpopular opinion of mine. I mean, maybe it’s not unpopular, but… I don’t think that containers and Kubernetes and cloud-native is a destination. When we were all on tin, everyone said “Okay, everybody is gonna be on VMs.” And then were all on VMs, and now everybody is gonna be on the cloud. And that’s where it started to get shaky, because not everything did move to the cloud… And now this idea that the next step in that evolution is containers and cloud-native I think is wrong.

I think there are workloads that are excellently suited to that, there are workloads that are suited well to serverless, but there will always be workloads that are suited to tin that is within a mile of your house. I think it’s a spectrum now. It’s like, stop trying to make a round peg fit in a square hole. Not everything has to be deployed into containers, which is exactly to your point. It’s often the easiest thing, the thing that is used by the most people, that turns out to be the best decision.

Brandon Keepers: I think that just looking at GitHub’s product strategy if you will, GitHub is a tool that has become literally the hub of software development, but we’re under no impression that GitHub will be the only tool that you use. Every software team is going to bring their own chat service, or bring their own CI, or have their own internal tools that they have developed over the years that they want to use, that should work alongside GitHub.

What I get really excited about with Probot, one of the reasons has kind of been this passion of mine, this idea that this could be something that starts to glue GitHub to those other things. It doesn’t matter if you’re using Slack or HipChat, or if you’re using Travis or Circle or Jenkins… What’s the connected tissue between all of those tools? Especially as more of our tools move into the cloud, I think it’s important that those connecting pieces are open and accessible to anyone to use and to modify.