Tuhin Srivastava: Yeah. Well, I think what you’re doing is that – you know, you can run them all on a [unintelligible 00:23:49.15] API. Great. You’ve got this model, you give it an input, it gives you an output. Fantastic. Let’s carry on. But it’s like, the depth of features and the creation of workflow which are really important. The depth of features is that - hey, if you can do that with [unintelligible 00:24:01.16] You’re still gonna have to set up all the scaling, you’re still gonna have to set up observability, you’re gonna have to stop logging, and whatnot; hardware management. But I think the workflow is probably more important, to be honest, because we’re creating a defined way for you to publish new versions of this… If you want to A/B test two models, you can have two models running at the same time… It’s really that – the removal of boilerplate, and the addition of some workflow, so that when you’re deploying this in production, and you need to roll back a version, you don’t need to go and scramble to find that Fast API file that’s required. We’ve all been there before. And so that creation of workflow is probably, I think, what a lot of our customers probably use us for, besides – you know, the production-grade inference is a given, but a lot of the differentiation comes from that workflow.

Peter Wang: We’re still focused on the software supply chain, but of course, I’m very exposed to the kinds of dynamics you’re talking about, because I see these conversations happening in the practitioner ecosystem in the conversations. There’s a really important dynamic that’s happening here. Without getting to hyperbolic and biblical about it, data science and machine learning - this represents essentially the transformation of the software industry. So for the last probably 40 years, since the dawn of the PC era, but even prior to that a little bit, software developers or software engineers have been able to think of themselves as a distinguished class, like “We do software.” The hardware people, they’re writing Verilog and taping things out, whatever that means, and making chips that plugged in, powered on, now then we come in and we do our jobs. And then of course, that’s separate from the DBAs, or this other weird class of Oracle-licensed whoevers. They just sit there, speaking a bunch of weird SQL all day long.

[40:12] So this deconstruction of the information system into hardware, software and data management is deeply unnatural, and it’s actually something that was not the case. If you go back to the Norbert Wiener, into the early cybernetics era, no one thought about it that way. You listen to any of the founding fathers and mothers of this space, it was like “Yeah, I’m gonna focus on hardware.” But the PC era, and then everything that came afterwards, on the server side, on microprocessors, all these things led to this deconstruction of an information system into these three primary axes… Yeah, I guess it’s a decomposition into three axes.

And what we’re seeing again now, with data science and certainly with ML and AI, is that we now have a synthesis again; we’re forced to do a synthesis. We have to understand the runtime. And actually, the runtime characteristics of your software is data dependent. How weird is that? Imagine talking to the Java head 15 years ago to say “Well, okay, Mr. Java architect, check this out - if I pass in certain values out of this database, your code runs ten times slower.” That doesn’t happen, because you write a CRUD system that pulls a row, does some crap, and pushes a row, and it’s done.

So this idea that “Okay, runtime performance –” So the hardware footprint is dependent on data… And additionally, correctness is value-dependent. Can you imagine writing a unit test, and you – now we have these for models; we have model tests. But prior to models, it was all just code. Can you imagine unit tests where it’s like ‘Well, one plus one is two, so my add function works, but it only works for even integers.” That’s weird, right? And yet, we know that when we build these AI systems, these models, their performance, their correctness is actually value-dependent. And this is a point that I don’t hear anyone else making, maybe because I come at it from a physicist perspective, and I think about deconstructing everything to fundamentals… But deconstructing the computation concept into fundamentals - for the last 40 years we’ve had value-independent processing.

Jim Gray has written papers about this, and people have talked about this, but your average coder nerd doesn’t think about this at all. They’re like “I’m a software dev. I’ll learn this thing; I’m gonna learn Go this year, and I’m gonna do something else next year.” But the whole field of software is going away, it’s melding into – we might call it model development, we might call it something else, but I call it value-dependent or value-sensitive computing, and now your management of your upstream data is as important as managing the upstream code.

The previous approaches to data management don’t work anymore. But of course, checking in every row of a database into Git doesn’t work either… So we have to develop an entire new set of practices for this new industry. All of the previous components, those axes are important, but they can no longer be seen as separable. They’re now integrated. Anyway, thank you for coming to my TED talk about that… [laughter] But that’s the lens I look at all this stuff through. So it’s no wonder that we have model hubs emerging, but I think the management of those things and how we talk about versioning of data, the model performance and characterizing it - all that is a nascent and emerging area, and it’d be fascinating to watch how that really goes as it meets production in the real world.

Nick O’Leary: Great question, Chris. We’ve got quite a healthy collection - both for an open source project - of other companies who are making use of Node-RED. To slightly avoid that question initially, this was one of the big things we decided with Node-RED, to develop as an open source project, and we got there really quickly. It was about six months into the life of it we decided to open source, because we saw the real value was building a bigger community around it. And that has certainly borne out.

Today, other than ourselves, Hitachi, probably the main other contributor - and they are very heavily using Node-RED as part of their own IoT platform… Great examples out there; you can go find about the sorts of things they use it for. It is quite central. So they’ve become a great contributor to the project. And then there’s a long list of – well, it’s interesting to see how it breaks down. There’s a load of companies who create hardware - industrial controllers, or industrial gateway-type machines; you’ve got people like MultiTech, Opto22, there’s a company in the U.K. called iaconnects… They all create hardware for interfacing, whether it’s with building management systems, or industrial automation… They create gateway devices, and they have built Node-RED as part of the software stack on those devices.

For them, it’s all about – their users are not necessarily developers, but they understand the environment they’re running this hardware in. They’re system engineers, or they own a building. So Node-RED and its higher level of abstraction makes it easier for them to define the behavior of this hardware.

Bryan Cantrill: Yeah, it’s a great question. I would like it to. I think we would like it to. RISC-V - so there are some great elements of RISC-V. I do love the instruction set. The instruction set is very thoughtfully designed. And the fact that we can have open cores for RISC-V is great. So the fact that you can have your own Bluespec core, what have you, RISC-V core - that’s great. The challenge with RISC-V is that – and we’ve seen this happen again and again, where you have something that has good intentions, and then kind of the tentacles of the existing way of doing things spread into it. And RISC-V has developed pretty proprietary firmware, for all of its openness. Yes, the instruction set is open. The actual systems based on RISC-V have got a lot of proprietary elements. And indeed, they have recreated some of the most gallingly proprietary elements of x86.

So things like SMM - SMM is the System Management Mode. It is a mode that the processor enters when it wants to, and does whatever it wants. And if you are accustomed to writing the operating system - I do OS Kernel development - the idea that there is some hidden mode that you can’t see, that gets to do whatever it wants in the machine… Well, that operates across purposes. I mean, that’s a source of security vulnerabilities, that’s a source of performance problems… And those kind of management modes should not exist. Those are antithetical to hardware/software co-design, because those modes – what those modes are saying is that there is some software beneath the system software, that is unseen, that is actually controlling the computer. And it makes it very hard to build a unified system. We actually – of all the things we’re doing at Oxide, and we’re doing our own compute slide based on AMD Milan, we’re doing our own switch based on Intel Tofino, we’re doing our own cable backplane, we’re doing our own operating system on a service processor, we’re doing it on Hypervisor… So we’re doing all these very big, bold bets. Maybe the boldest is there is no AMI bias in our system. So am I American Megatrends Incorporated, which even the name drips from the ’80s…

Justin Searls: [00:20:07.11] It’s not that hardware is easier than software, it’s that software requires a much higher degree of discipline, because it doesn’t talk back to you. You don’t push on it with your thumb and then feel that resistance from the thing. And the physicality of hardware and the iterative nature of how we can manufacture things, even proofs of concepts of things, lends a certain tangibility that even senior-senior management and executives, five levels up, can grope and engage with in a certain way. But when you’re on stage and you’re giving a demo… Like, Craig Federighi’s very first demo, I think it was in OS X Lion, and it was at a time when Steve Jobs was clearly trying to groom a few people beyond just Phil Schiller to get up on these keynotes. You see Federighi there, with the hair and everything, and the personality, but he’s using a magic mouse and his hand is so god– I forgot, we don’t swear so much on this one. So sweaty… [laughs] Sorry, Breaking Change is an extremely explicit language podcast, and if you do subscribe and you’re very surprised by that, you’ve now been warned. But his hand was so sweaty that the magic mouse capacitive touch wasn’t working, right? Because he was trying to demo the multi-touch features.

And you can just see – as they zoom in further, as it continues to not work into the poor guy’s hand, you could just see the trepidation and the quiver in his finger.

Jerod Santo: It’s now time for Sponsored News!

WarpStream lets you bring your own cloud

WarpStream’s BYOC deployment model for Kafka-compat data streaming gives you the best of both worlds between self-hosted & cloud. This model also give you a lot of zeros (that’s a good thing)

Zero disks - WarpStream’s Zero Disk Architecture eliminates local disks entirely and reduces storage costs by more than 24x.

Zero inter-zone networking fees - More than 80% of Kafka costs are not hardware – they’re inter-zone networking fees. Because WarpStream runs on top of S3-compatible object storage & doesn’t manually replicate data between zones, those fees are completely eliminated.

Zero ops auto-scaling - WarpStream replaces stateful Kafka brokers with stateless Agents, so your team can skip the weekly burden of partition rebalancing, scaling headaches, volume management, capacity planning & more.

Zero access - WarpStream Agents run on your VMs, in your cloud account / VPC & storage data in your object storage buckets. Traffic flows seamlessly from producers to consumers without ever leaving your virtual private cloud (VPC).

Here’s a non-zero - WarpStream works in every cloud! They have native support for AWS S3, GCP GCS & Azure Blob Storage. It also works with any cloud or self-hosted solution that has an S3-compatible object storage.

Head to warpstream.com/byoc to learn more and get started for free.

Don Beyer: Before answering that though, you touched on something that has always fascinated me. I don’t know if you’ve read Kim Stanley Robinson’s Mars trilogy. It’s a wonderful book. It got me through a gubernatorial campaign, 15 pages a night. And one of the things in it that I really took away from it was the idea of both leadership from the balcony, and leadership from the field. That one of the lead characters in it would spend a year or two managing the planet Mars from like a general manager level, a presidential level, and then he’d go out in the field for two years and live in a tent by himself, looking at Mars biology.

[00:10:04.04] And when I was in the car business, just to be mundane once again, I’d always spent two weeks every summer working as a technician in the shop as a mechanic. By the end of the third day, I was cursing [unintelligible 00:10:13.13] and I’d hate management by then, by the third day. But I’d come away from it at the end of those hot summer days with cut hands and a real appreciation for what it was like to work on cars all day long. Replace the water pumps, and trying to figure out where that obnoxious knocking noise is… And now I’m finding the exact same parallel with artificial intelligence, and serving on the AI Task Force, and the AI caucus, and the various committees that we have, that it’s really fun to see – and I’m still very much a rookie, a baby in this field… But I can imagine what two or three years later a senior software engineer is doing, the people that are building these wonderful models right now, that understand how the neural networks come together.

On the other hand, trying to figure out, what do we do about deep fakes? What do we do about hallucinations? What do we do to protect our electoral systems? To look at the policy side also. And I’m blessed to have people like – I have a wonderful tech fellow, which means some other foundation’s paying for him for a year, who’s an MIT computer scientist who works in AI… And he actually knows the math and the computer science and the hardware of it, to help with me on the policy side.

Bryan Cantrill: Exactly. And so actually, with this particular firm, I remember telling them that the – also, in particular, on our first slide, it’s like Jeff Bezos, at the time - now it’s Andy Jassy… “This is the company. Jeff Bezos is not going to own and operate every computer on the planet. Not every server is going to be a public cloud machine. There is going to be on premises computing. Folks that run on prem, whether that is for economic reasons, as Samsung had to, whether that is for risk management, or security reasons, as certainly you see in federal spaces, government spaces and so on, whether that is for latency reasons, as you get people kind of collocating near their data, and so on… There are lots of reasons to run on prem.” And they’re like, “Okay, yes.” But then you get to the next slide and their first objection is like “But won’t it all go to the public cloud?” I’m like, “Okay, this is why we shouldn’t be here… Because you only see public cloud computing. And we are very pro-cloud. I love cloud, I think it’s great.” I think for startups, public cloud is perfect, because when you are starting from zero, of course you should not buy your own hardware. Like, go spin that up in the public cloud. That’s awesome. But you get to a certain scale, or you have certain constraints, you will need to own and operate your own computer, in your own data center, collocated data center. So they just don’t see that. They don’t see that world.

[01:24:18.14] So with this particular firm – so every time I’d be like “You know what, this is why we should leave.” So kind of like “We should leave.” “No, no, no, no. Stay, stay, stay. No, we’re interested. You’re making us more interested.” And like “Oh, God, what are we doing? Why are we doing this?”

And the real problem is they would kind of convince us to stay, and we would say. We should have just like walked out. At one point I did tell that particular firm, I’m like “You have to understand, we are going whaling. You should be fishing from the pier. You should not come whaling with – it’s like, stay here. You don’t want to come out on this whaling vessel with us. You do not want to be on this multi-year, dangerous voyage.” But again that did nothing. And of course, they spent a ton of time, and then ultimately, they realized, like, “I think we should fish from the pier.” It’s like, “Yes. You should fish from the pier.”

Louis Pilfold: It’s like the Rust thing. Like, I love Rust. And Gleam’s compiler is written in Rust. But so often I just wish it didn’t have the memory management stuff. Like, I just want to have a nice, fast compiling to native language, and not have to worry about lifetimes, and all this sort of thing. So I think having a language that is more general works really well. Maybe I’m wrong. I’m pretty heavily invested in the Gleam thing, so maybe I’m not the best person to ask… But I think we’ve got a really nice sweet spot. I’m trying to find like the smallest set of generally useful tools. And I guess it is, to some degree, specialized. This language has been made from over a decade of my working in the industry, making boring business software; websites, and clients for things, and background processing… And it’s just the language that I wanted to use for all those things. I worked very briefly in video games, and music hardware… Probably not a good language for those places. But for their business systems - yeah, it’d be good for that.

Kris Moore: So we do address some of that. We have our mini line of TrueNAS’es, we have our minis, which are like little desktop units that are quiet and efficient, and you can do four drives, five drives, eight, whatever… And then we also just this last year unveiled the Mini R, which is a rack-mountable Mini. So great little system; you can get those if you’re a homelabber. Again, some homelabbers want to go build their own, which I get, that’s part of the fun; it’s the hobby thing. But if you’re like “Yeah, I’m a homelabber, but I don’t want to go do my own hardware, and I want something that iX supports, and has all the testing and vetting… Or I want, like, say the enclosure management bit”, because that’s a piece that’s really hard to do when you build your own hardware; we can’t just map things for you automatically… That’s a perfectly valid option.

And so yeah, we offer those; they’re up on Amazon, they’re up on the company website and store, and we see people grab those… And that’s a good gateway device, especially even larger enterprises maybe will try that just see how it performs in their space, and kick the tires on an official platform before they go and scale up.

Jack Shanahan: Yeah, a few thoughts on that. And while I remain at AGI skeptic, I do believe these machines, these AI-enabled or smart machines will get so good, the human interfering with the machine could be a worse outcome than human-machine together. And the example that I’ve heard - I think was Gilman Louie raised this about AlphaGo and the Move 37. If a human was there to override the machine, it would have lost the game. The machine said, “No, you really do want to make this move. Leave it alone”, and it won. So that’s an example of - humans are going to get to the point where they’ve got to be more comfortable with this technology. And they’re not necessarily today. Why? Because just like when I use ChatGPT, whatever version of a large language model, it does get some things still very wrong. Maybe the ratio is only 10%, versus 20% just a year ago. But if that 10% is dangerously wrong, then do you really want that in military operations? Not yet you don’t. So how do you do that? This is the core of your question. To me, it is testing evaluation, it’s experiment, it’s putting it into the hands of users as an MVP, a minimal viable product… Which again, is a little bit different, or a lot different than the way the Department of Defense has fielded systems in the past, where you field any system when it’s as close to perfect as you’re going to get… Which may have not been perfect, but close enough for government work, as we say.

But in this case, you need to put things in the hands of users sooner rather than later. This is why I’m a big fan – despite my reluctance to say “We should not be using large language models for putting items in the presidential daily intelligence briefing”, you should have all sorts of experimentation being allowed in all these federal government organizations and agencies. In fact, the White House executive order on AI says “Go try this out.” It does – it basically says “Go do it.” So I’m a fan of the experimentation, what we’d call sandbox experiment; put it in users hands. Because we don’t know yet. But part of that is core testing evaluation. We do not want a short-circuit testing evaluation. And why is this so important? Because I think that’s where we’re gonna see the biggest risk, at least initially.

[00:42:10.24] Some risks are going to be hard to figure out until you use them in an operational world. We just know that, just like in commercial business. Some things are going to surprise you. But that’s why you then update the models, because you learn. But in that early stages of design and development, before you get to the fielding part, there is this thing that we have done really well in the government for many, many years, especially on the hardware side; it’s called Test and Evaluation. That does not go away when we talk about AI. In fact, AI is still novel enough, and I think it’s more important than ever to get spend a lot of time on the T&E. It doesn’t mean you’re going to go slowly relative to sort of putting out other things, but I am a little bit cautious about moving too quickly on some of this, because we just don’t know what those risks are yet.

There’s a lot of work going on, including some things I’ve been working on on a risk management framework for AI systems in the military. Because this risk - it is a hierarchy, one, and their AI-enabled weapons are really, really bad. On the other end, there’s process automation for a finance system that touches nothing having to do with warfighting. Negligible risk, move really fast on that one. And then a lot of things in the middle. You’ve got to do test and evaluation to determine how many risks there are, and then come up with risk mitigation strategies.

Jamie Tanna: Yeah. But I like it, because it’s kind of a reflection of my inability to name things well, and think of catchy stuff… But yeah, so dependency management data is a project I was building while I was at Deliveroo… And it was coming up to Hacktoberfest, and I was a big proponent - and still am a big proponent of open source, of giving back… And aside from issues that maintainers have during Hacktoberfest, I’m still very pro “Let’s get some more Deliveroo engineers contributing”, because everyone should be trying to at least contribute to that.

And one of the questions I was asked a number of times was “Well, which project should I contribute to? Because it’s hard to know where to start with open source.” And instead of sending a “Good first issues” list, I was like “Let’s look at the dependencies that Deliveroo has”, and let’s say “Of all of these, these are the ones we use the most. So maybe have a look at these ones.” And Deliveroo is like a very good data-driven organization, s I was like “Okay, I’m gonna get some data to actually show that.”

So I ended up building some very horrible hacky scripts, which would take output from GitHub’s Dependabot graph and it would then parse those and spit it out into an SQLlite database. Over the next few months I realized that actually this is quite a useful thing - being able to have a database of all your dependencies, and being able to query them, and being able to add additional insight is actually really powerful… So I started playing around with EndOfLife.date, which is an API for looking up common end of life or deprecation dates… So for instance, how long is Node.js supported? I don’t think they have any physical hardware… But as lots of programming languages, they are very open to supporting new things… And so I was starting to add this data and realized that this actually shouldn’t be a thing that just sits within Deliveroo. It’s something that I can absolutely see other people getting benefit from. And I was spending a lot of evenings and weekends on the idea anyway, so I started an open source project for this called DMD… And it has just passed its anniversary of its first birthday, so I’ve been spending just over a year… I think one in three days of the last year I have been committing to the project. So I’ve spent a lot of time working on it.

[01:06:16.14] And as well as like being able to just have a list of those dependencies, and being able to say things like “Where are we using TerraForm in the organization?” Or “Which Go HTTP servers are we using, and how many a different team’s using?” You can also do things like looking up OpenSSF scorecard data, to say things like “Of the dependencies, my most critical application in our business support relies on, how many of those do zero code review? And how many of them are just yeeting things into prod?” And it turns out, it’s quite a lot more than you would think.

So having a lot of that data is really useful… And both my time at Deliveroo, and now at Elastic, we’ve been spending a lot of time using that data to make some really interesting decisions, and interesting understandings of our data… Yeah, you can find some very interesting things. And I’ve said interesting about a dozen times there.

Techno Tim: No, I like it. Yeah, that’s my stack, too. So first, I start with a machine, a hypervisor. So if you’re not familiar with hypervisor - most people are, but if you’re not, it’s just really a machine that you can run virtual machines inside of. There are different types of hypervisors, but for all intents purposes, run machines inside of machines.

My choice has been Proxmox. It’s been for a long time. I used to run Hyper-V, which was a Microsoft product, I’ve run ESXi at home, which is a VMware product… None of those gave me the flexibility that I needed at home without buying expensive licenses. And so Proxmox was there, it was open source, and early in its development, and it hit all of those things that I wanted; a web-based hypervisor that was performant, that’s Linux-based, that’s open source, and has some enterprise features if I want them.

And so it’s Proxmox… From there, I usually virtualize everything. So everything’s virtualized. I tried to virtualize everything, just because it makes management and backups a lot easier. Pi-hole is definitely one. Pi-holes - I have gone a little overboard; I have three Pi-hole servers. Two of them are load balanced behind a load balancer, with a software load balancer; not important.

But then I run containers. I’ll say I run containers… I used to do a ton of Docker. So I run Portainer, which is a Web UI for Docker, which makes Docker very approachable for someone who doesn’t know Docker. It gives you a nice Web UI. And so I’ve run a lot of containers in there; Plex, Pi-hole, you name it. [unintelligible 00:27:47.13] A lot of dashboard stuff…

[27:52] And then since I did a lot of Kubernetes at work, I decided to do Kubernetes at home. I will say that it takes a lot; it takes a lot to grok Kubernetes, to understand Kubernetes, and then to run it at home and maintain it. So I don’t recommend it for the average home user. It’s definitely not approachable. There are some things that make it a little more approachable, like Rancher, and even Portainer, but for the most part, not approachable… Because it’s made to be this huge state machine for running high-availability applications. And so you need lots of machines, lots of hardware, and lots of resources. You can minimize all of that, but at the end of the day it’s going to take mostly three machines, which is out of the question for most people.

So yeah, I containerize everything. I always try to containerize everything, mainly because I want to run multiple containers on one machine, so Docker containers on one machine. I don’t want to have to worry about global dependencies, or all of that stuff that you worry about if you have a virtual machine. If you think about it - you know, back in the day I used to install WordPress on a machine. And then I would install - you name it, some other software on a machine, because I wanted one machine to do everything. Well, at the end of the day, when you start doing that, you might, say, for WordPress need PHP 8, and MySQL 7, and then you install something else on that machine, which requires MySQL 8, and now you have these broken dependencies… So that’s where Docker comes in and kind of keeps all of those in little containers and silos, so that you can run multiple things without them affecting each other.

But yeah, everything’s containerized. I run a ton of stuff… I run a ton of stuff at home. Uptime Kuma, I monitor and log all of my services… I have a lot of custom code that I wrote, a lot of websites that I write… I mean, I do a ton of Node.js, I do a ton of Docker, I do a ton of TypeScript; so I build and deploy – I have CI running in my homelab… I write code, I push it to my CI, it runs my tests, it builds containers, and then it puts them in Kubernetes. It used to put them in Docker, but yeah, now it puts them in Kubernetes. And that’s all a home. I learned how to do all of that kind of in my homelab, so… But yeah, it’s a lot of services. Probably too many. Some of them are for testing, but some of them have stood the test of time, like Pi-hole. I absolutely love it.

Bryan Cantrill: And part of the reason that Steve went to a cloud computing company in 2009 is because he couldn’t really get Dell to understand the importance of cloud computing. And you see this over and over and over again. Go look at the backgrounds of people doing cloud at Google, at AWS, at Meta, and you’ll see the Dell and the HPE and their own past, and you know that they left because those companies didn’t get it. As those companies didn’t get it, they got further and further apart. And so those designs haven’t moved from 20 years ago.

So in order for us to be able to go deliver that hyperscale-class infrastructure, hardware and software together, we’ve gotta go back to where the trails diverged, and we’ve gotta go down the right path for on-prem. The problem is they diverged so long ago that we have to take on a huge, huge problem. And the minimum viable product for this company is enormous. As Steve was alluding to, it included the networking switch, it included getting rid of the Baseboard Management Controller (BMC), doing our own service processor, doing our own software all the way up and down the stack. So VMware does not run on this box. ESX does not run on this box.

Christof Henkel: I would say there are a lot of synergy aspects. So doing a Kaggle competition is really very similar to doing a project at work, which is about performing a first prototype. So in a Kaggle competition you get a problem, which you’re not familiar with, often from a different domain; it can be from biology, it can be from astrophysics, it can be from chemistry, it can be Bengali language, sign language… Just so much different problems that you have no clue about when you start. And then you finally have like three months’ time to find like the best possible solution, and also compete with other data scientists.

This prototype project character is very seminar. So you have like this three-month time window. Then you have a collaborative part. In Kaggle you can also form teams, so you can participate in competitions in a team, which is very similar to working in a team in your job, with all the ups and downs, I would say, with working in a team, under pressure often… So Kaggle competitions can create quite some pressure, more pressure than you might feel in your day to day job. So you also get used to working efficiently with others, in terms of coding, in terms of reading their code, in terms of structuring the project… So really, all aspects of project management are also important. And also things like optimizing runtime, and optimizing code structure. You wouldn’t think that it’s quite important, but I think it’s quite important also for Kaggle competitions, because recently, they run the competitions on a restricted hardware; so you just submit your code, and they will run your code on their infrastructure, using their Kaggle notebooks. So you need to have your code in a way that it’s kind of production style; that’s also what you would do in a project - so you would develop ideas, and so on and so forth, but at the end, you will want to productize your code, and you need to think about all these MLOps problems as well. And you also train those skills in Kaggle competitions.

[18:08] So I really like the parallel between the two worlds… That said, I’m gonna say that two things are really different between Kaggle and the real world project. First thing is data acquisition. That’s like a very big topic. In like the real world it’s no topic– not no topic, but a very minor topic on a Kaggle competition you already have your training data. Of course, you sometimes can expand your training data by looking for more data online, but in general, you already have like a fixed training set you can work with… Whereas in the outside world, on the real world, that could be like the main problem, just to require some data.

And the second thing is definition of the metric. So in Kaggle, people are evaluated based on some metric, and this metric is predefined before the competition starts. Whereas in the real world, that can be a discussion which takes for ages between like data scientists, the business, and just creating a metric that is representative of the business problem can take a lot of time, and you don’t have these issues and discussions in Kaggle.

Kenneth Auchenberg: I actually think of infrastructure in the same ways, in terms of it being like pieces of components that you can leverage to kind of build the experience that you want to provide. But I probably don’t have a tight connection between infrastructure and hardware, in the sense that I see that as an abstraction ladder. Like, let’s say that I want to build a PaaS today. You will probably rely on another computing abstraction that you’re using, that is so far away from hardware anyway, that you have a virtual machine of some sort, and you have another runtime. So even as you kind of compose kind of like a new kind of computing abstractions as infrastructure, hardware for you anyway is abstracted away.

[30:12] So when I think about infrastructure in a DX or in a software context, is that we think about how I’m composing a modern, let’s say software as a service business, I probably need some sort of compute layer; I probably need like a Vercel, or a Netlify, or AWS Lambda, how I’m putting my stack together. But at the same time, I probably need an authentication piece. That is not a hardware piece, but it’s a set of services I’m using to provide auth, right? Or maybe even I’m using a combined provider that does all that for me, like Firebase.

So what I mean by that is that to me, infrastructure is the business components that you’re leveraging to kind of put your product together. And the argument I’m making for DXI as a category is that we have reached this maturity point where if you have to provide all the aspects of a great modern developer experience covering your product, docs, content, community, you cannot build it in-house anymore. You have to rely on these new infrastructure pieces that are emerging from this new market landscape of companies that are saying, “I want to be the best in providing SDKs. I want to be the best in providing documentation infrastructure. I want to be the best in doing community management.” And then I as the founder, I as the C-level at a company, instead of sitting with my budget and saying “Should I allocate 100 engineers to build internal developer experience?” No, I would rather use a different part of my budget, and use a vendor for these kinds of things.” So it’s through that lens that I’m seeing infrastructure.

Luis Ceze: Yeah, so I would say, again, being able to get a model that was freshly created by a data scientist, a machine learning creative, without thinking about the systems aspect of deployment, having their models benchmarked, knowing how it runs and know how much it costs, they’re going to click deploy, it’s going to produce a box that the DevOps folks can go and deploy within the application… I think we have line of sight to that, and I hope next year in the show here, we’re going to be talking about all the ways that people are using that.

But now, if you ask me to think about what about five years out, I think what’s interesting to think about here is that - how do you even stop thinking about what’s edge, what’s in the edge, what’s in the cloud, what runs where, and just think about, “Hey, I want to solve this problem that involves machine learning”?

So users are on the edge, computers are in a building somewhere, spread all over the world, right? And I think an application creator - I don’t want to use the word developer; an application creator here should be able to specify, “Here’s what they want to do”, and then the system should automatically figure out, “Okay, so we need these kinds of models, and this one should run in the edge, this one should run in the cloud, this one might run in a cell phone-based station”, and automatically split what should run where. That should be done automatically, because again, that’s an optimization problem that once you define the constraints, you should be able to place the right piece of your model in the right physical place automatically. And if we nailed the automation that we just talked about from a model to the thing that runs in a hardware really well, that part is already also done, right? So now we can go and work on a higher-level problem, like how do you break down your model into the pieces of “What’s in the edge, what’s in the infrastructure, what’s in the cloud?” just the right way.

I also think that that should be possible, because hey, we have machine learning designing chips today. That’s pretty hard, right? So you have machine learning designing much better ways of doing power management in large-scale data centers. These are all low-level things that we used to do by hand, or with heuristics, and we abstracted that away. As this goes up and up the stack, I’m very excited about the future of creating exciting applications without having to worry about all the design constraints that we have to worry about today.

Deepthi Sigireddi: So there’s definitely a hardware cost to it when you start deploying Vitess. Typically, what people will do is they have their MySQL instances already, and they’ll just put Vitess in front of it, and start routing the application queries through Vitess. Obviously, you have to run these additional components that Vitess brings in. For every MySQL database that you have, you run a management component called vttablet, and you run a bunch of proxies which will receive the MySQL queries connections from clients, and then pass them to the tablets which will eventually execute those queries on MySQL. So there is a hardware cost associated with it.

There is also compatibility. So Vitess is MySQL-compatible, but MySQL query syntax and constructs keep evolving. Vitess started (I think) maybe on MySQL 5.5, and we now support almost all the constructs in 5.7, but not everything that is 8.0, because a lot of new syntax was introduced in 8.0. And where this becomes hard - it’s easy if you’re unsharded. You sort of just have to understand the syntax and you can pass it through. But in a sharded system, what you end up doing is that you take each query and you have to plan that query, and you have to say which of the backing MySQL should this query eventually go to? Because it’s not going to go to all of them; it’s going to go to either one of them, or a subset of them. In rare cases - yes, if you do select star from a sharded table, you have to go to all of them. But usually, you want to go to a subset or a single one. And query planning is where we figure this out. What that means is that any new syntax or construct that MySQL introduces, we have to be able to understand that, because maybe it’s a function, and the function is being applied to the results of multiple rows, and you have to know how to implement that function in Vitess in order to still provide that transparency. What’s another example…? Or it’s a join, and if it’s going to be a cross-shard join, then you have to do some of the processing in-memory, after fetching the results from different individual MySQL databases. So all of this is happening in Vitess, which means that to this date there is a compatibility gap between Vitess and MySQL, and we are trying to close that.

[48:16] We started the compatibility project formally in… I have to think. I think January of 2020. So it’s been two years. And we are still working on it. We have a sub-team at PlanetScale, all of whom are Vitess maintainers, who focus on closing this compatibility gap. And the way we did it was we said “Okay, we’ll take some popular developer frameworks, like Ruby on Rails, or try to run WordPress on Vitess and look at the queries that are being executed.” And especially with frameworks, they execute some preamble queries, some information schema, metadata queries, and so on. And then we start adding support for those things.

But to come back to the original question, which is “What does that journey of adopting Vitess look like for someone who starts on Vitess?” You may find that there are some queries that you are using which don’t work with Vitess in a sharded mode. It’s much rarer with unsharded, but there was a user on open source who said, “Oh, I’m using common table expressions. They don’t work with Vitess.” And I had to say “Yeah, they don’t work with Vitess because we haven’t added support yet.” But most of the time it’s not unsharded but sharded that causes the incompatibilities. So people will take a test environment, they’ll have some test data, they’ll put Vitess in front of their MySQLs, they will start sending some test traffic through Vitess, and then they’ll discover what other things that don’t work. At that point they have a choice of either contributing a fix upstream to make it work, or changing how their application works, so that those queries are not produced anymore.

So that’s the sort of journey people tend to go through when they come to Vitess. And then, once they know that either all the queries work, or we’ve made the - usually not big changes; minor changes needed for the application to work, then you can start moving from test into production.

Zac Smith: When 5-6 years ago that just wasn’t right? It was x86 or you’re done. And now you’ve got serious languages that have been made multi-arch and have the build capacity and the CI pipelines and the related ecosystem to make that continue, and “build upon itself.” That’s happened faster than I expected, where the software has met the hardware… And the hardware is also changing so rapidly that there’s just so much to do.

So I envision that over time we’ll create a much better what I’m gonna call HCL (hardware compatibility list) for the internet, that effectively can be an idempotent view of every single piece of hardware ever, and that would allow all the software to be able to choose and understand how to work with it. We’re pretty far off from that… But I think we can get somewhere there.

But I think that’s where I’m gonna give your answer, is just like being super-crazy, laser-focused on what we do… And I’ve spent a lot of time in my first few years at Packet – I’m not gonna say fending off revolutions, but a lot of people… The clearest one on my mind was I almost had most of my management team walk out because they said we had to launch load balancers, otherwise nobody could use our thing. I said, “No… I think software will figure it out. Let’s just provide really easy, smaller hardware instances, and they’ll figure everything else out”, and they’re like “No, it’s too hard. We’ve gotta do load balancing.” And then look a couple years later… You’ve got Ingress controllers, and service mesh, and all these kinds of different – BGP control in Golang… It’s cool. I mean, it’s not for everybody, but software solved the problem.

So a lot of that was just staying super-focused on what we did, and I think some of those other providers that you mentioned, of which I’m huge fans, and know the founders of most of those businesses, that moved our industry in their own way… But they became (I’m gonna call it) all-purpose platforms in a lot of ways. And that’s probably right, in some regards… A lot of the industry has moved to that direction, especially with hyperscale clouds, having these just robust software catalogs and ecosystems… We’ve been fortunate enough to have venture backers at Packet who really saw our vision for what it was, which is staying fundamental in the primitives business… And frankly, here at Equinix, which really knows that it is a builder of physical infrastructure that can move at software speed. That’s our job. Our job is not to do all the things; our job is to enable an ecosystem so that they can do all the things. So that has allowed us to continue to focus in on just like “Let’s be the best at this, in the whole world.” That’s it.

Evan Kaplan: [04:05] I do have some stories, man. When I take it back a little bit about my history, and how I show up meeting Paul at Influx, and all the dynamics around it… So my background is a native New Yorker, and I moved to the West Coast because I was really interested in climbing and skiing… And I didn’t have a real job until late twenties. I’d mostly been working for organizations like Outward Bound, or guide in climbing or skiing, or working with the judicated youths or things like that. So totally nothing to do with tech. Always interested in tech, but just from a distance.

My first job was working in aerospace, actually not doing anything technical, but doing management training and development… I did that for a year, and got really into the technical side, and I jumped into a program management role for flight computers, working closely with Boeing, and Airbus, and companies like that… And I just really enjoyed working with engineers and developers, software and hardware.

I had a climbing buddy who had started a company in Seattle; probably people don’t know that anymore, it’s now part of the roleup it’s called WRQ it’s a super fast-growing company… And I got involved with them, and I joined them, and was around for the early days of TCP/IP…

Ville Tuulos: [27:54] Good question. Metaflow, since day one, has been built with this cloud-first mindset. I think when it comes to things like compute and storage, we live in a bit of a post-scarcity world. It’s actually interesting when you think about it, that many of the systems that we even use today, like databases, even things like Spark and Hadoop, they were built with this idea that you have constrained resources, and really the engineering challenge is that “Okay, how do you allow people to run compute given that you have only 200 servers, or something like that, and you have to do the resource management very carefully?”

The mindset that we adopted with Metaflow, which I think is really useful, especially in the people’s productivity point of view, is that you work with the cloud, the cloud provides you at least this abstraction of having basically infinite scalability… So you can use some cloud-based platform… Again, we rely on existing systems like Kubernetes, like AWS Batch, what have you, to kind of farm out the containers to the cloud. You can specify the resources you need, if your function needs GPU… So you can say “I need GPUs in this case.” In other cases maybe you need a lot of memory.

I think the interesting challenge with machine learning - I think this also, like I said, it’s machine learning apart from many previous data-intensive applications is that the needs are so different. Some types of models are really IO sensitive, like maybe you need to read tons of images, tons of videos, but the model itself maybe is simple. In other cases it might be something super compute-intensive, but not IO sensitive. In some cases you absolutely need a GPU, or maybe even some custom, crazy hardware. In other cases it is cost-prohibitive to do that, and it’s very hard to have any kind of uniform one-size-fits-all. So I think having that scalability that goes in all the different dimensions - vertical, horizontal, you name it. So that’s super-useful. And cloud makes it possible. So we rely on these systems and we take care of then packaging the code, sending it to the cloud, executing on the container, handling retries… All that kind of basic, basic plumbing there.

And then yeah, the same thing like with the orchestration - the DAG execution at scale; if you have 100,000 DAGs running in parallel, it’s a thing of its own. There are some systems that do it well; we integrate with AWS Step Functions, now we are integrating with Argo, maybe one day with Airflow, what have you…

The idea is that you should be able to test these things locally. Metaflow always comes with the local mode, so you can kind of test any [unintelligible 00:30:10.13] the same way you do in a notebook… But then when you want scale, like when you want something that’s production-ready, you can use your existing production infrastructures.

Luis Ceze: Well, there’s just so many things. Let me start with the shorter term and then go to the longer term. So in the shorter term I think it’s really exciting what’s going on in doing more and more hardware-aware network architecture search. There’s some companies doing it, but essentially, as you evolve how your model looks like with network architecture search, you do that in a hardware way. I think it’s super-important, because it’s complementary to everything that we talked about here. What we talked about here is having a model that is ready to be deployed and you’re just gonna compile it; but now having the model in the first place will actually be better for the hardware you create. So that’s one of them.

[44:07] The other one is more automations in the data management side. For example, I love what the Snorkel.ai folks are doing. I don’t know if you’ve heard of them, but they essentially have tools that enable you to construct synthetic datasets in a programmatic way, that essentially automates a lot of the work that’s required for you to start building a dataset to train models.

And then on the hardware side, what I think is happening right now that is really exciting is just to see more and more reconfigurable architectures coming in the mix. I’m sure you’ve heard of CPUs, GPUs, and TPUs like accelerators… You’ve probably heard of FPGAs as well, which is essentially a hardware fabric that you can program very much in the same way that you design hardware, but it’s not quite as fast as a truly application-specific chip, but it’s pretty general, and then you can do a lot of meaningful things with it.

I find it exciting that those FPGAs are getting more and more tuned for machine learning. Xilinx has offerings that way, Altera is enriching their FPGAs with more blocks that are more suitable for new models. I find this exciting in the short term.

In the medium to long term I just love what’s going on in between machine learning and life sciences. Just seeing machine learning enabling very large-scale genomics studies, just crazy amounts of data to go and make sense of, data that is incredibly complex, because nature is a complex beast… And then also using machine learning to design systems. People are using machine learning to design molecular systems, people are using machine learning to design aircrafts, people are using (what was it again?) reverse design, where you give it the properties and it synthesizes something that has those properties using machine learning… These are all things that I find really, really exciting to think about, because machine learning itself is also systems.

So using machine learning for machine learning improvements is pretty interesting. We do that to some extent, but I feel like we’re just scratching the surface. So you can use machine learning to design machine learning chips, you can use machine learning to optimize machine learning training systems… And when you close the loop there, that’s when you should embrace and let it evolve, right? [laughs]

Dave Lacey: I think it does a lot, actually. I think CPUs are amazing; they’ve been so successful in fixing an architecture, fixing all aspects of the architecture - the way the instruction set runs, the way that memory hierarchy works… And then applying it to lots of different fields and having that success I think is a remarkable thing. I don’t know if it works for all fields though. And I think what we find in AI is one thing that we’ve been very keen on right from the start, and I definitely see other people in the field talk about it, is this idea of co-design.

[19:58] Co-design is this general idea where you want to design several aspects of a system together. So clearly, one of the big aspects of co-design is design your software stack and your hardware architecture at the same time. That’s something we’re very keen on at Graphcore, and actually we’ve been, right from the beginning - we were very keen on making sure we worked on how the software was gonna target the chip, make sure we modeled everything, right up to the full neural network applications as we designed the chip, and pushing that all through.

I think later on in the company what happens then is you need to extend that software stack and really invest in building that core for really robust solutions… And now we’ve got more software people than hardware people. It’s a really big part of what we do. We’re as much a software company as a hardware company. But even right at the beginning it was very much “Let’s design these together.”

I don’t think it stops there, actually. You could also say you co-design - and we’ll see more and more of this - the machine learning algorithm architecture itself, along with the software and the hardware, and the system; so the kind of way you – particularly for training, for large-scale, large data machine learning training, which is the world we’re often in at Graphcore… You’re not talking about one process here, you’re talking about hundreds of thousands of processes working together, and they all need to be connected via networking cables, and put into racks, and then the power management, and all this kind of stuff has to be put around it. And really, you need to co-design all of that together. I’ve had someone use the term algotechture before to describe this, and I think “Yeah, why not. That’s a fair enough term.”

Marcin Jakubowski: Exactly. And initially, I’m thinking “Oh yeah, TED talk time, all this amazing interest, and all of that”, but it’s a hard management thing, because to take it from a prototype to the product, it’s like you’ve gotta do not one or two prototypes, you’ve gotta do 10 or 100. I mean, literally, it’s kind of like software, where you fix a bug - done. Okay, next bug. Next bug. But how many bugs do you have? Thousands. Right? It’s the same with hardware. You’ve fixed one thing, you learn a new thing, and then you can keep improving this and improving this, and it takes a long time.

Well, then how do the proprietary guys do that? Well, my answer to that is that their equipment is crap. In general. I mean, it’s pretty high-performance, but it’s nowhere near what it could be if you unleash a collaborative effort to do so. That’s what we’re going after.

John Graham-Cumming: Well, I don’t think they have, not from an optimization perspective… Because actually, what happens is we have a financial model where we can say how much saving CPU time saves us in terms of money… Because we’re growing very rapidly, and if we can not buy new hardware quickly, then that saves us a significant amount of money… So the optimization - you can look at a dollar amount on it.

Now, what’s interesting is we’re optimizing for different things. One of them is CPU utilization, because we’d like to buy less machines. Another one is latency, because we’d like you to have the fastest experience when you use our service, you go to a website. To a certain extent they’re the same thing, but not always; it depends on what the actual workload is on the edge machine. Some things are not involved in the critical path/latency part, but might be using a lot of CPU.

[31:56] So the only real trade-off tends to be “Is it better to build this new feature, or is it better to do an optimization?” That’s probably the trade-off that comes into play… Because there might be some new product you wanna build, that you wanna get out, and there’ll be some predicted revenue around that product, or you want to be first, or you want to beat a competitor, or something… Versus “Well, that same engineering team could save us X million dollars by shaving x% off of this CPU utilization globally.”

Those decisions have to be made, and we have a product management team that helps us make those decisions, about where to spend engineering resources.

One interesting thing that involves Go is we have an internal product which is a sort of load balancer. If you think about the growth of Cloudflare over the last 8-9 years, we’ve got multiple generations of hardware; we have 194 cities where we have hardware, and there are multiple generations, with different performance characteristics… And we wanted to make sure that basically every machine is running at the same CPU utilization within any city. That was not the case if you do naive load balancing, because all the machines can’t handle the load.

A quite typical example would be some machines running at 50% load and some at 75% or 80%, at the same time. So we wrote this coordination layer which is actually measuring the performance of the machines, understanding what they’re capable of, and then real-time directing traffic between the machines… And the coordination of that is Go. That has actually brought us into line. There were quite dramatic graphs in Cloudflare where you can see this mixture of CPU percentages in a column and suddenly you turn on this thing called Unimog and the entire graph flattens out. Every machine is running exactly the same CPU utilization. So there’s many things that Go is being used for.

Evan Sparks: I think there are a couple of key things. One, we’ve got some pretty unique expertise on the team, in this space. These are problems we’ve been thinking about really deeply, both in the academic but also a professional setting for, collectively (the team) dozens of years. And we’ve got a track record of delivering some really popular and influential technology in the space.

The other thing I’d say is I think the cloud vendors are there to build their platforms to help monetize their hardware. The GPUs they’ve invested in, they wanna get people using, and so on. So all of it is – you know, Google pushing Google Cloud, or Amazon pushing their cloud, and so on… Where we differentiate ourselves is by being really neutral to the vendor. We will give people access to the best, cheapest, correct technology for their particular workloads… And you’re already seeing signs of vendors getting custom hardware for these particular tasks.

Google has TPUs now, Microsoft just announced a partnership with Graphcore, and you can bet that Amazon is working on AI-specific hardware… There are gonna be a bigger menu of hardware choices to be available to help you solve these problems down the road, and we think that developers, in the same way they don’t wanna be worrying about the resource management and the calendar system, they definitely don’t wanna be worrying about reprogramming their applications, and figuring out which chip is best for this version of my language model, and so on.

We think that a layer of abstraction from a systems level can offer that kind of flexibility. So you submit your job to us, we figure out what the best hardware to run it on is, we go acquire that for you, your job gets built and run, and then those resources are released - that basic idea I think is something that we can do and we’ll be able to do better than the larger cloud vendors, because we won’t have these exclusive ties to one or the other.

Marty Schoch: I’ve given a talk at the DC meetup that we spoke about earlier, about – in particular with memory management in Go, and how that relates to optimizing application performance… And I think Bleve is maybe somewhat unique in this regard, but if you think about all the stuff we’ve talked about earlier about how this search engine works, one of the things you realize is it’s a lot of just loops within loops within loops. What I mean by the is “Okay, indexing? Well, I’m just gonna loop over all of my documents as the first loop. Now, my documents all have fields; I’ve gotta loop over all those fields… Now, for each term that I find inside of those fields - I’ve gotta loop over that and do some work.” So you have this loop structure. And what the downside is - you write your API in a really clean way the first time.

You just sit down at your editor and you write the API, and this is nice and clean, and then you find out “Oh, it turns out every time I call this function, it has to allocate something to return it.” So your simple API that you just thought up in the head ends up being one that not only performs poorly, but it’s magnified across all these loops. So when you look at your profile, your CPU profile, you see “Oh, I’m spending all of my time allocating memory and handing it back.”

Now, there are techniques… You find a million Go talks on techniques to avoid that. And in my experience, what gets a little frustrating is when I have to, in my mind, clutter the API, I have to add a new argument to my API, so that instead of always allocating something, it could optionally reuse the thing that I’ve passed in. So now my API has gotten cluttered, and it just hurts readability. The thing I love about Go is how readable the code is. I can pull up code I wrote two years ago, and it’s pretty straightforward. The code does what it says; that’s what’s to like about Go. And my concern is the memory management hurts that. It’s code that we write, that ends up – I have to remember “Why did we do it this complex way? Oh, that’s right, it was a performance optimization.”

[55:58] And the pushback is like “Oh, well in so many cases that’s just premature”, but what we find is for some reason in all the code I need it’s not premature optimization. It’s what we have to do to get it to the place where it’s meeting the metrics. So if I have one concern with Go, for me it’s – I don’t mind garbage collection in principle, and I think the improvements to the garbage collector could actually address some of the issues, or some of the mid-stack in-lining, it would solve some other concerns… But it’s just one of those minor gripes, where it’s like “I have to write code in a less clear way in order to get it to perform well”, and that’s not the spirit of Go. The spirit of Go is you just write it in this really clear way, and you just run it on faster hardware. That’s more of the spirit of how Go programmers think about things.

Kyle Daigle: So as we start to explore that as a company - which we talked about a unified business identity and things that help that use case - we’ll have a better place, as well as authorization management to do that. Because that’s where this sort of starts to become difficult; we totally believe that organizations want to say – for example, one of the things that almost every business customer that I’m aware of in GitHub does is every time a new repo is created, takes some sort of action. Sometimes it’s auditing, sometimes is making sure the right people have the right permissions… Any business of medium to large scale does some automated action that way, and of course, we’re looking at actions and going, “Well, obviously, this could make this very easy, because it’ll be auditable, it’ll be always run for them”, but the trick is “Who should have access to this? Where should it live?”, all of those sorts of questions.

So I think we have probably a little bit more of a foundation to lay, to let those use cases be simple still inside of GitHub, and not sort of get into a horrible authorization management structure that all apps that sort of go that way tend to fall into… And still let our customers execute things for every single repo, or every single issue, or every time a user is added, send them a welcome message; any of those sorts of things that currently our customers unfortunately have to do on their own hardware, by building their own things.

Autumn Nash: I also think that when you allow an engineering team to focus on kind of figuring out the first solution, they can kind of do a better job. And when management kind of shields you from doing a million things at once, you can focus, get that out, and then go to the next project, or whatever. But like you said, you want your customers to feel like [unintelligible 00:56:37.11] So it’s cool to hear somebody who is in leadership at like a huge company for so long talk about empathy, and talk about the struggle it is to keep all the people happy, you know what I mean? Because I feel like we don’t have enough empathy in tech, and a lot of it is like “You just have to be like a hardcore developer times 100, and sleep at the office”, and nobody talks about that empathy. And I think we would have such a better culture if more people – like, it makes me happy that you’re CTO now at a startup, because the fact that you talked about that empathy, and also the struggles of wanting to do right by everybody… We need more leaders coming from that place, and talking about that… Because that is rarely the answer when you ask people about what you need in engineering. So I just think that’s really cool.

Isaac Schlueter: Absolutely, absolutely. And it’s actually competitive, because now people who are providing alternative registries, who are competing with us - they have to either seem a little foolish, because they don’t support the whole feature set, or they have to be spending time and energy scrambling to keep up with us, and we’re sort of always fundamentally in the lead, and really taking advantage of our privileged position there.

[56:10] But the other thing about it - there are features that open source users fundamentally don’t need out of their registry service. They don’t need private packages, for instance. They don’t need the single sign-on, and management and policy compliance stuff that you’re gonna see inside of really large enterprises.

There are certainly some arbitrary lines to be drawn. I think GitHub just recently started giving away private repositories for unpaid users, which I think is interesting. They limit the number of repositories you can have. All of their free users get now three private repos, and I can see us doing something similar to that, and continuing to evolve where those lines are drawn, but fundamentally, the needs – there are needs that a large enterprise of 1,500 developers… They absolutely need a way to say “Here are 25 different projects, and I need to see what’s going on in all of them.” Nobody needs to know all of the things that are going on in all of the open source projects. That’s just – you couldn’t, a. And b, it’s not really reasonable to want that.

We can build a lot of really compelling features that are structured in a such a way that we’re… As one example - we haven’t built this is; this is vaporware, a handwave on the roadmap, blah-blah-blah… But for example, whenever you run npm install now, we run an audit of all the packages in your dependency tree. That’s something that benefits every single one of the people using npm today. We give that away for free, and why do we do that? Because one of the things that you need in an enterprise is you need to know how are all of the developers at my company – have any of them installed something that is insecure? Have any of them built an application and shipped it to production that is insecure? Can I go and verify all of the different licenses that are in use in all of our different projects? And these are the kind of things that that same auditing infrastructure enables us to do.

So we’re in a position where giving things away to the open source community actually does serve our long-term product and company needs as we go to market in the enterprise.