William Falcon: Yeah. So I think if you’re working at a company - or any team really, even research - if you’re working with multiple people, you need the ability to share code. And if you’re at a company, or even university lab, you wanna share code across teams. And that’s really hard to do without something like Lightning. Because what happens is people tend to intermingle a lot of stuff, like data, model and hardware into the same files. Well, one team may not have GPUs, or may have different types of GPUs, or may only be using CPUs, or your production requirements mean that you can only use CPUs for inference. So there are a lot of constraints there. And I guess if you’re not thinking about it how we are, from the abstract level, you won’t really realize that a lot of the reasons why a lot of that code doesn’t operate together is because you’re mixing the hardware with the model code. And that’s something that took us four years probably to get there, to see those, to have these insights… And what that means is that we can factor out deep learning code into three major areas; well, at least four, I guess. And we’ll find more; it’s ongoing research. So one is training code - this is anything that has to do with linking your model to the machine specifically; so how do you do the backward paths… You know, backward pass and distributed is very different from just on CPUs… At least technically speaking. What happens if you have half precision there? What happen if you’re using stochastic weight averaging? What happens if you have truncated back steps, right? There are a lot of details that go into it.

So all of that is handled by the trainer. And this is the stuff that you’re gonna do over and over again. It doesn’t matter if you’re doing audio, or speech, or vision, you’re always gonna have a backward pass, you’re always gonna have a training loop, and so on. The model is the thing that changes. The model is not just – I like to think about models… In Lightning we have this concept of a module, and to me a Lightning module is more of a system.

We can think about a model like a convolutional neural network, or a linear regression model. Just like a self-contained module. Today’s models are actually not models. We need a new name, because there’s something that doesn’t exist, and I think the Lightning module, which is a system, because models now interact with each other. Like, what do you call an encoder and a decoder working together to make an auto-encoder or variational encoder. a They’re not models; it’s collections of models interacting together. Same for transformers.

[16:07] So that’s really what the Lightning module is about - you pass these models into it, and then how they interact together is abstracted by that. And I think that’s a missing abstraction that was not there, which is why people were jumping through so many hoops, to be like “Oh, well how do you do GANs? How do you do this other stuff?”

So it’s important to decouple that, because now I have this single file that’s completely self-contained, that I can now share with my team across in a different division, and their problem might be completely different, with a different data set, and they don’t have to ever change the code on that model; all they have to do is change what hardware they’re using and then what the dataset is. As long as it conforms to the API that the model is expecting, it works. So it makes code extremely interoperable.

I think people come to Lightning because they wanna train on multiple GPUs and so on. And under the hood we have this API called Accelerators that lets you do that. But that’s only a very small part of it. I think once you get into it, you see that the rest of it is the ability to collaborate with peers, and be able to have reproducible and scalable code.

Jerod Santo: Hello friends, I’m Jerod and this is Changelog News for the week of Monday, July 4th, 2022. But we’re shipping out on Tuesday instead of Monday, because [Freedom…! 00:00:16.11]

First up, thanks to everyone for sharing your thoughts on this little experiment of ours. We are listening and adapting, so please continue with the feedback. Here’s what’s different this time around. One, almost everyone said to make it longer. [As you wish… 00:00:37.10] This episode is about twice as long as last week’s.

Two - many of you also read Changelog Weekly, and since we’re covering the most interesting links from Sunday’s edition of the newsletter, it makes the audio version less new or interesting. So we are now mixing in stories that we haven’t previously covered. Hopefully, that keeps it fresh for everybody.

Three, many of you think the music bed is distracting, and asked us to turn it lower, or turn it off altogether. So let’s try it without, and see how that sounds. Nice and quiet.

Okay, let’s get into the news.

On DX tips, Swyx writes about the 4.5 kinds of dev tools platforms. This is his attempt to make sense of the overwhelming world of developer tooling. If you work for or invest in dev tool startups, I think this mental model is a good one. What are the 4.5? One, application platforms that make the user productive. Two, infrastructure platforms make the application run. Three, data platforms make the data useful. Four, developer platforms make developers productive. And the point five, internal services with cross-cutting needs to interface with every platform. Those are his high-level conclusions. Read the whole article for his thinking behind the matter.

Did you know there are things you should know about databases? Lots of things. And did you also know there is an excellent - and I mean, excellent post on architecturenotes.co to teach you some of those things. It’s focused mostly on indexes and transactions, explaining them in great detail with some seriously impressive diagrams to make sure it all clicks. Not much else to say about this one, except that it was the number one clicked link in the newsletter, and there’s a link for you to click in the show notes.

Brandon Rhodes wants you to start all of your commands with a comma. Why is that? If you have your own Bin directory somewhere in your path, which you probably should, and you put all your own little scripts and executable bits in there, like you probably should, eventually you’ll hit a problem or your script names start to conflict with the built-in and bundled commands. But if you start all your commands with a comma, like Brandon does, this problem melts away. [You cursed brat! Look what you’ve done! I’m melting, melting. Ohhhhh, what a world, what a world. 00:02:56.22]

As a bonus, team this technique with tab completion, and it’s super-easy to browse your collection of commands. Just type comma, then tab, and there they are.

Lobsters turns ten. Over the weekend, the website’s creator, Joshua Stein, tweeted “Ten years ago today, I “pooped out” a better version of Hacker News in my spare time.” As Thomas Ptacek eloquently put it. It didn’t really mature into the thing that I hoped it would, and I don’t use it much anymore, but I’m happy some people are still finding it a nice place to be.

Over on the site itself, current administrator Peter Bhat Harkins says “Over that time, our community of 15,013 users have submitted 87,530 stories, written 381,460 comments, and cast 2,536,117 votes.” That’s a lot. I am a fan of the site, which usually features more esoteric and engineering-heavy posts than other tech aggregators. The conversations there can still get toxic from time to time, but were on the internet is that not the case? Seriously, if you know of such a place, holler at me. [I want to go to there 00:04:18.09]

[04:20] James Hawkins and his team at PostHog have interviewed 725 people, and one of his big takeaways is “It’s normal for candidates not to ask harder questions about our company, so they usually miss out on a chance to 1) de-risk our company’s performance, and 2) increase the chances they’ll like working here.” So he shares some really important job interview questions engineers should ask, but don’t. Questions like “Does the company have product-market fit? How much runway does the company have? What’s the culture like?” and a whole bunch more. Definitely worth a bookmark for the next time you’re job shopping.

Go Time listeners already know this, but for the rest of you, our recent episode with Ron Evans a.k.a. @deadprogram is just too good to miss, even if Go isn’t your thing. Here’s the conceit - the year is 2053; the tabs versus spaces wars are long over. Ron Evans is the only Go programmer still alive on earth. All he does is maintain old Go code. It’s terrible. He must find a way to warn his fellow gophers before it’s too late. Good thing he finally got that PDQ transmission system working. What results is a hilarious and somehow still insightful conversation with so many funny moments like this one about the future of social media.

“You know, when you’re building something that’s gonna survive a two year trip to Mars, believe me, your mp3s sound pretty funny by the time the ship gets to its destination… Or so I’ve been told. I don’t know, actually those might be AIs sending back those reports. There might even be no humans that survived the trip. There’s a rumor going around they’re all just AIs.” “How’s it going around? Who’s it going around?” “Social media still exists in 2053.” “Oh, thank goodness. I don’t know what you’d do without it.” “I use Minder. You know, it’s where you’re allowed to dump your actual mind directly.” “That’s cool.” “Is it text? Is it visual?” It’s more like a feeling.” “It’s just Hex.” “Remember the feeling you used to get when there was somebody being wrong on the internet? It’s like that all the time.” “Is it XML though?” “No, you just plug directly into your brain-computer interface, and you’re just really mad right away.” “Oh, I love it.” “Yeah, it’s beautiful.”

Listen to the rest in our Go time feed, or on the web at Gotime.fm/235.

Last up for today. Upptime. Upptime. No, I wasn’t stuttering. [Did I stutter? 00:06:43.00] That’s U-p-p-t-i-m-e. That’s how you say that, right,? A super-cool uptime monitor and status page powered entirely by GitHub. It uses GitHub Actions as the uptime monitor itself, checking your website every five minutes. GitHub issues for incident reports, opening new ones when endpoints are down, and assigning team members appropriately, and GitHub Pages for a status website built with Svelte and Sapper. This project is not affiliated with or endorsed by GitHub, but it is super-cool, super-free, and endorsed by over 1,000 people who have set it up for themselves.

That’s the news for now. Let us know in the comments if you liked this better than last week; or worse, or whatever. Seriously, we do want to hear from you. We have an excellent conversation coming up for you on Friday. Brian Cantrell from Oxide Computer joins the show for a deep-dive on their attempt to build servers as they should be - hardware, with the software baked in for running infrastructure at scale. Adam’s on vacation this week, so I have a special guest/co-host joining us as well. Can you guess who? We’ll talk to you then.

Jerod Santo: And people liked it so much that I was like “Okay, they’re gonna like whatever else we do, as long as we’re on point…” And so that’s why I liked it, because it was a prototype for also the other Mat Ryer episodes, which are always my favorite, because the guy just makes me laugh endlessly with his little quips, and his non-sequiturs, and then his songs… So “Git with your friends’, and then the other one that I’ll mention is “Bringing Whisper and LLaMA to the masses.” Talk about people who are dedicated to a particular craft - Georgi Gerganov, the work that he’s doing on that specifically, allowing people to run Whisper and LLaMA and continuing to hack on it on commodity hardware is really I think yeoman’s work, and I think that so many people are gonna benefit by being able to run models on their own commodity hardware, without having to shell out and bend the knee to big tech companies. So I like talking with him. He’s very smart, very humble, and I think that episode was an awesome one, because we got to shine a light on the work he’s doing.

Jack Shanahan: [00:25:47.02] That was quite a landmark paper. Ironically, for those of us who are in industry and maybe you’re not military related at all in the audience, you’re used to Daniel and I always talking about “AI is still part of the software. It’s all bound together, you can’t do AI without the hardware and the software and the systems written together to make it practical AI that’s usable.” And you really went there in software-defined – it’s called Software Defined Warfare Architecting, the DoD’s transition to the digital age… And it was quite a landmark paper for those of us in that industry, because it really laid out the future of how software needs to be integrated. Do you have anything that you wanted to comment on about that, just given – I thought it was very important to anybody concerned with AI, in the DOD.

Daniel Whitenack: Yeah, I think that there’s a few elements of this… I think it has been interesting - the last episode that we had, we talked about these open rail licenses, and one is utilized by Stable Diffusion. So there is some explicit things you have to agree to when downloading the model. On Hugging Face, for example, you have to click a button that says “I agree to this stuff”, and then you can download it, and you have to use your Hugging Face token to download it… But it is open in that sense, in a sort of unique way.

But I think that if we look at models like this and ones that are released open source, I think you kind of saw in software, over time, as it was open sourced, a lot of software applications or kind of specialized software things going from kind of specialized expert groups using them to a general-purpose technology that was used and integrated into a whole variety of things that the original creators didn’t even have in mind, right? So I think we’re in a similar place here, where we’re going from maybe models that were being experimented with in sort of siloed places… But now, as you were mentioning, there’s all sorts of ways you could imagine using this model. And because I can access it, and because I can run it without expensive hardware, and because there’s good tooling like the Diffusers library, which I can pull in and do this in eight lines of code, then who knows how people will use this and sort of hack it, in a good way. So hacking it for useful, kind of pragmatic purposes.

Erik Dunteman: [07:57] What is publicly known - you’ve got to think about getting… Well, firstly - constraint. Constraint - you cannot take up GPU RAM. If you have a 40-gigabyte A100 machine - if you put a model into that RAM, that portion of the RAM, or like that machine entirely, if you’re not virtualizing it, it’s just like taken. You paid for it, it is dead space; if you’re not using it, that’s massive GPU burn without any utilization. So a constraint model can’t sit in RAM, at least GPU RAM.

So when we go about the cold boot problem, what we’re really thinking about is, “How do we get the models, specifically the weights, as close to RAM as possible, without actually occupying resources, or more precious compute resources, like 40 gigs of limited RAM?” That’s hard. But if you have a terabyte of storage on the machine, you could at least have local caching the model. So you could take that up passively between calls, without sacrificing that piece of hardware, because you could fit so many more models onto the disk.

Jay LaCroix: Yeah. I mean, I think there’s nothing wrong with that. We have something like that with System76, although it’s not quite the same thing, and Tuxedo, and a number of others. I mean, we kind of have that… But there’s going to be this mentality that comes, and this is another issue that happens often, is that people will try Linux on computers that they have no business trying Linux on, and then they’ll hate Linux for its inability to work on the hardware that it was never meant to work on in the first place. And I feel like this myth comes from the Linux community itself, and I feel like there’s good intentions here.

I mean, if I like a movie, I’m probably going to tell you about it. “You should check this movie out. It’s really cool.” As human beings, we love to recommend the things to other people that resonate with us, because it might build rapport with other people. So you can imagine somebody tries Linux for the first time, they love it, they think it’s the greatest thing ever. Then they want to recommend it to all their friends. They might say “It runs on everything. Just install it, it’s great.” And then the person installs it without checking compatibility, and nothing works. And then we’ve lost them forever. They’re never going to try Linux again, because we recommended them to install it without checking first is their hardware compatible, before they actually install it? And that also creates negative feedback. Even though I say, “Linux has really good hardware support”, no operating system has 100% hardware support, and Linux is no question.

[38:13] So if hardware is not built for Linux, or has drivers for Linux - well, guess what? You’re going to have a bad experience. And what’s really strange is we have this feature that’s amazing called Live Mode; you could boot your distribution in live mode from the USB, and demo it first… And you’ll know if your Wi-Fi card is detected, right then and there. You could go on YouTube, play some music videos or something, you know your audio is working… But everyone seems to avoid using it, and then they’ll post messages… “I installed Ubuntu, but nothing works.” And I’m thinking, “Did anyone tell you about Live Mode, that you could have demoed this first? And why did you replace your operating system before you actually verify compatibility?” Maybe they’re just that eager to try it out and just excited, and they impulsively just wipe their machine, or something… But people do it, apparently.

Bill Dally: [41:50] I think it’s just a very exciting time to be working in AI, because there are so many new developments happening every day. It’s never a dull place. In fact, there’s so much stuff happening that it’s hard to keep up. As a hardware engineer, I think it’s also very rewarding to know that this whole revolution in deep learning has been enabled by hardware. All of the algorithms - convolutional nets, multilayer perceptrons, training them using stochastic gradient descent and backpropagation… All of that has been around since the 1980’s, since I first started playing with neural networks, but it wasn’t until we had GPUs that it was really practical. GPUs basically were the spark that ignited the revolution.

The three ingredients were the algorithms, the large datasets - those were both there, but then you needed the GPUs to make it work. For computer vision it wasn’t until AlexNet in 2012, where using GPUs he was able to train the network to win the ImageNet competition, that deep learning really took off.

So I think GPU’s are what ignited this, and I think GPU’s are still really the platform of choice, because with the Tensor Cores they provide the efficiency of special purpose units, but without the inflexibility of a hardware ASIC like a TPU, so you get the best of both worlds. You can program in CUDA, but get the efficiency of a Tensor Core.

Brian Ketelsen: Yeah, those are things that I want to mock immediately. So I built all the interfaces first, and I’m building mocks now for all of them; even without having all of the hardware, I’ll be able to prove most of the application logic is good before we get it put together.

Steven Wu: …building a product. Yeah products get better; our product will get better, their product will get better… Maybe in 10 years we’ll be forced off by market monopoly pressure when AWS decides to release exactly our product, but better, like they always do… I think that’s just a threat that pretty much every cloud provider has, which is that “Yeah, why doesn’t the cloud provider just use their hardware to do your thing, but without your margins?” I think that’s very reasonable.

Gerhard Lazu: That’s a nice one. By the way, you’re saving the company, but what is not said, that you’re also saving users money. Because you, running, will be paying less, regardless where you run, by the way. Even if it’s not AWS, by using a language which is very nicely optimized, latency is better, memory is better, you’re paying for less, you can do more with fewer hardware, fewer resources… Who would want that, without compromising on latency? And there’s something even more important; apparently, you will love it. [laughs] That’s what Stack Overflow survey says. Right?

George Neville-Neil: I would like to see Rust proven to be better than way. I would be very happy to trade in C for a safer C. The reason C never gets supplanted isn’t the safety issue… Because it turns out that most people who are producing products for you don’t care about your safety… But don’t tell them that. What matters is performance. So no one has come up with something other than hand-coded assembly, which is portable and as performant as C. Now, Rust is getting very close.

So if Rust can keep the safety qualities and be as fast as C, then yes, we should chuck as much C out as possible… And all the people who work with me on the FreeBSD Project right now are sharpening knives, and once the pandemic is over, they’re gonna come to my apartment and cut me into little ribbons. [laughter]

But yeah, I definitely think that C is… There are projects in the world right now… There’s the cherry work that’s been done at the University of Cambridge - Robert Watson, who I’ve worked with on various things - trying to change hardware, so that hardware is safer, and we can actually use pointers without foot shooting. And a change in hardware is gonna take a while, so we’ll see how that works. I think it’s very promising.

Stevenson Jean-Pierre: [56:40] I think a lot of modern developers end up doing a lot more glue work and stitching work than just straight up development… Because traditionally, there were systems that you would have to write yourself, but because they’re being abstracted and they’re being written for you and they become kind if provider-driven, you’re doing more stitching work nowadays, you’re doing more glue work.

And running infrastructure just to do glue work is kind of demoralizing, and you kind of have to maintain these things… But I’ve really found that’s the sweet spot for me with serverless - being able to write all these integrations, write all this glue work, but have that infrastructure also be that thing that’s abstracted away, so that these systems flow as if it’s a pure vendor solution, without having to run your own underlying hardware or your own underlying instances, and things like that. I really think that’s what makes it worth it. If you think about your own workload, you’ll find that you’re writing a lot of glue layers for things, integration layers and glue layers… So I think that’s definitely a good reason to learn it.

Also, I think it helps you practice stateless programming, and making sure you’re building these distributed applications without having to purely get down into the nitty-gritty of building distributed systems, and things like that. So it’s a good epic entrypoint to understanding how these things start working together to form wider systems that are achieving a common goal.

Anna Bethke: For sure. I think one of the things that’s been the most beneficial of this last nine month or something period is to really start to see how to do that. Coming from an engineering background, I didn’t really have a lot of business classes, I didn’t really have a lot of marketing classes… But one of the things that I have been doing a lot is talking about these projects, both internally and externally, and showing a few different things.

So when I pitched it, I didn’t give any business objectives, or any metrics, or things like that, and now I’m starting to put those together. A lot of what we’re seeing though - it sort of helps the business in a few different ways. One is marketing. Talking about these really socially beneficial projects gives you all the warm feels, and they’re lovely to talk about, they’re really interesting as well, so that’s one thing.

The other is hiring and retention. A lot of the workforce today really just wanna work on these projects that are impactful. Recommender systems are great, or figuring out the sentiment of Twitter - also great. These projects have a place in things, but a lot of the workforce wanna do something that is more impactful. So instead of looking at any social media (we’ll just anonymize the social media source) or online source for sentiment or categorization, instead looking at it to figure out what is harassing text or not, or to try to figure out what types of information do kids globally have access to. That’s the things that we really wanna be working on.

The third actually is really relevant also to our hardware. At Intel we sell a bunch of hardware, that’s our bread and butter, and without being able to do these types of projects we wouldn’t see the entire range of use cases. We’ve done a bunch of different medical types of projects; one of them is using very large 3D images and trying to figure out where tumors are, so basically revolutionizing the healthcare industry.

The issue with these datasets though is that the images are so large, so it takes a large amount of memory to put them in your compute power. And there’s something called tiling which exists, so if you can’t fit it all in memory, you can chunk it up… But that doesn’t do very well if you’re doing a segmentation type of deep learning, where you’re trying to show an entire area. So you wanna keep your image whole. That really helps us then be able to make certain that our hardware is designed in a way that supports this dataset. If we were all just looking at ImageNet, then it’s like these tiny, tiny images… And that has an area and it has a place, but we wanna see the breadth of what is out there. And that’s just one example. A lot of these other datasets are also very large, very messy, so creating the tools to support those…

Jerod Santo: That’s where I’m trying to get to in my mind, because I’m starting to think of adoption, you know? Like, how do we get more people who want to loop to be able to? Like you said before, a lot of it is, well, they’ve gotta switch pumps, or something. A lot of it is cost, or timing around healthcare and blah-blah-blah that holds them back, and it seems like, well, if we had interfaces into more of these devices and we had a layer of abstraction where this product that you’re thinking about had a layer where it could talk to not just this specific pump that you’re coding against now, but could work against these seven, which cover 90% of the market, or something like that, you open it up to a lot more people without having to have them own specific hardware.

Adam Argyle: I’d say “hands”, but… Is that hardware? [laughter] And I know people can code without their hands, right? You’re just like “Nah, I don’t need hands to code. I’ll code with my voice.” Code in VR with your elbows, or something; I don’t know. I am gonna say “hands.” Fingers and hands.

Jerod Santo: Yeah, it’s all the same thing. You just add a “qu” in front of it, that’s my take-away. One other point to add to that, the Y2K analogy… More on the opportunity front - the people who really dove into… I’m looking at this on a parallel with machine learning; I just can’t let go of it. The people taht dove into that five years ago, seven years ago, ten years ago, are making very good money right now. And I think the people that are going to dive into quantum computing and able to implement things, and know about things, and leverage this coming technology I think will have opportunity at very good money; so if money motivates you, there’s some motivation as well.

[01:04:14.03] Johan, I’m curious if there’s a place where your average developer like myself could keep up with the state of quantum computing - kind of like see new things coming out, new algorithms - without being overwhelmed with the maths and the hardware folk… Is there any approachable waypoints in the community? Is there a forum, is there an RSS feed I could subscribe to just to stay abreast of what’s going on and developing in the quantum computing community?

Gerhard Lazu: I think you want the purity, right? You want the purity of hardware. So keep that experience as pure as possible, without adding any of your daemons, or any of your agents, or whatever you wanna call them… So keep it as pure and pristine as possible from what you would get if you were to physically provision it in a rack… But then make it easy for people to add whatever they want in the best possible way, including Kubernetes. So what is the best way of providing Kubernetes on top of bare metal? That’s where you come in; specifically you, David. Or at least that’s my understanding.

Yonatan Geifman: [24:02] Yeah, so you got it right. Let’s formulate the equation that the neural architecture search is trying to solve. We are talking about minimize the latency of the model on a given hardware, subject to getting an accuracy that is above a given threshold. So the latency of the architecture or the model can be measured without training in most of the cases on the hardware. So we don’t need the data to understand what is going to be the latency of ResNet-50 on a CPU of Intel. But the accuracy is data-dependent. And if we want to put that constraint - and obviously, we want to put that - we need to have the data and verify that the model that is elected by the minimization problem of the latency still meets the accuracy requirements. Because [unintelligible 00:24:47.05] the accuracy constraint, we’ll end up with a model with one neuron that’s prediction nothing. So this is kind of the composition between the latency that is measured on the hardware and the accuracy that is measured on the data.

Lukas Biewald: Yeah, I’ll give you an example. Fundamentally, the way this works is you import a library, and then a library basically collects lots of system metrics and other metrics as it runs. Kind of similar to TensorBoard, but we collect a lot more stuff. I’ll give you an example of the passiveness of it. One thing you should probably do before a big training run is commit your code into Git. So it could just connect each training run you do to the commit SHA, and we do do that if you want us to.

But one thing we’ve found - because we’re talking to people all the time - is most of the time people don’t actually commit that code before it runs. So we capture not just the commit SHA, but also a diff against the latest commit SHA. So that way, every time you do a training run, we keep track for you the exact state of the code, the way it was when it trained. That way, a month later you wanna go back and look at something and be absolutely sure what the state of the code was; you can do that, but you never need to think about it.

Once you turn that on one time, now you’ll be sure that every single time your model trains, there’s a snapshot of exactly the state of the code. And you might say “Well, that’s a waste of space”, but come on, if you look at the size of training datasets, they’re gonna be like petabytes. So you could do this all day long, capture every metric you might care about, all the state of the code, everything going on; you know exactly what kind of hardware it was on… You can capture all this without really making a dent into a modern machine’s storage.

Bill Kennedy: You know, my interest in this actually started to develop when I came into Go. Because before that I was in C# and we had lists, we had keys, we had stacks, we had data structures, right? And even C++ gave us all these data structures. And when I came into Go, I was like, “Where are all my data structures? I don’t understand this.” I’d just see an array, I’d see a slice which I honestly didn’t understand at that time, and I’d see maps. And it’s really silly, because I didn’t really understand what slices were. I just thought that they were really just arrays, and back in school we were really taught that arrays are difficult to work with. And I actually avoided slices for the first couple of months working in Go using linked lists, because I honestly didn’t understand why we didn’t have data structures, and eventually at some point I realized that everybody is using slices and the language is pushing you towards slices, and I figured out I had to really learn what this is.

Now when you step back and you look at it from this point of view, the underlying data structure for the slice is an array, right? The slice is the most important data structure in Go. And as I peel this onion every month, more and more about Go, all I keep seeing is how Go is pushing us towards writing sympathetic code. Go is pushing us towards doing the right things without anybody realizing it. Go wants us to work with these slices because then we’re really working with arrays and contiguous memory, and it’s giving us our best opportunity to have these sympathies without even realizing that we’re being sympathetic with the hardware. So Go to me is just an incredibly fascinating language when it comes to that.

And other areas of the language too, where you see that you’re really being sympathetic with the operating system scheduler on the concurrency side, without even realizing it. Just these idioms and these things that we tell people to do all the time, they’re based not on just, “Hey, we want you to do this”, they’re based on real things around performance, simplicity, readability, those types of things. It all kind of comes full circle.

Josh Aas: So what I’ve just described is the basic process of getting trusted from scratch. And that does require a big commitment. It requires quite a bit of money to get set up to a point where you can pass audits and even apply, and then every year while you’re waiting for all this stuff to happen for 6-10 years you have to stay compliant, get audited every year. So you’re talking millions of dollars and 6-10 years before you can even be a publicly-trusted CA in any meaningful sense. That’s the basic process.

There is a way to make a shortcut, which is how Let’s Encrypt was able to start without waiting 6-10 years first. So we did go through the process that I’ve just described, building up our own root of trust from scratch… But the world right now does not really rely on that yet, because it has not been long enough.

[31:52] And somewhere around mid-to-late next year we’re gonna switch over to our own root that’s trusted from scratch. But from our inception through now, we have what’s called a cross-signature… Which means we knew we didn’t wanna wait 6-10 years to start offering Let’s Encrypt services, so we’ve found another CA that understood what we were trying to do and was willing to help. They had a root of trust that was already trusted.

What essentially happens is we create a contract, an agreement between us, and then their root of trust essentially lends its credibility to us. So they issued a certificate that our root is trusted by their root, and their root is trusted by the browsers, basically. So that’s called a cross-signature. We acquired that before we did much of anything… Because without that agreement in place, there’s really no CA.

So one of the first things we did was get that agreement, because without that agreement in place, there’s no point in buying hardware and doing anything else… Because –

Adam Stacoviak: I believe they’ll let you use it to a certain capacity for free. You can’t change the desktop background, you can’t do certain things… But it will be a machine I believe that can run in perpetuity without any restrictions. There’s like certain things you’d want to do you can’t do without getting a license key.

And so I really wish – like, that would be… Maybe they would lose tons of money, but that would be cool. Like, make Windows free, like macOS is free. macOS is essentially free to anyone who can run the hardware.

Kurt Mackey: Well, so there’s the startup tension of sometimes if you know too much about a problem, you talk yourself out of doing it. Like, running managed databases is a giant pain in the ass. In the same way running global hardware infrastructure is a giant pain in the ass. But the difference between managed databases and global hardware infrastructure is I’d done one of those things before and realized how much of a pain in the ass it was. And I wasn’t like gung-ho, naively jumping into a problem without realizing the downsides. And I think one of the problems with – it’s always interesting to me that people start startups… I don’t know if this is true, and it’s probably not, but I’m really aware of people who start startups, they have no real credentials for it. Not credentials, but like they don’t have the right experience to build this startup. And in some ways that’s an asset, because they’re so naive that they don’t even notice what they can’t do. And so a lot of times they end up finding stuff that seems impossible to people who know better. So for me, managing databases is a really known, complicated problem.

And there were two things at play for us in particular. One is I’m like hyper-focused on like doing something novel, which you might’ve noticed when I was talking about people that come and sign up and just wanna run their apps, how I’ve had to really let go of not trying to do something novel for them, trying to just give them a good experience based on what they already want. But I’m hyper-focused on doing something novel, and I’m very skeptical of doing something everyone else is doing, that’s really complicated and hard. And so databases fell in that for me. It’s like “I know how hard it is to run databases. I also wanna do something – I wanna put all of our energy into doing something that’s really meaningfully different than what already exists out there…” And so I skewed way hard to “Let’s do the meaningfully different than that exists out there.” Which in some ways wasn’t wrong, because we got this machines API three years before people needed to realize they needed to run like LLM-generated code in a safe environment. We didn’t know LLMs would even be writing code at that point, but it was a thing that we kind of saw the need for vaguely, and needed for ourselves and managed to ship.

But I think that I over-fixed it on how hard managed databases were… And then a component of that is – and so I know what it takes to run a managed database, and it’s really just a lot of people who know how to operate that database, and a lot of tooling that lets them know when to go look at a customer’s problem, and do something about it. And so that’s kind of why we opted for Percona, because they’re really good at running people’s Postgres. So we know that within 15 minutes of us noticing an issue with the customer database, we can get a hold of Percona if we want, and have them looking at it with us, which is the key, I think, to a managed database service from day one, is just fixing problems so people don’t have to. And it’s almost like a human element. It’s almost like a human and support problem, and less of a product problem, in some ways.