Jerod Santo: And people liked it so much that I was like “Okay, they’re gonna like whatever else we do, as long as we’re on point…” And so that’s why I liked it, because it was a prototype for also the other Mat Ryer episodes, which are always my favorite, because the guy just makes me laugh endlessly with his little quips, and his non-sequiturs, and then his songs… So “Git with your friends’, and then the other one that I’ll mention is “Bringing Whisper and LLaMA to the masses.” Talk about people who are dedicated to a particular craft - Georgi Gerganov, the work that he’s doing on that specifically, allowing people to run Whisper and LLaMA and continuing to hack on it on commodity hardware is really I think yeoman’s work, and I think that so many people are gonna benefit by being able to run models on their own commodity hardware, without having to shell out and bend the knee to big tech companies. So I like talking with him. He’s very smart, very humble, and I think that episode was an awesome one, because we got to shine a light on the work he’s doing.

Jack Shanahan: [00:25:47.02] That was quite a landmark paper. Ironically, for those of us who are in industry and maybe you’re not military related at all in the audience, you’re used to Daniel and I always talking about “AI is still part of the software. It’s all bound together, you can’t do AI without the hardware and the software and the systems written together to make it practical AI that’s usable.” And you really went there in software-defined – it’s called Software Defined Warfare Architecting, the DoD’s transition to the digital age… And it was quite a landmark paper for those of us in that industry, because it really laid out the future of how software needs to be integrated. Do you have anything that you wanted to comment on about that, just given – I thought it was very important to anybody concerned with AI, in the DOD.

Daniel Whitenack: Yeah, I think that there’s a few elements of this… I think it has been interesting - the last episode that we had, we talked about these open rail licenses, and one is utilized by Stable Diffusion. So there is some explicit things you have to agree to when downloading the model. On Hugging Face, for example, you have to click a button that says “I agree to this stuff”, and then you can download it, and you have to use your Hugging Face token to download it… But it is open in that sense, in a sort of unique way.

But I think that if we look at models like this and ones that are released open source, I think you kind of saw in software, over time, as it was open sourced, a lot of software applications or kind of specialized software things going from kind of specialized expert groups using them to a general-purpose technology that was used and integrated into a whole variety of things that the original creators didn’t even have in mind, right? So I think we’re in a similar place here, where we’re going from maybe models that were being experimented with in sort of siloed places… But now, as you were mentioning, there’s all sorts of ways you could imagine using this model. And because I can access it, and because I can run it without expensive hardware, and because there’s good tooling like the Diffusers library, which I can pull in and do this in eight lines of code, then who knows how people will use this and sort of hack it, in a good way. So hacking it for useful, kind of pragmatic purposes.

Erik Dunteman: [07:57] What is publicly known - you’ve got to think about getting… Well, firstly - constraint. Constraint - you cannot take up GPU RAM. If you have a 40-gigabyte A100 machine - if you put a model into that RAM, that portion of the RAM, or like that machine entirely, if you’re not virtualizing it, it’s just like taken. You paid for it, it is dead space; if you’re not using it, that’s massive GPU burn without any utilization. So a constraint model can’t sit in RAM, at least GPU RAM.

So when we go about the cold boot problem, what we’re really thinking about is, “How do we get the models, specifically the weights, as close to RAM as possible, without actually occupying resources, or more precious compute resources, like 40 gigs of limited RAM?” That’s hard. But if you have a terabyte of storage on the machine, you could at least have local caching the model. So you could take that up passively between calls, without sacrificing that piece of hardware, because you could fit so many more models onto the disk.

Jay LaCroix: Yeah. I mean, I think there’s nothing wrong with that. We have something like that with System76, although it’s not quite the same thing, and Tuxedo, and a number of others. I mean, we kind of have that… But there’s going to be this mentality that comes, and this is another issue that happens often, is that people will try Linux on computers that they have no business trying Linux on, and then they’ll hate Linux for its inability to work on the hardware that it was never meant to work on in the first place. And I feel like this myth comes from the Linux community itself, and I feel like there’s good intentions here.

I mean, if I like a movie, I’m probably going to tell you about it. “You should check this movie out. It’s really cool.” As human beings, we love to recommend the things to other people that resonate with us, because it might build rapport with other people. So you can imagine somebody tries Linux for the first time, they love it, they think it’s the greatest thing ever. Then they want to recommend it to all their friends. They might say “It runs on everything. Just install it, it’s great.” And then the person installs it without checking compatibility, and nothing works. And then we’ve lost them forever. They’re never going to try Linux again, because we recommended them to install it without checking first is their hardware compatible, before they actually install it? And that also creates negative feedback. Even though I say, “Linux has really good hardware support”, no operating system has 100% hardware support, and Linux is no question.

[38:13] So if hardware is not built for Linux, or has drivers for Linux - well, guess what? You’re going to have a bad experience. And what’s really strange is we have this feature that’s amazing called Live Mode; you could boot your distribution in live mode from the USB, and demo it first… And you’ll know if your Wi-Fi card is detected, right then and there. You could go on YouTube, play some music videos or something, you know your audio is working… But everyone seems to avoid using it, and then they’ll post messages… “I installed Ubuntu, but nothing works.” And I’m thinking, “Did anyone tell you about Live Mode, that you could have demoed this first? And why did you replace your operating system before you actually verify compatibility?” Maybe they’re just that eager to try it out and just excited, and they impulsively just wipe their machine, or something… But people do it, apparently.

Bill Dally: [41:50] I think it’s just a very exciting time to be working in AI, because there are so many new developments happening every day. It’s never a dull place. In fact, there’s so much stuff happening that it’s hard to keep up. As a hardware engineer, I think it’s also very rewarding to know that this whole revolution in deep learning has been enabled by hardware. All of the algorithms - convolutional nets, multilayer perceptrons, training them using stochastic gradient descent and backpropagation… All of that has been around since the 1980’s, since I first started playing with neural networks, but it wasn’t until we had GPUs that it was really practical. GPUs basically were the spark that ignited the revolution.

The three ingredients were the algorithms, the large datasets - those were both there, but then you needed the GPUs to make it work. For computer vision it wasn’t until AlexNet in 2012, where using GPUs he was able to train the network to win the ImageNet competition, that deep learning really took off.

So I think GPU’s are what ignited this, and I think GPU’s are still really the platform of choice, because with the Tensor Cores they provide the efficiency of special purpose units, but without the inflexibility of a hardware ASIC like a TPU, so you get the best of both worlds. You can program in CUDA, but get the efficiency of a Tensor Core.

Brian Ketelsen: Yeah, those are things that I want to mock immediately. So I built all the interfaces first, and I’m building mocks now for all of them; even without having all of the hardware, I’ll be able to prove most of the application logic is good before we get it put together.

Steven Wu: …building a product. Yeah products get better; our product will get better, their product will get better… Maybe in 10 years we’ll be forced off by market monopoly pressure when AWS decides to release exactly our product, but better, like they always do… I think that’s just a threat that pretty much every cloud provider has, which is that “Yeah, why doesn’t the cloud provider just use their hardware to do your thing, but without your margins?” I think that’s very reasonable.

Gerhard Lazu: That’s a nice one. By the way, you’re saving the company, but what is not said, that you’re also saving users money. Because you, running, will be paying less, regardless where you run, by the way. Even if it’s not AWS, by using a language which is very nicely optimized, latency is better, memory is better, you’re paying for less, you can do more with fewer hardware, fewer resources… Who would want that, without compromising on latency? And there’s something even more important; apparently, you will love it. [laughs] That’s what Stack Overflow survey says. Right?

George Neville-Neil: I would like to see Rust proven to be better than way. I would be very happy to trade in C for a safer C. The reason C never gets supplanted isn’t the safety issue… Because it turns out that most people who are producing products for you don’t care about your safety… But don’t tell them that. What matters is performance. So no one has come up with something other than hand-coded assembly, which is portable and as performant as C. Now, Rust is getting very close.

So if Rust can keep the safety qualities and be as fast as C, then yes, we should chuck as much C out as possible… And all the people who work with me on the FreeBSD Project right now are sharpening knives, and once the pandemic is over, they’re gonna come to my apartment and cut me into little ribbons. [laughter]

But yeah, I definitely think that C is… There are projects in the world right now… There’s the cherry work that’s been done at the University of Cambridge - Robert Watson, who I’ve worked with on various things - trying to change hardware, so that hardware is safer, and we can actually use pointers without foot shooting. And a change in hardware is gonna take a while, so we’ll see how that works. I think it’s very promising.

Stevenson Jean-Pierre: [56:40] I think a lot of modern developers end up doing a lot more glue work and stitching work than just straight up development… Because traditionally, there were systems that you would have to write yourself, but because they’re being abstracted and they’re being written for you and they become kind if provider-driven, you’re doing more stitching work nowadays, you’re doing more glue work.

And running infrastructure just to do glue work is kind of demoralizing, and you kind of have to maintain these things… But I’ve really found that’s the sweet spot for me with serverless - being able to write all these integrations, write all this glue work, but have that infrastructure also be that thing that’s abstracted away, so that these systems flow as if it’s a pure vendor solution, without having to run your own underlying hardware or your own underlying instances, and things like that. I really think that’s what makes it worth it. If you think about your own workload, you’ll find that you’re writing a lot of glue layers for things, integration layers and glue layers… So I think that’s definitely a good reason to learn it.

Also, I think it helps you practice stateless programming, and making sure you’re building these distributed applications without having to purely get down into the nitty-gritty of building distributed systems, and things like that. So it’s a good epic entrypoint to understanding how these things start working together to form wider systems that are achieving a common goal.

Anna Bethke: For sure. I think one of the things that’s been the most beneficial of this last nine month or something period is to really start to see how to do that. Coming from an engineering background, I didn’t really have a lot of business classes, I didn’t really have a lot of marketing classes… But one of the things that I have been doing a lot is talking about these projects, both internally and externally, and showing a few different things.

So when I pitched it, I didn’t give any business objectives, or any metrics, or things like that, and now I’m starting to put those together. A lot of what we’re seeing though - it sort of helps the business in a few different ways. One is marketing. Talking about these really socially beneficial projects gives you all the warm feels, and they’re lovely to talk about, they’re really interesting as well, so that’s one thing.

The other is hiring and retention. A lot of the workforce today really just wanna work on these projects that are impactful. Recommender systems are great, or figuring out the sentiment of Twitter - also great. These projects have a place in things, but a lot of the workforce wanna do something that is more impactful. So instead of looking at any social media (we’ll just anonymize the social media source) or online source for sentiment or categorization, instead looking at it to figure out what is harassing text or not, or to try to figure out what types of information do kids globally have access to. That’s the things that we really wanna be working on.

The third actually is really relevant also to our hardware. At Intel we sell a bunch of hardware, that’s our bread and butter, and without being able to do these types of projects we wouldn’t see the entire range of use cases. We’ve done a bunch of different medical types of projects; one of them is using very large 3D images and trying to figure out where tumors are, so basically revolutionizing the healthcare industry.

The issue with these datasets though is that the images are so large, so it takes a large amount of memory to put them in your compute power. And there’s something called tiling which exists, so if you can’t fit it all in memory, you can chunk it up… But that doesn’t do very well if you’re doing a segmentation type of deep learning, where you’re trying to show an entire area. So you wanna keep your image whole. That really helps us then be able to make certain that our hardware is designed in a way that supports this dataset. If we were all just looking at ImageNet, then it’s like these tiny, tiny images… And that has an area and it has a place, but we wanna see the breadth of what is out there. And that’s just one example. A lot of these other datasets are also very large, very messy, so creating the tools to support those…

Jerod Santo: That’s where I’m trying to get to in my mind, because I’m starting to think of adoption, you know? Like, how do we get more people who want to loop to be able to? Like you said before, a lot of it is, well, they’ve gotta switch pumps, or something. A lot of it is cost, or timing around healthcare and blah-blah-blah that holds them back, and it seems like, well, if we had interfaces into more of these devices and we had a layer of abstraction where this product that you’re thinking about had a layer where it could talk to not just this specific pump that you’re coding against now, but could work against these seven, which cover 90% of the market, or something like that, you open it up to a lot more people without having to have them own specific hardware.

Adam Argyle: I’d say “hands”, but… Is that hardware? [laughter] And I know people can code without their hands, right? You’re just like “Nah, I don’t need hands to code. I’ll code with my voice.” Code in VR with your elbows, or something; I don’t know. I am gonna say “hands.” Fingers and hands.

Jerod Santo: Yeah, it’s all the same thing. You just add a “qu” in front of it, that’s my take-away. One other point to add to that, the Y2K analogy… More on the opportunity front - the people who really dove into… I’m looking at this on a parallel with machine learning; I just can’t let go of it. The people taht dove into that five years ago, seven years ago, ten years ago, are making very good money right now. And I think the people that are going to dive into quantum computing and able to implement things, and know about things, and leverage this coming technology I think will have opportunity at very good money; so if money motivates you, there’s some motivation as well.

[01:04:14.03] Johan, I’m curious if there’s a place where your average developer like myself could keep up with the state of quantum computing - kind of like see new things coming out, new algorithms - without being overwhelmed with the maths and the hardware folk… Is there any approachable waypoints in the community? Is there a forum, is there an RSS feed I could subscribe to just to stay abreast of what’s going on and developing in the quantum computing community?

Gerhard Lazu: I think you want the purity, right? You want the purity of hardware. So keep that experience as pure as possible, without adding any of your daemons, or any of your agents, or whatever you wanna call them… So keep it as pure and pristine as possible from what you would get if you were to physically provision it in a rack… But then make it easy for people to add whatever they want in the best possible way, including Kubernetes. So what is the best way of providing Kubernetes on top of bare metal? That’s where you come in; specifically you, David. Or at least that’s my understanding.

Yonatan Geifman: [24:02] Yeah, so you got it right. Let’s formulate the equation that the neural architecture search is trying to solve. We are talking about minimize the latency of the model on a given hardware, subject to getting an accuracy that is above a given threshold. So the latency of the architecture or the model can be measured without training in most of the cases on the hardware. So we don’t need the data to understand what is going to be the latency of ResNet-50 on a CPU of Intel. But the accuracy is data-dependent. And if we want to put that constraint - and obviously, we want to put that - we need to have the data and verify that the model that is elected by the minimization problem of the latency still meets the accuracy requirements. Because [unintelligible 00:24:47.05] the accuracy constraint, we’ll end up with a model with one neuron that’s prediction nothing. So this is kind of the composition between the latency that is measured on the hardware and the accuracy that is measured on the data.

Lukas Biewald: Yeah, I’ll give you an example. Fundamentally, the way this works is you import a library, and then a library basically collects lots of system metrics and other metrics as it runs. Kind of similar to TensorBoard, but we collect a lot more stuff. I’ll give you an example of the passiveness of it. One thing you should probably do before a big training run is commit your code into Git. So it could just connect each training run you do to the commit SHA, and we do do that if you want us to.

But one thing we’ve found - because we’re talking to people all the time - is most of the time people don’t actually commit that code before it runs. So we capture not just the commit SHA, but also a diff against the latest commit SHA. So that way, every time you do a training run, we keep track for you the exact state of the code, the way it was when it trained. That way, a month later you wanna go back and look at something and be absolutely sure what the state of the code was; you can do that, but you never need to think about it.

Once you turn that on one time, now you’ll be sure that every single time your model trains, there’s a snapshot of exactly the state of the code. And you might say “Well, that’s a waste of space”, but come on, if you look at the size of training datasets, they’re gonna be like petabytes. So you could do this all day long, capture every metric you might care about, all the state of the code, everything going on; you know exactly what kind of hardware it was on… You can capture all this without really making a dent into a modern machine’s storage.

Bill Kennedy: You know, my interest in this actually started to develop when I came into Go. Because before that I was in C# and we had lists, we had keys, we had stacks, we had data structures, right? And even C++ gave us all these data structures. And when I came into Go, I was like, “Where are all my data structures? I don’t understand this.” I’d just see an array, I’d see a slice which I honestly didn’t understand at that time, and I’d see maps. And it’s really silly, because I didn’t really understand what slices were. I just thought that they were really just arrays, and back in school we were really taught that arrays are difficult to work with. And I actually avoided slices for the first couple of months working in Go using linked lists, because I honestly didn’t understand why we didn’t have data structures, and eventually at some point I realized that everybody is using slices and the language is pushing you towards slices, and I figured out I had to really learn what this is.

Now when you step back and you look at it from this point of view, the underlying data structure for the slice is an array, right? The slice is the most important data structure in Go. And as I peel this onion every month, more and more about Go, all I keep seeing is how Go is pushing us towards writing sympathetic code. Go is pushing us towards doing the right things without anybody realizing it. Go wants us to work with these slices because then we’re really working with arrays and contiguous memory, and it’s giving us our best opportunity to have these sympathies without even realizing that we’re being sympathetic with the hardware. So Go to me is just an incredibly fascinating language when it comes to that.

And other areas of the language too, where you see that you’re really being sympathetic with the operating system scheduler on the concurrency side, without even realizing it. Just these idioms and these things that we tell people to do all the time, they’re based not on just, “Hey, we want you to do this”, they’re based on real things around performance, simplicity, readability, those types of things. It all kind of comes full circle.

Josh Aas: So what I’ve just described is the basic process of getting trusted from scratch. And that does require a big commitment. It requires quite a bit of money to get set up to a point where you can pass audits and even apply, and then every year while you’re waiting for all this stuff to happen for 6-10 years you have to stay compliant, get audited every year. So you’re talking millions of dollars and 6-10 years before you can even be a publicly-trusted CA in any meaningful sense. That’s the basic process.

There is a way to make a shortcut, which is how Let’s Encrypt was able to start without waiting 6-10 years first. So we did go through the process that I’ve just described, building up our own root of trust from scratch… But the world right now does not really rely on that yet, because it has not been long enough.

[31:52] And somewhere around mid-to-late next year we’re gonna switch over to our own root that’s trusted from scratch. But from our inception through now, we have what’s called a cross-signature… Which means we knew we didn’t wanna wait 6-10 years to start offering Let’s Encrypt services, so we’ve found another CA that understood what we were trying to do and was willing to help. They had a root of trust that was already trusted.

What essentially happens is we create a contract, an agreement between us, and then their root of trust essentially lends its credibility to us. So they issued a certificate that our root is trusted by their root, and their root is trusted by the browsers, basically. So that’s called a cross-signature. We acquired that before we did much of anything… Because without that agreement in place, there’s really no CA.

So one of the first things we did was get that agreement, because without that agreement in place, there’s no point in buying hardware and doing anything else… Because –

Adam Stacoviak: I believe they’ll let you use it to a certain capacity for free. You can’t change the desktop background, you can’t do certain things… But it will be a machine I believe that can run in perpetuity without any restrictions. There’s like certain things you’d want to do you can’t do without getting a license key.

And so I really wish – like, that would be… Maybe they would lose tons of money, but that would be cool. Like, make Windows free, like macOS is free. macOS is essentially free to anyone who can run the hardware.

Kurt Mackey: Well, so there’s the startup tension of sometimes if you know too much about a problem, you talk yourself out of doing it. Like, running managed databases is a giant pain in the ass. In the same way running global hardware infrastructure is a giant pain in the ass. But the difference between managed databases and global hardware infrastructure is I’d done one of those things before and realized how much of a pain in the ass it was. And I wasn’t like gung-ho, naively jumping into a problem without realizing the downsides. And I think one of the problems with – it’s always interesting to me that people start startups… I don’t know if this is true, and it’s probably not, but I’m really aware of people who start startups, they have no real credentials for it. Not credentials, but like they don’t have the right experience to build this startup. And in some ways that’s an asset, because they’re so naive that they don’t even notice what they can’t do. And so a lot of times they end up finding stuff that seems impossible to people who know better. So for me, managing databases is a really known, complicated problem.

And there were two things at play for us in particular. One is I’m like hyper-focused on like doing something novel, which you might’ve noticed when I was talking about people that come and sign up and just wanna run their apps, how I’ve had to really let go of not trying to do something novel for them, trying to just give them a good experience based on what they already want. But I’m hyper-focused on doing something novel, and I’m very skeptical of doing something everyone else is doing, that’s really complicated and hard. And so databases fell in that for me. It’s like “I know how hard it is to run databases. I also wanna do something – I wanna put all of our energy into doing something that’s really meaningfully different than what already exists out there…” And so I skewed way hard to “Let’s do the meaningfully different than that exists out there.” Which in some ways wasn’t wrong, because we got this machines API three years before people needed to realize they needed to run like LLM-generated code in a safe environment. We didn’t know LLMs would even be writing code at that point, but it was a thing that we kind of saw the need for vaguely, and needed for ourselves and managed to ship.

But I think that I over-fixed it on how hard managed databases were… And then a component of that is – and so I know what it takes to run a managed database, and it’s really just a lot of people who know how to operate that database, and a lot of tooling that lets them know when to go look at a customer’s problem, and do something about it. And so that’s kind of why we opted for Percona, because they’re really good at running people’s Postgres. So we know that within 15 minutes of us noticing an issue with the customer database, we can get a hold of Percona if we want, and have them looking at it with us, which is the key, I think, to a managed database service from day one, is just fixing problems so people don’t have to. And it’s almost like a human element. It’s almost like a human and support problem, and less of a product problem, in some ways.

David Heinemeier Hansson: [00:49:38.25] A little bit, right? So I think it is all quite fluid. And also, fundamentally, I’m fine with that fluidity. I don’t care. I just want you to have a path, a roadmap, a tool to get from A to B to C to D to E without it feeling like there are hard lines and hard walls. I want you to be able to start on a VPS, or a VM, or a dedicated box, and move every which way around them.

In the Kamal config, for example, that we ship with new Rails 8 installations - Kamal is the deployment tool that we’ve made default - it should be as simple as changing the IP deployment addresses. And that IP can point to the VM, it can point to the VPS, it can point to the dedicated box, it can point to your own hardware, and Kamal won’t care. It’ll be the same.

Now, part of the pitch for Kubernetes, for example, was that that was going to be true sort of at a broad scale, and [unintelligible 00:50:29.05] was not true at all. I’ve never, ever seen a Kubernetes installation of any sophistication being able to just, “Oh, we’re on Amazon? Flip the switch, George. Ket’s move over to GCP.” “What?! No.” That’s like “Flip the switch, George. We’ll start having the meetings about the meetings about how we can move out four years from now.” That’s what the Kubernetes migration path often is. And I was just like “Yeah, there’s got to be a better way for that”, at least for some style of application.

Now, again, I actually think Kubernetes is an amazing piece of open source infrastructure, and if you have it in your heart that you want to start the next AWS, or the next GCP, or the next even Digital Ocean, you should probably look into Kubernetes. You’re going to be operating at that scale, and you’re going to have hundreds of engineers, and it’s going to be very complicated. Great. Kubernetes sounds wonderful for you.

I’ve looked at Kubernetes enough. We ran Kubernetes on AWS for long enough that I realized that is not what I want to do with my time. And it is not proportionate to the kind of problems that we have. Something far simpler is possible. And if that simplicity is achieved, we also get portability. And I think portability is actually less of a thing at super-high scale. It’s not that common that huge applications and businesses move from one provider to another. Where I want to make sure that the portability is present is in the early days. Is that you start your startup, and maybe you start it on Heroku, whatever, or something else like that, and you could just move on. You don’t get locked in. You don’t get locked into a cost basis that’s prohibitive.

Jerod Santo: It’s now time for Sponsored News!

WarpStream lets you bring your own cloud

WarpStream’s BYOC deployment model for Kafka-compat data streaming gives you the best of both worlds between self-hosted & cloud. This model also give you a lot of zeros (that’s a good thing)

Zero disks - WarpStream’s Zero Disk Architecture eliminates local disks entirely and reduces storage costs by more than 24x.

Zero inter-zone networking fees - More than 80% of Kafka costs are not hardware – they’re inter-zone networking fees. Because WarpStream runs on top of S3-compatible object storage & doesn’t manually replicate data between zones, those fees are completely eliminated.

Zero ops auto-scaling - WarpStream replaces stateful Kafka brokers with stateless Agents, so your team can skip the weekly burden of partition rebalancing, scaling headaches, volume management, capacity planning & more.

Zero access - WarpStream Agents run on your VMs, in your cloud account / VPC & storage data in your object storage buckets. Traffic flows seamlessly from producers to consumers without ever leaving your virtual private cloud (VPC).

Here’s a non-zero - WarpStream works in every cloud! They have native support for AWS S3, GCP GCS & Azure Blob Storage. It also works with any cloud or self-hosted solution that has an S3-compatible object storage.

Head to warpstream.com/byoc to learn more and get started for free.