Adam Jacob: I mean, there’s certainly no one left to hang out with at the end… And certainly they’re the people who are most likely to be able to talk to you about how it is. Like, they’re the ones you can most talk to about “Oh, that’s awful. I’m so sorry that happened to you. I know what that feels like.” They’re also your peers. They’re the only other people who know what it’s like. So when you think about destroying that pool of people… Like, of course I wanna help people up in that way, because I need those people. Someday I’m gonna need them. I’m gonna need to call them up and be like “I had to do this hard thing…” And they’ll be like “Oh, yeah, I understand.” They’ll be there for you in a way that they won’t, obviously, if what you did was punch them.

Jerod Santo: Yeah. I remember I was at a hotel one time and I was hanging out in the pool, or maybe the hot tub, and there’s a sign that said “This pool is not for defecation purposes.” Yeah, which was a very strange sign. And that might not be verbatim, and I can’t remember if it was defecation, or – it was very formal though, so it probably did say that. And I thought “Yeah, somebody pooped in this pool at one point.” And there was an incident where they said “We’ve gotta put a sign up, because –”

Jerod Santo: I’m not gonna wade into that pool. We’re gonna let that Adam have the final say on that. Alright, Nick, anything else you want to say about this or any topic before we let you go?

Tejas Kumar: Yeah, that’s a great question. You’re unlikely to crash things, especially like – so I’m working on a 2021 MacBook Pro with an M1 chip. It’s pretty old. It’s three years old. It’s a very old device. And it works just fine. It’s pretty much impossible to crash, unless you really get around it.

For example, the macOS kernel is really extremely world-class at making sure you don’t crash the system. There’s plenty of safeguards in place. So what will typically happen is your application itself will freeze, so not your entire system. And at some point it will say “Hey, this thing’s taking too long. Do you want to force it to quit?” and the kernel will just pull the kill switch. Very cool.

There’s a way around this using an environment variable. So PyTorch is the thing that’s causing the memory problems. PyTorch is an open source library from Meta that helps with machine learning. And so PyTorch allows you to set an environment variable called mps high watermark ratio. And this is at what point do you throw an out of memory exception. Because high watermark literally means you’re about to reach the watermark, the level where if this is a tide pool, you’re going to start losing water; you’re going to overflow, literally. I love the language we have in computer science. Overflow, watermark etc.

So you set this, it’s a threshold before overflow, at which point PyTorch will just kill the process. You can set that to zero. And then what will happen is you just completely bypass – so you’re like “You know what? If we have an overflow, we have an overflow. I’ll just like hard reboot.” And so you set that to zero, and then you’ll crash your system. Because all your GPU, all your CPU is going to be consumed, and you’re not going to have free resources to respond to the Caps Lock key and turn the light green. You’re not gonna have resources to –

Jesús Espino: Okay, next step - one of the things that I was intrigued about Go, because it’s one of the key things, is goroutines. It’s something that everybody wants in Go, everybody loves in Go… And I did want to understand better what a goroutine is, and how it works… And I started digging into the source code and I started investigating how the goroutines work under the hood, from the perspective of the goroutine, of the lifecycle of the goroutine. And actually, for that I prepared another talk that is called “The secret life of a goroutine.” And that in that talk, what I try to do is follow the process of when a goroutine is created, how it’s created, and what are the steps that the goroutine goes through, and how it changes from one state to another, and how the reasoning around that changes.

And one of the things that I discovered while I was doing that was it’s something that I already have kind of the intuition around that, or the understanding about that, that is goroutines are cooperative. But whenever I started seeing the code, it was more obvious and evident that it’s a cooperative approach.

In threads, in operating system threads, you have the operating system deciding when to cut something and assign another thread to the processor. In Go, the goroutine itself is responsible of saying “Hey, I’m going to stop myself, call the scheduler to select another task, or another goroutine, and the goroutine is going to start running.” Not only that, I just stop myself because I’m waiting for something, and the one that is going to wake me up is another goroutine. For example, whenever you send something to a channel, and it’s another goroutine waiting, the goroutine that is sending to that channel is going to wake up the other goroutine. So they are collaborating together to do all the processing.

There’s some special goroutines like the system monitor goroutine, that is doing certain things, or the net pool that also - it’s kind of monitoring certain things for getting back that goroutines whenever they finished… But in general, the goroutines are waking up each other or are sleeping by themselves and calling the scheduler to select another goroutine. So that cooperative nature of the goroutines is something that I’ve found very interesting, and it was an a-ha moment for me also. Yeah.

Chris Benson: I think there’s certainly a chance at it, and I would argue – it’s the same argument I’ve made in previous shows where we talked on similar topics, is that we’re seeing… As the AI industry is has been maturing these years at an incredibly rapid pace, but we’re still seeing many of the things occurring that we saw when the software world was really maturing over several decades. And the place where open source has really, really worked are in common touchpoints where all organizations or many organizations need a common thing. And they might build something differentiated on top of that, for their revenue, to drive profitability… But there’s so much that is underneath that point of differentiation that they and many other organizations can get the benefit out of a lot of effort, a lot of work; a lot of times they’ll have paid employees do it…

So there’s a point where working together and doing open stuff makes sense for business, and it drives profitability. It may not be your single point of differentiation, but if it’s anything under that, why not? Why not share the costs, and pool expertise for the best possible foundations? And so what I’m hoping is that we continue to see that play out in the AI space. We’re seeing – if you look at Hugging Face, we’ve already talked about the fact that a couple of months ago they announced that they were hosting a million models; those are all open source. Really, really impressive. And so I think that there is a good chance that a vibrant, open community around AI can and will continue, and it will have a lot of corporate players involved in it. So I’m very optimistic in that way.

Break: [00:13:19.26]

Adam Stacoviak: I’d like to see better security tooling, that aids a developer. Does that shift left. Versus being like “Nah, third party. Not cool. Rug pool, not cool.” Super-not cool. I would like to have more security tooling in there that’s for developers, and developer-focused, and doesn’t become this - and I hate to use the word “like” in there… Doesn’t become this signal versus noise issue where you’re just getting alerted to things [Dependabot!] that don’t matter. Right? We talked about this recently on securing GitHub, with Jacob DePriest. And so I think my stance is really like “This sucks.” That linking out to third-party CDNs - that’s dead to me, based on this for sure… Because you can’t trust the domain ownership anymore, right?

Gerhard Lazu: I really like how we are improving Jerod’s pool. This is amazing.

Adam Stacoviak: The last time Justin was on “Rug pull, not cool”, which has become a catchphrase when it describes the inevitable potential rug pool that is not cool that’s happening, and continues to happen in open source…

Autumn Nash: It’s what’s available to you. A lot of times you may have a database that might be a better solution, or you may have an infrastructure that’s a better solution, but where do you find the people that know how to use that, and know how to use it well enough at scale? So I think that a lot of it too is that colleges make application engineers, and they don’t make infrastructure engineers. You know what I mean? You have a bigger pool of people to hire for that. Which - I mean, that’s debatable. We need to do more infrastructure in college, whatever. But that’s a debate for another time.

Kelsey Hightower: And I think that what people want from you is number one, they want the spectrum to be expanded. I know what my people are telling me, I know what my customers are telling me. “Let’s check in on Kelsey, to see if he can add something to this for us to consider.”

So let’s say it’s bad advice. So then you’re on the other end of the spectrum, of things we shouldn’t do. It’s bad advice. And as long as they’ve got something on the other end, now they’ve got something to compare and contrast to, and they can go Why. So for example, I won’t mention the company name, but they’re going to do an acquisition. I did the review and said “I wouldn’t buy this.” But they bought them anyway. So was it bad advice? We’ll find out in a couple of years, whether that thing adds to the revenue pool or not, or the people are still there or not after a couple of years… But I think it was one of these things like “If that’s the worst that Kelsey can come up with, we’re comfortable with that. We just needed to hear, so we can understand the full complete picture of the risk profile.”

So I think as an advisor, your goal isn’t to necessarily be right or wrong, because that’s tricky. Because it also depends on their ability to execute. So everyone kind of knows that my job is to say “Here’s what I believe, and here’s why.” And the math is clear. And if it’s wrong, then “Look, great. Don’t do that.” But at least we have the spectrum. That’s what people are paying for. They’re paying to expand the risk spectrum, so they can make a good decision.

Jerod Santo: We need some brand clarity here. Free advertising for that pool there in Vegas…

Alex Kretzschmar: Mergerfs literally globs together any disks that you supply underneath a specific glob point. So for example, you could have three or four disks in your “array” today. And you could pull one of those drives; immediately, it would disappear from the pool. There’ll be no data loss, because the magic of Merger is that each disk has an individually readable and addressable filesystem on it. There’s no striping of the data.

Downside is that there’s no parity either. So you need a third-party tool to generate some kind of redundancy for a mergerfs array. So something like SnapRAID, which is a snapshot – it basically takes a snapshot of your data disks, and then calculates parity on that. A little bit like UnRAID, although UnRAID does that in real time, versus SnapRAID, which does it as a cron job that you run every day, or something.

With ZFS, as we’ve said, you have to expand the vdevs very carefully. But with Mergerfs you can add and remove drives willy-nilly, because each disk is an individual file system; it doesn’t matter. There’s not going to be any data corruption, or anything like that.

So if you are just storing a bunch of movies and TV shows, and you don’t need the utmost in bit rot protection and all the fancy stuff that ZFS offers you, it’s a very cost-effective way to build out a small system. The performance is pretty good, too… So it’s based on top of Fuse, which is a software layer of like a file system that basically runs in user space, rather than kernel space. So there are some performance limitations. It will easily saturate a gigabit connection, don’t get me wrong… But you’ll use a little bit of CPU as you do it. So things like the ZimaBoard, that are a little bit less powerful, may struggle to saturate a gigabit connection with Mergerfs, for example… Whereas with ZFS, because of all the caching and stuff, they might be fine doing the same thing. So really, it just depends. The answer in building servers and infrastructure is always “It depends.”

[00:27:39.08] I just, for example, this weekend wrote a blog post about doing a bunch of Quick Sync hardware video testing to like figure out “Is an eighth gen Intel CPU that much better than a 13th gen, versus an Arc Pro for basically people running Plex Media servers?” And the outcome, much to the chagrin of Reddit, was “It depends.” [laughs] It depends on what your budget is, it depends on how many streams you’re doing, it depends on what the availability of specific used hardware is in the marketplace this month, et cetera, et cetera. And the same is true across – when you build a certain system, whether it’s a media server, whether it’s a network, whether it’s deciding what to have for lunch - there’s always compromises you’ve got to make. “Oh, I’ve run out of cheese, so I can’t make this particular thing.” Or “Oh, well, I can’t get this particular brand of whatever thing I like.” It’s a tenuous analogy, but bear with me…

Adam Stacoviak: But it gives me the chance to choose my funding amount. It defaults to 20, because the minimum is 20… But I can certainly change that to a couple hundred bucks. What I can’t understand so far looking at this UI is if it’s been – I think that means it’s been 500 funded. There’s no requests on a certain amount, so I could just be throwing money onto the bucket, and whoever eventually does this just gets this pool of money. I don’t know what the value is necessarily of it… I’m not even sure what my question is, but I’m just looking through what the user experience is, and I’m like real-time analyzing in a way, and to some degree not doing a great job… But it’s not a bad UI. But you’re giving this on-ramp to folks, they understand what the issue is, they understand how much it’s been funded, and they can give money, or as we just mentioned to be confusing, they could also contribute, that is not funding. It’s actually code, potentially.

Amal Hussein: …there’s a commuter pool of people.

Justin Garrison: One thing I’ve found fascinating about some of the talks you’ve given and information is the fact that Meta is still notably an on-prem company. You have your own data centers, you have your own regions, you have machines… And it doesn’t seem like you try to hide that from people. You don’t try to abstract it away. At least I haven’t ever seen a reference to like “It’s our internal cloud.” No, it’s like a pool of machines, and people run stuff on the machines. And the software and the applications running on top of it are very much like a – this is just like a Systemd unit; you’re just running it containerized.

What other types of services do you have internally that people need? I mean, I saw references to things like sharding for like “Hey, we need just fast disk places, and we need some storage and databases externally.” But what are the pieces that you find that are like common infrastructure for people to use?

Johnny Boursiquot: It takes some practice, I won’t lie. The way at least I think about it is if I can classify in my mind where a certain set of analogies sort of fall into play… So think about it this way - there’s a fundamental sort of concept, there’s fundamental slots you can fit different ideas into when you’re explaining a particular topic, whether I use… Alright, let’s take semaphores for example; whether I say “Well, for a semaphore, imagine that you go to a restaurant and you are ask the host for a table. It’s you and three other friends. There might be scattered spots at different tables that could sit all four people. But in this case, you need all four seats available in order to sort of enter the restaurant and sit down and eat. So when I explain the concept of a semaphore, I’m reaching back and sort of using that to say “Well, think about it this way… You can have a size, a certain set of things, goroutines that you need to be able to execute at one time. Think of it as going to a restaurant and waiting in sort of the waiting area before all four of you can actually go in. You don’t just want one or two or three, you want all four of you.” So using that as a sort of framework, and then say, “Okay, well, maybe it’s not a restaurant, maybe it’s the pool, and they have a certain number of people that can be in the pool at any given time.” So you start to sort of swap out these sort of blocks, and then you can come up with all kinds of different analogies where as long as you can identify in real life, in the real world, similar sets of things that people are coming across in their daily lives, then you’re just like swapping the different scenarios. But it’s the fundamental still same basic concept. So that’s how I think about it, and that’s what allows me sort of come up with scenarios on the fly if I need to, because there’s this fundamental concept I’m just fitting a block into, that I already have in my mind and I’m trying to convey.

Adam Stacoviak: One thing that Vicki says that I thought was kind of notable… Because I think, okay, there’s details here, but then what’s the trajectory of Redis, and those that intend to be the official fork, or the new fork, or the new Redis, whatever it is. And she says, “The old Redis was for developers. The new Redis is for enterprise sales and for generative AI. It’s true that is not yet entirely clear what all this means for the future of Redis software.” So that’s what I think about it, is like “Okay, great. Rug pool, bot cool.” You can put them on a T-shirt.

Jerod Santo: This is a peeless pool.

Felix Geisendörfer: Yeah, so I think for me the most common case why we built it into Datadog was you have distributed tracing already, which is now a pretty common way for people to look at their request latency… And at some point, you get to a span in your distributed trace where there’s a slow request, but it’s the last leaf span; it’s slow, it’s in your application, and you don’t know what’s going on. And so basically, breaking that down was one of the big motivations for us, and what we’re doing is showing a few similar to Go Trace UI, where we show one lane per goroutine, but we filter all the goroutines to just show the ones related to the request. So that was use case number one.

The second one that I actually didn’t expect so much was also throughput-related, but it’s actually pretty common for people to have systems that read stuff from queues like a Kafka queue, and they observe that – they see a lack of throughput; so there’s a big lag building up, like messages in the queue not getting processed, in the backlogs that they need to crunch through… But then they look at the resource utilization metrics of their application and what they see is that the CPU utilization is actually low as well. So they’re not making progress on the workload, but their application is not working hard. And they’re like “Well, I pay the cloud provider for all this hardware. Why is it not doing the thing I want it to do?” And this is where execution tracing can really shine, because you look at actually the classical view of execution tracing, which is one that’s scheduling-oriented, and what you’re looking for is all the gaps. Because you want this view to be like packed; you want each lane to be full of goroutine execution. And if you see patterns of gaps, sometimes a pattern can tell you a lot. Sometimes you see execution, everything is busy, it’s super, and then you see like a big vertical empty space of nothing happening, which means everything had piled up probably on the same thing, and now you can sort of click around and figure out what that thing was, and solve that throughput issue. So that is definitely another use case that I’ve seen quite a lot.

And last but not least, sometimes - again, the goroutine view can bring things home that you could already see through other signals. For example, if you have a goroutine pool that’s not sized correctly. So you have a goroutine pool that’s maybe making some outbound network requests, but that becomes a bottleneck for your application - especially in the scenario that I’ve just mentioned - you can look at the block profile, but honestly, it wasn’t a recommended profile type because of the overhead, which actually is still a problem, because we haven’t ported the fix for the execution tracer’s faster stack traces to the block profiler yet… But basically, what you’re looking for is you’re trying to figure out which goroutine pool, if you have multiple of them, is sort of the bottleneck. And the way you can see it is the bottleneck goroutine pool on the other side, like the goroutine sending to that, will be blocked on send. So you’ll see a lot of contention on Send operations on the channels. And on the other side, on the outgoing side, you’ll see starvation on receive. And that’s something you can kind of see in the block profile, but it’s really a difficult profile to read, because it often gets also kind of dominated by benign receive waits… So you have like maybe another goroutine pool that’s completely irrelevant for the problem you’re trying to debug - it has more goroutines than everything else, and now it’s just going to drown out that profile type with the receive operations.

And so it’s really something else to then look at that data with execution traces, where you can really see each goroutine, and what it’s doing over time, and you realize “Hey, I have a pool of 50 goroutines here, but they’re spending 95% of their time waiting on networking, and 5% executing. No wonder why I’m not making use of my CPU resources. I need to significantly increase this pool, so not only that pool can do more work, but all the other pools on the left and the right side fit can as well.” So I think that’s – yeah, another pretty common thing.

Adam Stacoviak: Part of me wonders though, however, if that is a reconfiguration of our brain OS, so to speak… Like, habits slash dopamine… Where do you get your dopamine? Because then you gravitate towards the places you get that dopamine hit. And we do have brains that control us in more ways than we realize it controls us. And so I wonder myself - this is not just a question for you all, but just generally, this is what I ask myself, “Adam, am I thinking about this while at the pool, on this vacation, or whatever it is, because I love it? Or is it because that’s where my dopamine hit comes from?”

Kris Moore: You want to stair-step it. You want to go to the next release, make sure everything looks good and is happy… And then if you’re adventurous and you want to go try out Dragonfish, give it a shot. And then just don’t upgrade your pool if you need to roll back, because beta is too scary… Fine, if you can do that.

Duarte Carmo: Okay, awesome. And then – so one of the things that surprised me about Go when I was writing is that for this specific project there needed something to happen. So I needed to basically look at five different Danish newspapers, get their RSS feed, and then automatically throw them at a large language model to get the translation for the contents of those news, basically, and the translation for the title. So I basically wanted to do a bare bones, really simple website that I can just read Danish news from a bunch of different sources related to where I live… Because I don’t do Danish yet. Because - yeah, exactly. I’m very bad.

[00:54:08.20] And so one of the things that I need to do is I actually need to request all of these RSS feeds, and I need to actively run all of these translations, so call all of these APIs, but I wanted to do it in a fast way. And when I think about Python, I was thinking okay, now I have to throw this thread pool executor, or I have to write a bunch of these boilerplate code to do some concurrency here… And I have to swim against whatever normal Python is to try to make this as efficient and concurrent as possible. And then I was like “How should I actually do concurrency in Go?” and there’s this great website that I’m sure you guys know almost by heart, which is this Go by Example website… And it showed me that all we need to do normally is like go, and then the name of the function, and it just runs it in the background. And I was like “What? Why don’t we have a go keyword in Python? This is great.”

And so I was really quickly in love with how I could actually call all these APIs and get these RSS feeds and call these translation services super-fast, and super-concurrently, without adding so much boilerplate to my code. It seems what it is perfectly natural to Go, because I think that’s one of the reasons why it was designed as well. And one of the questions I had is, I started noting that there’s things – like, I started doing things like channels, then I said that there’s also like wait groups… There’s a bunch of perhaps different ways of doing concurrency… Is there a way that – for example, let’s just put a simple thing. You need to call an API in parallel for five different – you want to call at the same time. Do you jump for channels? Do you use wait groups? What do you go for?

Jerod Santo: Well, there’s your answer then. It’s just content creation. Why Elixir is better than Next.js. Why Elixir is better than Rails. Obviously, these are two different things, but you get the point. And then you just create content and you grow the pool of Elixirists. And then your business grows, if you’re fine with growing a little bit slower, a little more tried and true… Unless, you know, vertical hockey stick, then you just work on the overall Elixir ecosystem, and as it goes, Oban goes.

Johnny Boursiquot: [laughs] If you have to - and actually, Miriah, as a data person I know the kinds of volume and the ways you’re dealing with them, of data that you deal with, is going to be perhaps different from somebody who’s writing sort of networked applications. Not that you’re not working with some network stuff as well, but I think if you’re doing things in – if you have unpredictable workloads, I should maybe put it that way… If you have unpredictable workloads, it’s going to be different than if you have a predictable set of data you know you have to deal with. You know you have to deal with five gigabytes’ worth of text processing… Perhaps your approach to writing your code is going to be a little different than if you have to write a network service that is supposed to deal with streaming data. It’s going to be different; at any one time you’re working with a subset of a larger sort of pool of data.

Deepak Prabhakara: Yeah, you bring up a very interesting point. I think, to your point, one thing we always overlook in the build versus buy is “Can you hire the right set of people to do that?” Internally. So your choice of, for example, today React versus something else. Or even like Golang versus Java, right? It’s fundamentally about who can you bring on board to kind of support it; even if you build something, it needs to be supported in the long run.

So I think a lot of the build versus buy choices also happen – it may not be the best one that you pick up, but it could be the one that has the most talent pool out there. Or you think that you will reach a point where this thing is gonna get big. A lot of people bet on React when they saw who was behind it, saying that “This will become big, so we will have enough people who picked up the skills [unintelligible 01:11:16.16]

So I think, to your point about that on-prem versus SaaS - we cannot talk about Salesforce as an example, right? They were the first ones to kind of – well, they did a great thing with that whole “No software in the cloud, you don’t have to worry about it.” And today, if you think about it, it helps you move fast, right? You’ve got not just a CRM, but a whole platform around. You can now do pricing quotes on the back of that, you can align your account management team around that, customer support can happen there… So you’ve suddenly gone from just a CRM to a whole range of things within that same platform, so to speak.

Techno Tim: And that did it for me. So it wasn’t until later that I learned about all of the cool things about ZFS, like snapshots. Snapshots are awesome. And in my TrueNAS system I take snapshots every hour, of every item in my pool. And that’s basically the diff, but really quickly, I can go back and revert to a snapshot if something goes wrong. ZFS snapshots are like lightning quick. I do them every hour, on everything, and throw out the ones that are empty… Because why not? And again, it’s only the diff.

So that’s awesome. I get quick snapshots, easy ways to revert… The integrity check, the scrubbing of the pools, the integrity check where however often it’s going to go through and – I don’t know if it’s a SHA, or hash everything, but it’s going to do an algorithm to figure out “Hey, are the things I have on disk - do they match the things in RAM?”, which is pretty awesome. And then all the other things you could do. If you have two systems, you could move those snapshots really easily to another system. So if you do a 3-2-1 backup at home, three copies to two mediums, one off-site - I always forget - spin up another TrueNAS server and there’s your second local copy, and the snapshots outside of the first one is just going to be the diff, so it’s really quick. You could do that off-site if you wanted to, really fast.

And so for me, it was just like, okay – Btrfs I typically see being used, and I think it’s unRAID, like you mentioned, but also in Synology… And Synologies aren’t devices that I have. I’ve had options to use them, but at the end of the day, hey, TrueNAS – I honestly think a lot of it has to do with just TrueNAS being a great product by iXsystems. And on top of that, Proxmox runs on ZFS, too. So it’s like, two of my favorite things are using ZFS, so it kind of came with the territory. And they made, I think, good technical decisions when they were building those two products. And for me, it’s – you know, I’m still uncovering all of the things that make ZFS a good file system. But I will say, it’s snapshots, it’s scrubbing, it’s integrity checks… And yeah, and then storing everything in RAM, and having it lightning fast. That’s kind of Linux’es way; that’s what Linux does really good, is store everything in RAM.

Allan Jude: ZFS and OpenZFS are licensed under the CDDL, which is a slightly modified version of the Mozilla Public License. And so the CDDL says - I’m not a lawyer; don’t take my legal advice. But in general, the license says that you can make a binary out of CDDL license code, and you can license that binary however you want, so that you can make a product out of it, or whatever. And so you can make a ZFS binary, a kernel module, or whatever, and license that even under the GPL, so it’d be compatible to integrate it into the Linux Kernel. But the GPL itself requires that any code you link into it has to be licensed under the GPL and the CDDL. It doesn’t let you just change the license on that code to be under the GPL. And so that’s where that kind of incompatibility comes in. But it’s not a very strong one, and Ubuntu is pretty sure that their legal way around it is working.

But because the FreeBSD license is literally two sentences, “Don’t take off the copyright” and “Reproduce the copyright”, it means that we could pull in all of ZFS, the codebase directly into FreeBSD. So if you go to the FreeBSD GitHub and the repo, under syscontrib there’s a directory called OpenZFS that has a whole copy of the ZFS source code. So this means ZFS is completely just built into FreeBSD. So the FreeBSD bootloader has full support for reading from a ZFS pool, and so you can boot directly off ZFS, even if you’re using all the advanced features of ZFS. So this allows a feature called boot environments, where your root filesystem, if it’s ZFS, it’s actually a virtual file system, and you can have multiple different ones, or you can clone it and take snapshots. So when I’m preparing to go to a conference or something, I have my laptop that runs FreeBSD, that I give my presentations off of, and I use a desktop most of the time when I’m at home… So I haven’t used my laptop much in the months between conferences. So I’m going to update the OS, because it’s been a while, but that’s when everything has the greatest chance to break, when you’re installing all these package updates, and so on.

So with ZFS I can take a snapshot of my root filesystem, and save that as “Before I broke everything.” And then update stuff. And it all seems to work. And I check at home, and I can play stuff on the TV, so my HDMI is working, everything’s fine. But say I fly to the conference, I get there, and something isn’t working right. I can just reboot in the bootloader, say “Hey, I want to use the image of before I broke everything, instead of the latest version of FreeBSD.” And it will roll back my root filesystem. But, importantly, it doesn’t roll back my home directory. So I still have the latest version of my slides, but I have an older version of my OS that still works, because whatever latent problem I found, that ex Windows didn’t want to support a certain resolution, or whatever broke. And so that can be really powerful.

Or if you’ve ever tried to update the firmware or the OS on like a little router or something, they often have two images; they always have like the one-year update, and then there’s like a spare one [unintelligible 00:31:15.20] switches to this, so that if you somehow botch the firmware update, they have a second firmware they can boot from in the meantime. So FreeBSD had a feature like that before it had ZFS called NanoBSD, that let you do that. You’d partition up your disk into two big partitions and one little one. And the little one would hold the configuration, and the two big ones would be like version A and version B of whatever you’re trying to install, whether it was a firewall, or a router, or whatever. And you’d be running off A, so you’d update B, and the next time you’d boot you’d switch to B, and then you can ping pong between these two images. But ZFS allows you to do that with an unlimited number of images. Because each is just this virtual file system in the pool, you can have as many of these as you want, and you can access them directly from the bootloader and decide which one you want to use.

[00:32:01.16] And when you’re switching to one, it doesn’t make the other ones go away. All the data is still there, because ZFS is copy on write, so only the bits you actually changed are taking space. So having seven copies of your operating system doesn’t take that much space, because you only have to store the differences, not all of the data seven times. And so that allows much deeper integration.

But also, if we go back – when we talked about the fact that the base system utilities, like top, for example, are not a separate package that came from a different maintainer somewhere. That is part of the FreeBSD operating system. So when you start top on FreeBSD, under the line where it tells you about how much free memory you have, it has two lines about ZFS, telling you “This is how much of your memory is being used by the [unintelligible 00:32:43.04] this much is for frequently used files, this much is for recently used files, and this much is data that’s pending being written. Here’s what your compression ratios are, and memory”, and all this other information that would only be there if you had ZFS. And because FreeBSD is integrated with ZFS, top can just be compiled against the version of ZFS [unintelligible 00:33:02.19] and be able to get that information all integrated.

Adam Stacoviak: [00:18:06.17] Whereas now I feel like if we can embed them into the shows, in the ways we think we can, and swap them out when necessary dynamically, then I think that’s a big win for us and a big win for the folks trying to find the right folks. And I think if we could do a good job on vetting who comes into that pool, and just some way to provide - like you were saying, Gerhard, quoting Mary Hightower… Hi Mary, by the way… You know, I think that’s great. Building great software, building great teams I think has always been the fun part of the conversations. We’ve just had that conversation with Dan more on Letters to Developers. Such a cool conversation, the first show out of the gate this year… And I think it’s gonna be a big hit for the year, and it’s show number one for 2024.

Christopher Hiller: Yeah… Can we do like a death pool? Or no?