Kris Brandow: Yeah, and I think that brings up another thing I’ve spent some time thinking about, which is kind of the comparison of software engineering is like writing, versus software engineering is writing. I kind of feel that using prose to write your test is more toward the “Okay, this is a good way to understand things, so we’re just gonna borrow this from (I guess you could probably say) fiction, or novel-writing, or that kind of structure of writing, and trying to apply it to software engineering”, which I think is a good first step. It does make tests easier to consume, but it is awkward, because you’re applying an analogy on top of software engineering, which is kind of like any other type of analogy. There’s some like “Software engineering is like gardening, or construction etc.” And it fits well at first, and then you start to see the places where it doesn’t quite meet…

And I think the next step we need to make is “Okay, they have some nice things in these other disciplines of writing. How can we take that and bring it into software engineering and adjust it to make it so that we can write tests that are easy to read and are pleasant to read, but don’t necessarily come with the baggage that comes from, say, the looseness of trying to write things with prose and paragraphs.

Adam Stacoviak: The “don’t care” aspect that Matt Asay brought up - I’ve been thinking about this as we’ve been having this conversation, so bear with me if this analogy is not perfectly on, but this is somewhat how I feel about it. And my story is almost like coffee, you know…? I’ve been a coffee drinker my whole life, and the majority of my life, I always wanted to have good coffee. But only until later in my life that I understood what good coffee really was, and began to truly care about how to make good coffee. And so for a while there, I was drinking - and no offense to the Keurig drinkers out there, or even Nespresso… If you’re drinking that, it’s not truly good coffee. It is coffee, but it’s not good coffee. And as a coffee drinker, sometimes you just want to have coffee, right? As a software maker, you sometimes just want to have software. And you can just use it, because it’s open-ish, or it’s source available, or it’s truly open source… And you just don’t care, because you can use it. But only when the rubber meets the road, when you have to have good software, or good coffee… And I’m not sure if good software translates very well in comparison, but the point I’m trying to make is that I didn’t truly care about good coffee until I truly understood what good coffee was. And then I began to care about just-in-time grinding, and I cared about water temperature, and the pH balance of the water… Just the different aspects. Then I began to care. Basically, when the rubber met the road of good coffee, then I cared.

[32:00] So I can kind of understand your sentiment where you have developers who just do their job. And I get it, you want – like Jerod in the past, or Jerod currently, with more care, you want to have access to good software out there, that you can use in ways that benefit you, and those that you’re trying to serve in your job.

Adam Stacoviak: And software development was so antiquated then, in comparison to today’s world…

Jerod Santo: You’ve probably head the phrase, “code is read more than written.” What’s the takeaway to that fact? maintainer > author

That’s true, but according to Facundo Olano, it’s not the whole truth: “Code is a means to an end. Software should have a purpose, it’s supposed to provide a service to some user. It doesn’t matter how well written or maintainable the code is, nor how sophisticated the technology it uses if it doesn’t fulfill its purpose and provides a good experience to the user”

This is why he emphasizes that “code is run more than read.” What’s the takeaway to that fact? user > dev

In addition to explaining the above, Facundo goes through a series of “greater-than” comparisons and describes their implications. Unusuable software? dev > user Works on my machine? dev > ops and so on… An interesting and fun read. Check it out.

William Nichols: Well, the team software process is really the industrial version of personal software process. We taught people PSP so that they would learn how to do things like record data, how to make estimates. The team software process was how to do this in a real industrial environment, so that you could coordinate with your team members, you could report up to management, you could make your estimates, you could see if an overall project was really on track. So the PSP was sort of subsumed into the team software process, but the team software process was for planning really big programs.

[48:10] I have some TSP data for projects that spanned multiple years. So you would look at the data very differently, and one of the things you couldn’t get from that is you’re not gonna have the same programmers developing the same thing. It just doesn’t happen. In the real world, everyone is developing a different piece of software. So they aren’t really apples to apples comparisons anymore, and you can’t treat them as such. You have to look at the data a little differently. But they record the same types of data - how long did something take, how big was it, were there defects, where did I find them, where did they come in?

Jerod Santo: That is the news for now. Of course, there are more noteworthy things going on in the software world. Which is why today’s companion email has an additional 19 links for you to comb through.

Jerod Santo: Grepping logs remains terrible

algernon first stated his public opinion that grepping logs is terrible a decade ago. The good news is he’s back, the bad news is his opinion hasn’t changed. After a quick comparison of running alike queries against 4.4 GB of JSON logs on his modern M2 SSD desktop and loading the same data into VictoriaLogs on an old Mac Mini, he finds:

What we saw here is that a computer with much more RAM, and computing power, and faster storage got absolutely obliterated by an overloaded Mac Mini that is ten years its senior. And not only in speed, but resource use too: I haven’t shown it, but working with raw text used a lot more CPU time, disk IO, and memory; meanwhile the Mac Mini barely blinked.

And not because it was all cached in memory! These are all cold queries. Cached queries are much, much faster. But how can it be that such an underpowered device runs circles around a powerful desktop?

Purpose-built software, dear reader. Purpose built software.

The moral of the story: databases are good!

Don’t grep your logs. Shove them in a log-ready database and query that. Not only will it require considerably less disk space, it will require less computing power, less RAM, and as a result, less time to perform the queries. And if that’s not enough, you can give the logs structure! You no longer need to remember the order of fields in a httpd’s Common Log Format, you can query against a named field instead. You can run aggregations, computations, build stats, correlate, join, and do all kinds of other database-y stuff. Not only does it perform better, you can do more, and do so more conveniently.

Jerod Santo: I want to show you how good this is. In the meantime, I do have a software development angle into this particular event. Here’s my software development take. So – gosh, this is really good. It’s a step change from the previous version. In fact, people were comparing this to like the Apple emoji stuff, the Apple genmoji stuff, where it will generate you things… There’s a great one as a comparison - maybe I can find it for the show notes - where it’s like “Here’s ChatGPT 4.0”, whatever new stuff. “Make me a picture of the main Severance characters, the four main Severance people as Legos.” And then the exact same prompt to Apple Intelligence, and it’s like night and day. I mean, the OpenAI one is just so good.

Ashley Jeffs: Yeah, exactly. You can’t just throw things over the wire without a response, some sort of acknowledgement, without having a compromise in delivery guarantees. But the idea of that actually being a formal understood thing… I still think people get confused about it all the time, in all sorts of places. There’s ways that you can have strong delivery guarantees, and then ruin that by adding just some extra piece to your software, or some extra thing in the pipeline, that again, kind of goes under the radar. And there’s all these other stream processing concepts… But also just the idea of like microservices back then were quite a new thing. People were working out “What does a good microservice look like?” So things like metrics logging, and the way that the config spec works, all that kind of stuff was – there’s a lot of things being discovered and learned at that point…

And then about 2015, I had enough traumas from deploying all those services that I kind of figured it would be an interesting exercise to try and solve those things in a way where you’ve got a piece of software that will do all the – I would have just called it plumbing tasks in stream processing land, because a stream processor tends to mean aggregates of data in Windows being continuously calculated. And that’s the processing part. You’re processing a stream, and the comparison is always to databases. So you want to process a real time stream as if it was just a database with static data.

But what we were doing was just plumbing. You’re just reading data from somewhere, changing it a bit, and then moving it along, passing it somewhere else… And trying to do that in an efficient way. But the main difficulty, in my opinion, wasn’t getting good throughput and the sort of quality of the data coming through. It was making those services nice to actually work with. So part of that is adding in a config language that allows you to make arbitrary changes without having to recompile, and stuff. So essentially, the things that Logstash and similar services were already doing for the logging side of things… But in stream processing world you need better delivery guarantees, better quality of service, and metrics, and stuff.

So I was kind of experimenting with that stuff as an open source project. I was also learning Go at the time, because I was using C++ up until then… And I was just messing around. I didn’t have like a goal. I was just seeing if I could do it.

Chuck Carpenter: Yeah, it’s software, and anybody potentially can do that. And I agree with that, in the sense that the gatekeepers in comparison to the lawyer, doctor and all that other stuff - now, obviously, we’re not saving lives, so… No humans are harmed by this, so of course we don’t need those kind of gates per se. So it’s a great counterpoint, I think, to that thing. But yeah, like Robbie and Jerod alluded to, or just straight up said…

[00:34:24.27] Now, the interviews, which are separate from all of that altogether, and where there are books and certifications to learn how to interview at Google, or some *bleep* like that… And I say that because we have – the badges.

Joseph Jacks: I tried to articulate it earlier. It’s extremely simple. And Zuck has been saying this publicly. So there’s two examples: OpenCompute and PyTorch. OpenCompute is the open standard, open source designs for top of rack switch, server, spec, chip design… All the data center infrastructure stuff that Meta does, okay? They open sourced that a number of years ago. My partner, Heather Meeker, helped them with the open sourcing. They have reaped billions of dollars in cost savings from open sourcing their designs for their data centers. Counterintuitively. Counterintuitively.

You would think, “Oh, wow. Why are they doing this?” They’re sharing their hyperscale designs with the world. If you all remember, Google was extremely proprietary and secretive with things like Borg, and Omega, and all of their networking Andromeda stuff… Like that was like top secret; a crazy small group of people had all the details of how that stuff worked. Maybe like 100 or 200 people at Google really knew all of the core inner workings of that. And again, this goes back to the view “The world is more efficient and better the more transparent you share cutting edge designs, cutting edge research.”

For a company specifically whose business model does not require monetizing a particular innovation that causes their business to be more efficient, they’re actually incentivized to be transparent and share that innovation with the world. They’re incentivized.

A couple of other examples would be Tesla basically saying “Look, all of our patents are belonging to you.” Microsoft also did that, by the way. This is also kind of shocking. People don’t quite recall… Microsoft opened and said “We’re giving away all of the patents in our entire–” I think 70,000 patents in their portfolio. Pretty recently. That’s an even bigger head-scratcher, to be honest, because Microsoft’s monetization model is licensing their software. And they said, “Look, we’re not going to go after anyone that infringes on our patents.” That was a few years ago as well.

The second project that Meta released that created huge derivative results and benefits for them was PyTorch. React also, by the way. React is just this enormous ecosystem at this point. So PyTorch and React, I think we’ll just focus on those two.

PyTorch allows them to both manage, shape, and influence the trajectory of the way deep learning and neural network development, hyper-optimization, and shipping work - these are artifacts called neural networks - because it’s the industry standard; completely demolished and destroyed TensorFlow, if you remember TensorFlow. PyTorch is the standard. It’s not even a comparison at this point. And then what React did was allowed them to have a similar type of influence and trajectory over the talent base of people building front-end web experiences, which are largely part of the entire portfolio of products that Meta releases. And then – okay, so again, also hugely derivative, positive talent acquisition efficiency benefits for their business, probably measuring in the billions of dollars.

[01:10:24.00] And so how much money did they actually spend on open sourcing and releasing these things? Maybe a couple hundred million dollars for both of those projects. Maybe a hundred million or so for React, I would estimate, and probably a couple or a few hundred million for PyTorch at this point.

Okay, now your question on LLaMA. Now we’re talking about not just a human thing, where you have Sumit, and Yann, and 100 or 200 people… There’s a decent payroll there. But we’re talking tens of billions of dollars in hardware that they’re buying, building… Buying and building. And they’re transparent about all these numbers. Like, they’re saying like what the numbers are; it’s tens of billions of dollars per year. Probably something in the order of 30 to 40 billion this year, and next year maybe even more. So to your question, “How the hell is this justifiable?”

Adam Jacob: If you’re pretty diehard… It’s tough right now. So we’re in the middle of basically swapping the engine… So the core part of that graph was built in a way that just wasn’t scalable. We thought it would, but it didn’t. And so we’ve been in the middle of rebuilding it for speed and scale, and that work is landing now. So I suspect in the next couple of weeks it’s going to be aggressively more stable. But right now if you check out System Initiative off main, it’s pretty tough; you’re gonna find bugs. But bugs that weren’t there five weeks ago, before we merged the new engine to main. But what you’ll see is that it’s incredibly fast in comparison.

So yeah, I’d say it’s a couple of weeks away from being stable again… And then once it’s stable again, then it’s sort of marching toward “What’s that release cadence look like?”, “What’s the SaaS look like?” We’ll probably do all of that at once, because it’s a pretty incredibly good user experience to hit a button, get a new workspace, and then just start working, which is basically what the onboarding experience of System Initiative is.

So it’s all coming. If you can’t wait, then what you should do is come join the Discord. We’ve got a big Discord full of people. You can look at the software now, you can run it, you can talk to us in the Discord, you can use the branch from before we merged the engine, which is slow, but worked, more stable…

Adam Stacoviak: You could use that tagline if you like it, Bunny. Bunny is funny, and fast. Yeah, I wish we had the enterprise account too make this comparison, because obviously… I guess within an hour you hit a roadblock, of Hitch, to build our own CDN… Ultimately, my desire would not be to build our own software. I think we’re not in the business of making software necessarily, although I think it makes sense when it makes sense… But as a media company, we’re in the position to promote those who are trying to innovate. We’re promoting the innovators. And in some cases, help them innovate by feedback loops and partnerships and usage like this. That’s where I think we really fit in the mold of the grand scheme of developer tooling. So my desire is really not to build a CDN.

Adam Stacoviak: I’m not trying to really go there necessarily, but I’m just kind of curious… If in 5 to 10 years there will be a shortage, that means we need to have more come into it, so that means there has to be desire to come into it, and potentially an environment where they’re in software development and call themselves a software developer, but then choose a lane that makes less money in comparison to other lanes they could choose to be in. So what is the environment? What are the opportunities? How does it work? Do they get great vacation? Are there other benefits? Like, what are the non-monetary, non-financial strictly salary speaking benefits of that environment for folks?

Justin Garrison: Absolutely. There’s lots of negatives and consumerism in that, but also, for me - I like how my house looks when it’s decorated. And I’m going to spend a day or half a day decorating my house. And then anyone that drives by, it’s free for them to look at. They can come – I want them to come drive by and see the cool thing that I created out of my creativity, and budget, and whatever, in my time. But at the end of the day, if they want me to make changes to it… Like, no one’s knocking on my door and says “Hey, can you get the bigger snowman next time?” Or “Can you move this light? Because I don’t like how that looks.” But that’s how software works. In open source, I put something out there and someone’s like “Actually, this flag is bad. Can you go change that for me?” No, this is not – this is mine. I decorated this, I built it, and I don’t really care to take outside contributions right now. But there is this assumption in open source that you can just go and do that, because you’re just like “Well, it’s open source. I’m gonna go add an issue and say “Please change this output to be something different, to fit my needs.” I’m like “No. Go fork it and do it yourself.” But no, that’s maintenance stuff for them. And they don’t want the maintenance, right? Because my house - no one has ever knocked on my door to say “Please change your lights.”

[00:54:08.29] It’s just like a different mindset of how we treat open source. And in this case, a scorecard gives them the visibility, like “I care about this thing.” I’m gonna do enough to provide this information to you of “Here’s the basics, and we do care about this. We’re going to try to improve it, or at least keep maintain this number.” We don’t ever want it probably to drop. But it does open it up enough for someone getting started to say “Hey, I actually want to contribute in this. I want to learn in this area. Would it be okay if I increased your score in these areas?” Or even just giving them a way to measure themselves against someone else. Like “Oh, that other repo has an eight. What would it take me to get to an eight? I don’t know. Let’s go look at that.” And it does give you that comparison as well, to know just a baseline of… I mean, grades at school are the same way. It’s like “Oh, I have straight Bs. Is that good? I don’t know. How many letters are in the alphabet? Is this out of 26 letters?” “No, it’s actually just five letters. We’re good. Just try to get the As next time.” And then this just gives you that sort of comparison level to let you try to improve, and to know where you’re not doing necessarily the best practices, or need some help.

Quinn Slack: And I’ve been reading – I love reading history of economics, and inventions, and so on… And I’ve been reading about the oil industry. The oil industry got started when someone realized “Oh, there’s oil in the ground, and this kerosene can actually light our homes much better and much more cheaply than other kinds of oil, from whales, for example.” And initially, oil was all about illumination. Make it so that humans can stay up after 6pm when the sun goes down. And that was amazing. But that’s not how we use oil today. Oil is just this energy that powers everything; that powers transportation, that powers manufacturing, that powers heating, and so on. And there were people that made fortunes on illumination oil, but that pales in comparison to the much better use of oil for our everyday lives. And now, of course, you have renewables, and you have non-oil energy sources… But for a long time, we saw that that initial way of using oil was actually not the most valuable way.

[00:46:14.09] So seeing that this just happens over and over, that a new technology is introduced and you’re not quite sure how to use it, but you know that it’s probably going to lead to something… And that’s how we always felt with code intelligence, and that’s – us getting new Intelligent Automation is so exciting for us now.

One of the really exciting things we’re seeing is because – so many people are shocked that these LLMs, you speak to them humans. They seem to feel much more human-like than what we perhaps anticipated AI would be like. We think of AI from movies as being very robotic, of lacking the ability to display empathy, and emotion, and thought processes. But actually, that is exactly how we see LLMs. I’ve seen some studies even the show that LLMs can be better at empathy than a doctor with a poor bedside manner, for example. And for us, this is absolutely critical, because all this work we put into bringing information about code to the human brain - it turns out that AI needs that same information. That AI - well, the human, if you started a new software engineering job, you get your employee badge, you go read through the code, read through the docs, if there’s an error message you’ll look at the logs, you’ll go in team chat, you’ll join meetings… That’s how humans get that information. And AI needs all that same information. But the problem is, you cannot give AI an employee badge and have them roam around the halls and stand at the watercooler. That’s just not how AI works.

Adam Stacoviak: And they keep making it more expensive too, by doing these things. Like, there’s just so many things in FreshBooks that we don’t even use… And I’m like “You’re the best at this one thing we really need, and everybody else pretty much sucks in comparison”, which is the good thing about FreshBooks… But everything else they offer, I’m like “I don’t want it, nor do we need it.” And so we have to pay way more than I think we should ever should have to for what FreshBooks gives us. And I love their software. I’m happy to pay for good software. I’m not being cheap by any means. But we’re like, it’s more expensive than necessary because they keep layering on these features.

Jerod Santo: How do you feel about Ubuntu? I know you said Canonical, there were issues when you were involved… But as an operating system, maybe in comparison to Debian… Because it does have – it’s more modern in terms of like it’s going to have more newer software on there for you, more maybe usable for a desktop experience… But what do you think about it?

Adam Stacoviak: “How exactly Cody is differentiated”, this is what you say, “over all the other coding assistants.” And you said, “Hey, they’re going to be here. They’re not going to go away. They’re inevitable.” And you go on to say “including Copilot and why we will always be the leader.” Those are strong words. You say it in episode three, and then you come up with something else. But then you said this; you said “Did I mention I’m lucky? Well, this is the world poker tournament.” I love the word you use when you write, by the way… “This is the world poker tournament of software showdowns. This is the big one. And I’ve got pocket rockets, because there’s truly no comparison to Cody. We’re so far ahead, it’s blowing people’s minds at Copilot shops. You’ll see soon enough.” Give us that soon enough. That was two months ago, basically, this post.

Adam Stacoviak: It’s something you really don’t understand if you’re really in the muck of it. If you’ve maintained a system, or deployed software, or built software… You only really understand the philosophies of open source truly if you’ve been in the mix. If you’re a purveyor or an outsider to some degree just helping things move along, it’s kind of hard to really understand the spirit of open source, if you haven’t been part of hearing people’s stories like we have on this show for many, many years, and being advocates for open source. Not because Jerod and I particularly have the kind of software you have out there used by the users you have; it’s because we understand the empathy we have for our fellow developer, and we want to see their freedoms be preserved, and the things they thrive for, and the things they put their work into be free in the way they want it to be free… And that’s what open source really is. Because my straight line - Jerod probably shared to some degree his, but I came through WordPress into open source. The fact that K2 had CSS available, and it was an open source theme - I learned what it was like to theme a website with CSS. This was back when CSS was taking over tables, way, way back in the day. And that’s my straight line into software. I didn’t go to school for it, I don’t have a CS degree… I worked for an IT company in bizdev, which I was very good at… But I started to understand open source by way of being a user; like you had said, a taker of it, not ever a contributor. And I’ve said, as much I’ve given back open source, it’s almost nothing in comparison to my usage of it. Maybe through this show and the things we do, we’ve given back…

Natalie Pistunovich: Once [unintelligible 00:28:50.13] not fast. But I think the current state of things generally, to answer your question - the comparison to Google, the way that it gives you this very personal thing is cookies. It collects a lot of data on you. Only the open data, only the data that you opt into give. Because you opt in to be on the internet, you opt in to be on the social medias, you opt in to have a cell phone, and so on.

It’s very hard these days not to opt in to give this data, and if anyone is good at that, it’s Germans, and even they give data to some level about themselves. And I think that right now, specifically, the ChatGPTs do not collect any data like this and do this personal thing about you, but one of the leading engineers/scientists/people who work there, called Andrej Karpathy - I will add his Twitter on the show notes as well… He describes in his Twitter bio that he’s working on a sort of a Jarvis at Open AI. So I think this one will be similar to what you have in mind when you think about that movie, because that movie starts with him activating the software computer - I forget what exactly - and then it kind of browses all the emails, all the internet presence he has, but also all the private things… I don’t know, things he typed and never shared with anyone. So that’s why it’s so personal. So it’s even more personal than Google right now, but it’s also an opt-in thing. And with zero knowledge, I’m guessing that this is something similar to what Andrej Karpathy means when he says he’s building a sort of Jarvis. But he also uses the word Jarvis, and not Her.

Adam Stacoviak: Yeah, I mean, realistically… I mean, you’ve done a lot of the hard work in quantifying the value of the data, and you’ve been consistent with the ability to capture it, and then report on it at a quarterly and yearly basis, which I just commend you on. Thank you for that. And you give it out for free. you don’t say, “Hey, for Backblaze customers, you can see the data.” It’s free for everybody to see. And I think you even have like downloads of the raw data, if I recall correctly. I didn’t know what to do with it, but I’m like “Great it’s there.” If I wanted to dig into it further, then I could. But yeah, there should be some sort of drive testing…

But what a hard thing to do. I mean, especially, as you probably know, models change so quickly, and the model numbers don’t seem to be like there’s some sort of rhyme or reason to them; they just seem to be like “Okay, we’re done with building that one, and now we’re going here.” And it’s also based on geography; it may be made in Taiwan, it might be made in Vietnam, it may be made somewhere else… And these things also play a role into it. It could have been something geographical in that area; there could have been a storm, there could have been an earthquake, or a hurricane, or something catastrophic, or who knows what. There’s things that happen in these manufacturing plants when they make these drives to get consistency.

[01:21:45.12] I’ve even heard to buy not in the same batch. So don’t buy more than x drives from, let’s say B&H. Buy two from B&H, two from CDW… Obviously, buy the same model, if you can, to try and keep the model number parity… But I’ve heard all these different – essentially, old wives’ tales on how to buy hard drives as a consumer. And really, it seems to be cargo-culted, or learned, from somebody else, or just fear, essentially. “This is why I do it, because it’s a fear.”

And the way I’ve kind of done it is based on the capacity, first. So I think, “How big do I need?” So I begin with my capacity. because I’m different. I want to get to price curve eventually, but my deal is “How much do I want to have? How many drives can I actually handle?” and then at that level, what’s my parity level? Can I afford to have a couple extra, so if those two fail in that parity, let’s say a RAID-Z2 given a ZFS file system array, as an example… If those two drives fail, can I replace them? Do I have two more drives to replace them if two did fail?

I hadn’t considered your cloning idea, which I think is super-smart. I’m gonna have to consider that. I might just do some hard drive failure tests just to see how that could work. That seems so smart, to clone versus resilver… Although I don’t know how that would work with ZFS, if that’s a thing or not. But capacity is where I begin. Then it’s like “Okay, for price, did I get that?” And then the final thing I do once I actually get the drives - I hadn’t considered running the SMART test right away to consider how many power-on hours it had, because I didn’t consider they’re doing tests in there… But I thought, “Well, hey, if Seagate is doing a burn-in of sorts on my drives, or some sort of test beforehand, let me know.” I would buy a model that has burn-in testing beforehand. Save me the week, if I’m gonna burn-in an 18 terabyte drive.

So when I bought this new array recently, the burn-in test lasted seven full days. I don’t know if you use this software now, it’s called badblocks… But you can run a series of tests, it writes three different test patterns, and then a final one, which is the zeros across it… But for each write, there’s a read comparison. So it’s a write across the whole disk, in one pattern, then a read, another write, then a read, another write, then a read, and then finally, a zero pass write, and then a recomparison to confirm that the drive is actually clean. But for an 18-terabyte drive, six drives, it took an entire week. And that’s just a tremendous amount of time for somebody who’s like “I just want to get onto building my thing… Come on now.”

But that’s the way I look at it. Like, that’s how I’ve learned to buy, is like “What capacity do I want to have?” And then “Can I afford it?” Just the drives alone. And then “Can I afford the extras if I need parity and replacement for that parity?” Of course you want parity. And then finally, doing a final burn-in before actually put the drives in the service… Which I feel like is a little overkill, to some degree… But you know what? The worst thing to do is to build this full array. I’m not a business, I have limited time… And then I’ve got to deal with failures a week or so later. Now, that burn-in test may not predict a week-long later failure, but it might mitigate it, because like, well, if drive four of six did not pass the sectors test in badblocks, well then, let’s send that one back for an RMA, or just a simple straight-up return kind of thing. And you know before you even build the array, you’ve got a problem child, essentially.

Tim Stiles: Yes, so for engineering DNA, which is very specifically – it’s very different from what a lot of other people do with DNA. A lot of times they’re looking for diagnostics, they’re looking for variants, they’re trying to assemble it into a whole genome for reference… What my package does is it tries to engineer DNA to be used for certain experiments and certain designs. And so what my package is mostly focused on is going from designing DNA as a concept into something that you can put into cells… Which is surprisingly unique. You’d think there’d be more of this, and there are; there’s a lot of companies that write this internally for themselves. Imagine like they’re writing their own React framework, and then not sharing it… And that’s what’s going on in almost every company that does this. They burn like millions of dollars doing this, and they don’t share back, in most cases. And so I wrote this as a thing where I was thinking, “I want to be doing this for the next 10 years, I want a really stable library, I want it to be really well tested, and I want it to have the features that I need to create cool stuff.”

[00:18:10.25] And so I started from sort of that principle, and it started with just parsers, which every bioinformatician will tell you that’s like the first thing they have to do, is they have to write parsers. It’s like the bane of our existence, is taking government data and putting it into JSON. I wish I was kidding. But after you get past that, then you get into stuff like, “Well, how do you manage having a hash ID for a circular sequence?” Because the string is circular; there’s no way of telling where it’s going to start officially. So it turns out there’s this old algorithm called [unintelligible 00:18:39.27] that given a certain sequence, a circular, it will always bring it to the deterministic point. And we use that to create hash IDs for these plasmids, because otherwise it would take an inordinate amount of time to figure this out. Because then you’re using like pairwise alignment algorithms… And these are not linear algorithms; they’re kind of gnarly, and not my specialty. People actually specialize in like string alignment algorithms for this specific stuff. [unintelligible 00:19:01.16] a professor at Harvard Med specializes in this, and his software runs the world quietly. He does not get enough credit for how much bioinformatics relies on his stuff. The practitioners know, but the people downstream have no idea that this one Harvard professor is keeping all of our – you know, if you’ve ever used 23andMe, Ancestry or anything else, or any genomic testing ever, your DNA has probably gone through one of his algorithms. He’s very prolific. And unlike myself – you know, I still have a small user community. I have some very hardcore users that have their own startups, that are raising money for it, and I have a handful of people that give me feedback, and I’ve even had large companies go, “Hey, we’re thinking about using this”, but I still feel like a small player in comparison to people who have come before me. Again, I wrote it because I want to engineer DNA, and I think it’s super-cool, and it was hard to do that with the tools available.

Ariadne Conill: So by comparison, apko is a tool that we developed at Chainguard. And what it does is it writes apk and uses apk to generate an OCI image. So you can have like a YAML file that says “I want these packages, from these repositories, signed with these keys.” It’ll go out and it’ll set all of that up, and it will build the image, and it’ll include all of those

various configuration files behind, so that you can then take a scanner and see what is actually in the image. And that’s a really big thing, because a lot of people who make small images, they use from scratch in a Docker file, and then they copy everything over. And if you do that, then your security scanning doesn’t detect anything. And that makes it a nightmare for your security team, because they have no idea what’s in these images. They don’t know how to remediate anything. They don’t know what to tell their developers to fix. So it’s very important that everything in an image is cataloged somehow.

[10:08] Basically, the thing about apko is that everything that goes into an image generated by apko has to be an apk package. And as a result, everything is catalogued in some way. So that brings us to SBOMs, the software bill of materials. That’s kind of the same mantra applied to all of container images at large, all of IoT, all of like industrial computing. And basically, the idea is that all software needs to be catalogued, and these catalogs need to be in standard formats, and all of that. So basically, the deal about apko is that it makes all of that really easy. And in fact, apko can generate an SBOM for you, based on inspecting the data that apk leaves behind when it generates the image. So that’s the difference between the two tools. One is the package manager, and the other is the thing that wraps the package manager, builds images, builds SBOMs, and all of that.

Sebastian Kister: Like I said, the platform itself, we have many workloads that develop that kind of stuff… But the connected car stuff, the in-car software, that actually runs in the car, in a productive state and customers are using it - that needs to run somewhere else, and on different kinds of systems. You don’t want that – like a comparison with an airplane, you don’t want to ride an airplane that has an overdue upgrade. Maybe. I wouldn’t…

Rob Mee: Yeah, I think we had a really interesting opportunity to take teams that we’re working in as optimal a way as we could figure out - incremental, evolutionary, building software quickly and getting it into production as fast as possible, getting feedback on it, and iterating. We were in a position to build a technology that was tailored to that, and to do it by interacting with those people. And in fact, building the technology itself using those processes. So it really was – you sort of couldn’t design a better crucible for heating and hardening something like that.

And at the same time, building a platform like that - they become quite expensive. I mean, it’s not just the core runtime, and all the tools that you have to build, but it’s all the integrations with technology, and the services, and so on. So team can really expand, as ours did. And so you end up having the development of a product like this push the process by forcing you into a situation of scale that you’ve never had, in terms of team size. And then the process itself is also essentially continuously critiquing the technology product.

One of the first things that I said to the Pivotal Labs team, after Pivotal Software was formed and we had sort of a nascent Cloud Foundry that we’d inherited from VMware, that wasn’t really in a production state yet, but we had it, and we decided “This is really the thing that we ought to commit to and build this out, because it’s going to be amazing…” And the Pivotal Labs teams were a bit apprehensive, and understandably so, because they said, “Are you going to force us to use this product now that we’re building the product? Of course, we’re going to have to use it, and our clients will have to use it, and so on.” And I said, “I promise you, you won’t have to use it until you think it’s the most appropriate thing for you to use, in a given situation. So you can make that decision.” And then we had to work really hard to make it so. But it was. I mean, it did become that thing.

And especially as Pivotal, in sort of its new incarnation is Pivotal Software shifted from working with 80% to 90% startups and internet technology companies, into working with 80% to 90% Fortune 500 and global 2000 companies, and the federal government, and so on - a very different type of client - Cloud Foundry became the saving grace of all of our engagements. Whenever we couldn’t use Cloud Foundry, the difficulty was profound in comparison.

[47:55] So the biggest advocates and champions for Cloud Foundry became the Pivotal Labs teams, because working without it was extraordinarily difficult, especially if you were building software that was deployed on-premises. But even if you were deploying on the public clouds, it was still substantially more difficult and painful to do it without Cloud Foundry.

Adam Stacoviak: Gotcha. We have changelog.com, and we always get compliments on having that domain. It’s a similar story, in that there’s a story. We didn’t know a guy, but I just looked, and thankfully, the person who did own it didn’t shy away from whois. They put their name on there, and they happened to be on GitHub as a user, and in tech, and in software. This is earlier days for us too, so we’re actually less of a business then, more of a hobby, and have since turned into a really great business, of course. But initially, I was like, “Hey, we’re doing this thing. I’d love to use the .com. I’m cool with using some sort of domain broker.”

They were like, “That’s cool”. We named a price and… I don’t want to tell you the price, but it was super-cheap in comparison to what it could have been, should have been. If I bought it today, it’d probably be unaffordable, basically. But several years ago, we probably got it maybe eight years ago, I want to say. I don’t even know what year it is anymore. It’s just 2022 barely for me, but I feel like time flies just so fast since the pandemic. But it was pretty long ago we got the domain.

I think there’s always some sort of story to your .com, and having changelog.com for us, it really got uniquely better and different, because we were the changelog.com before. Before that, we were changelogshow.com. It sort of legitimizes a thing when you actually have the .com as your core brand name. So I can imagine that it felt the same going from .tv to .com for you guys.

Solomon Hykes: Ha-ha! Yeah, I mean - the comparison to containers is good, because it’s about consistency, and “Hey, it works on my machine; why doesn’t it work over there?” But it’s applied to a different area. Instead of applying it to the application or applying it to the pipelines that ship the application.

We’re doing it because we spoke to people, and they complained that it was a terrible experience, so we got pulled into that. But then it made sense to also… It just kind of makes sense when you look at the current state – the state of the art, I guess. Because there’s so many different ways to organize an application. I mean, there’s literally software for everything. So it’s very hard to standardize everything about your application, to force people to do it all the same way.

So there’s a lot of component reuse, there’s a lot of standardization, but it’s very fragmented, because there’s so many different kinds of software. The pipelines are pipelines. So instead of a degree – it’s one degree removed from the application itself, so if you look at a banking software and a video game and whatever is monitoring your robot vacuum cleaner… Incredibly different software, different architecture, different languages, different everything. Different people even, different specialties, but one level removed, they all need pipelines to deploy that stuff to the cloud. And the pipelines - they’re different, but they kind of look the same architecturally. So that’s kind of where we’re focusing. It’s an opportunity to standardize and just waste less time repeating yourself, and we just think you can standardize more of it… Because there’s less of an excuse to not do it the same way as everyone else.

There was always an excuse to not use Docker. Or use it differently. So Docker and containers actually – you know, they ended up being a very widely adopted technology, and they still are… But it was never 100%, and it never will be. There’s lots and lots of software that don’t run in containers. And within containers then there’s all the competing ways to run containers, and thank God at least we standardized the base image, but we didn’t go much further than that.

There’s images, and there’s pods, and then there’s of course all the orchestration stuff, and all the – then there’s Kubernetes, and then there’s all the different fragmented ways of doing Kubernetes… So it’s like a fractal, and it’s only gonna get worse. So there’s limited reach. You can help, but you can’t help through ubiquity. So in a way, the original vision of Docker in that way was not realized, because we cannot standardize as widely as we hoped… But with pipelines I think we really can. Because like Andrea said, how many different ways of building a React app do you need? How many different JavaScript frameworks do you need? Yeah, a lot. Yeah, sure. That’s normal. But once you’ve picked one, how many different ways do you build it and integrate it into a pipeline? Hopefully not hundreds, because that’s too many. Anyway…

Adam Stacoviak: I totally agree with everything you’ve just said. It was a challenge even been pitching this idea for this show, because I had paid attention to Pieter on Twitter, had been paying attention to what he had shared, his blog posts and whatnot… And terrible news, obviously. He had obviously family, I believe he had a daughter as well… And as someone who is also a father, I just looked at this main story and just thought – just total sadness for him. But then I was like, “I would love to know more about how he’s living.” Because his whole idea was how to pass on the software in a way that didn’t interrupt everyone else’s ability to use his creation. He respected the process of open source and of software so much that it really mattered to him to spend time with people like us, on our show, to share what was happening; this protocol basically for dying, how to do it well.

So I had this thought internally, like “I want to get this kind of a show. I would love to do this show.” I had no idea what it might be. I’m like, “There’s no way I’m going to reach out to them. There’s just no way.” And then I saw him tweet, like “Hey, if you want to reach out to me and talk to me…” Like, he invited people to reach out to him. And so then I was like, “Okay, great. I can do it without being like a total jerk.”

So if he hadn’t had that invitation, I probably wouldn’t have emailed him or got in touch. I would have just wanted to do the show and we just didn’t do it. But he put the invitation out, I reached out and he’s like, “Sure, let’s do it.” And then we talked basically, like, as soon as possible, because of his situation. And I remember near the end of the show even, that we were like – like, e say goodbye to everybody on the podcast, right? A podcast is normally just a podcast. There’s no seriousness to this point of anything. In comparison to life and death, most of what we do on a podcast is pretty meaningless, in comparison, to life and death. No one leaves our podcast and passes away days later. Pieter did.

So when we were saying goodbye on the show, it was challenging. Like, even almost now I’m a little crackly, because we were really saying goodbye to Pieter… Because one, I’d just met him, never met him before, had a deep conversation about his life and his legacy, what mattered to him most, and his daughter and his family, and all his plans… And saying goodbye at the end was like “We’re literally saying goodbye to this guy.” And I know Jerod almost cracked up in terms of tears.