Justin Searls: Yes, that’s a great question. And I think that a guiding light for me on the most successful teams that I have either been a part of or that I have witnessed, has always been a shared and common just understanding of what their purpose was. So I was part of an organization, a consulting company, just prior to founding Test Double. So we founded Test Double in 2011, so it was like 2009-2010. And they were in that era where it was known as like an agile software consultancy. And so they were peddling, pushing their own kind of blend of agile engineering practices like Scrum and extreme programming. But they did an interesting thing in their sales process of really pushing business value.

And so if user stories rolled up into like Epics – Epics, in their sales parlance and also in how they practiced and delivered, would roll up into business value stories, or value stories. And we would start each engagement by actually getting the whole team in a room - developers, QA, product owners, business stakeholders alike… So it wasn’t behind some secret veil of like a product organization. I didn’t even know that might be considered desirable in certain organizations; I was sufficiently naive to this experience. And what was great about it was we would just have an open and honest put up on the board, like “Hey, executive or stakeholder or person who brought us in here to like build this thing, how is X, if delivered as conceived, going to make or save your company money?” And just boil it down.

And first of all, a lot of executives, it turns out, are uncomfortable with being put on the spot to answer what shouldn’t be a simple question such as that… But when you really sat with it, and as a team forced the conversation out, and then you followed through, not just on – I don’t know… Here’s a project example that we did - currently, our system is so slow that sales reps who go to restaurants to sell food supplies, end up just spending multiple minutes just waiting for pages to load, and they could hit three or four more restaurants a day if it was fast. And that would result in like X dollars. And we’d follow through and be “Okay, so what is X? How would you measure X? How will we assess that X has been attained after we’ve delivered it?” And not only in the kind of initiation phases and discovery of the project, but how will we, on an ongoing basis, track that as the primary metric for success for this project, as opposed to arbitrary story points, right? Because there’s no way to know whether you’re going in the right direction or the wrong direction if you don’t have a shared understanding of what the point of the thing that you’re building is. And most software teams, they don’t know what the point of the thing that they’re building is. Or in this day and age, to know it would be to not want to work on it anymore… You know, whether for ethical reasons, or just because it’s a lot of the stuff that gets built these days is kind of slimy.

And even though, in my practice at Test Double, our clients - they work on fantastic and wonderful products - I think that we have sort of been encouched into this default relationship where product throws “Here’s the features that we want” and “Here’s the things that we need.” There’s a disconnect at the developer level, at the team and engineering level, where we lose sight of or aren’t really bought into or aren’t really included in the discussion of “But why?”

[12:11] And I have seen teams where developers know the answer to “Why”, and when a product owner says, “Hey, here’s how these comps should go… And you click this and then you click this, and then you click this”, a developer who knows what the ultimate goal is, in terms of like business value or whatever the overall organizational objective trying to be met is, can successfully have a real two-way discussion with that product person and push back, or offer alternative ideas, or even find shortcuts that would make things faster. And in the absence of that, everyone just becomes an order taker… You know, I receive these marching orders and then I go and build the thing. And I think that sleight of hand is what actually facilitates and enables a lot of the negative externalities that we see in our industry.

Mat Ryer: It’s a much smaller one. The thing about Testify is one of the patterns for the open source thing was if somebody made a contribution, then they would just add it as a contributor to the project; this was an idea that was a bit popular at the time, a bit trendy… So anybody that needed to assert some things, like “I wanna assert that this slice of string contains a particular value” or “I need to assert that this value is within an average, within a range of values” and all these sorts of things, the kind of floodgates were open, to some extent, and they all got added.

If you look at Testify today, it’s a massive API footprint. It just has kind of everything. The nice thing is it all works really well, because it’s been used a lot and tested a lot. But I’ve just found it was too big; even I couldn’t really figure out what I would need to use, and I ended up just using two or three of them. So that is a project that’s basically taking just the few – it’s like a really minimalist version of Testify… But it’s really the same things. It gives you single-line assertions, and that’s it. So I tend to do that… Or if it’s a simpler package, I may just use the standard library, if it’s suitable.

One of the other things with the standard library is when it fails, you have to provide an error message and include the values, and things. That’s where it gets a little bit repetitive, because you’re saying “I expected this, but I got this.” That’s what you care about when a test fails. So these packages let you just pass in the two values, and it will print out if there’s a failure. So that’s really all it’s doing to help…

There are some people – Francesc Campoy is classic; he has a go at me for Testify all the time. He really doesn’t like these assertion patterns, and stuff…

Lucy Lu Wang: Yeah, I think the domain knowledge is one of the greatest barriers for working on this dataset… And thanks for mentioning the CoViz project. That’s a tool that was released by Allen AI for exploring this dataset in a slightly more meaningful way. And for that tool, we essentially ran models to perform extractions of these entities from the text; entities of different classes, like drugs, genes, diseases, phenotypes, things of that nature… And created a vizualization to allow you to browse the relationships that are most prevalent between pairs of these entities.

So that’s a great way to explore what’s in the dataset… There’s also just exploring the articles; we have a CORD-19 explorer to help you do that. But in general, I think unless you’re willing to spend a couple years of your life in medical school, it is very hard to understand what some of these terms mean. Certainly knowing what class or category of entity is being mentioned is important… So knowing something is a protein, knowing something is a receptor, knowing something belongs to a particular biological pathway - these are key for gaining an initial understanding of what is being said in a text snippet. But that’s also why we need medical expert to assess the actual utility of some of these extractions.

Mat Ryer: Yeah, that’s what I mean though - is it worth having? Is it worth getting promoted? Because I feel like testify made the case for that style of testing… And to be honest, if you look at the API for testify, it’s really grown. It was a very liberal project, where anyone’s contributions were very welcome. And what’s happened is it’s now quite a big package. It has its own dependencies, quite a lot of them… And really, like you say, people are just after the assert equal, and a few, a little handful of very common things. Could that not be on the t, so you could do t.assert, or is it just not worth it?

Charlotte Spencer: You should probably assess the skill levels of the people. Hoodie had done events where it’s your first ever contribution, but they’ve also done events where “Hey, it’s just some friends who are quite confident in their abilities hacking together” and a shared goal. So know the comfort level of the people who are coming to the event first, and be specific about what kind of event it is going to be as well. If it is an event for beginners, then organizers should be prepared to be spending a lot of time helping people. But if it’s just a casual hackathon thing where at the end of the day you might have made a pull request, but it’s also just kind of a bit of a get-together, then it would definitely be less hands-on.

Again, we go back to intention. What is your intention? Know that from the beginning; communicate with the project, and then it just comes down to admin, which is difficult… But once you know the intention, it’s easier to do.

Julian Shapiro: Libscore came about from two things. One was after marketing NameLayer I realized how important it was to actually assess how effective your marketing is. It seems self-evident, but when it comes to open source, there was actually no way to do that. Even if you are not marketing your open source project, let’s say you just get it out there on GitHub - you have no idea who’s using it. If you scroll through the list of everyone who’s starred it, maybe you can find in their bios they’re the employee of Yahoo!, but that’s not a representative metric, that’s not enough. A lot of people don’t star, a lot of people use npm, so in open source in general, I just had no way of figuring out how effective was my marketing. All I could track were page views on VelocityJS.org. So I knew I needed to close that feedback loop that any growth-minded entrepreneur would wanna close. That was the first thing. That was like the seed of the idea.

At some point, I heard that Stripe was having this open source grant program which hadn’t been done before. It was a for-profit company saying, “We wanna sponsor interesting open source developers, working on projects that we think will have an impact, and maybe tangentially related to the work Stripe does.” I got the grant, and they took me out to San Francisco.

On that grant I realized I’m actually mostly done building Velocity. It was at the point where it worked well. It wasn’t encountering much bugs, and so I thought, “Well, maybe I could actually use this as an opportunity to pitch Stripe on a brand new project, something else.” And I told them the story about how I marketed Velocity and how I wanted to track its usage across the web, but doing so seemed very intimidating. It didn’t seem like something I could do in my room.

[40:09] I felt like I needed to be with a bunch of very technical people; I needed the processing power, I needed some expertise in scraping, so I took advantage of all these people around me at Stripe to put together a small team and a little bit of sponsorship to build Libscore, with the intent of scraping the top million most trafficked sites on the web, to determine which JavaScript libraries those sites were using.

It turned out that part was actually very easy. At that point in time module loaders weren’t very heavily used to the point where they were obscuring the footprint in the browser of which libraries were in use, so I could just sniff all the top-level variables and run a bunch of heuristics to figure out, “Is this variable random-leaked stuff, from code that was messily written? Or is this the exposé endpoint of an intentionally installed third party JavaScript library?” So I would try to figure out how to do that, and then I would extract that list from the web page, and then index that across a bunch of things, put that into this big database and then make that searchable.

That was the intent, but all I knew how to do myself was that scraping portion to figure out which libraries were in use on the page. So I got together with this incredible designer named Jesse Chase. He was the creative director of Digital Ocean at the time, who built the whole UI and the graphing system. And then Jason Chen, founder of another open source project called Quill, which is a rich text editor. He built the actual backend architecture and the system that scraped the web. And collectively we built this thing that finally gave a footprint to the impact of open source projects, whereas it prior did not exist whatsoever. You didn’t know who was using the project. So the outcome was not just an idea of, “Okay, great, people are using my project”, but also “Oh, wow, Uber.com, WhatsApp, Twitter - these companies that I use, these sites that I go to every day are using my thing. That’s really cool.”

So what I saw was Libscore motivating developers who may not have had that insight into who is using their stuff to be like, “Oh wow, I really wanna make sure I continue maintaining this. I want to merge these PRs, there’s a lot of people depending on this, and now I know who that is.” And it provided that tangible footprint, which was pretty exciting.

Karl Fogel: Yeah, I think the past ten or fifteen years was a time of experimentation among CLA. For listeners who don’t know, a CLA is a Contributor License Agreement, which is basically like there’s some upstream open source project that you wanna contribute some code - maybe you fixed a bug or added a new feature for them - you send in your changes, but they want you to also send in some kind of assertion, digitally signed or maybe fax in an actual real signature saying that you are giving the project this code donation and all future donations made via the same mechanism under a certain open source license or under certain terms, so that they can incorporate it safely without any fear of legal repercussions later into their codebase. The idea is later if you change your mind and you’re like “I didn’t give you that code to distribute under the GNU general public license, I’m suing you now”, they can say “Well, we have the CLA that says you can’t turn around and do that.”

What some companies started doing - and they’ve mostly stopped now, because this got unpopular, although a few still do it - is they would have CLAs that would say “You agree that you’re donating this code to project XYZ and that we, company Q, which is a major sponsor or the founder of project XYZ, are allowed to redistribute your changes under any terms we want.” That includes the open source project license to the project, but it also means that they could make a proprietary fork of the project. Some of these companies did that in order to retain the right to do a proprietary fork, or sell licenses to, or something like that.

Those kinds of CLAs have gotten pretty unpopular, because a lot of developers just said, “Well, I’m not gonna give you asymmetrical rights. I’m giving you code under the license here, giving the world, including me, code under license. Let’s just keep it symmetrical and not give you rights that I don’t have.” This became objectionable enough that then when a company or a project would set up a CLA of that style, they would just immediately be noise.

What surprised me though - this is something I discovered during the research for the book, and I have to give a shout out to Bradley Kuhn, who follows this stuff; he’s at the Software Freedom Conservancy, and was able to tell me a lot about what had changed in the CLA landscape and point me to examples. It’s that not only have those particular odious kinds of CLAs become less popular, but CLAs in general have become less popular. More and more projects have just said, “Look, as long as you certify that you are the author of this code or that you have the right to contribute it under our license to the project, and that we can redistribute it under that license, then we’re good.” So that’s not really a licensing agreement for the contribution, it’s called more of a DCO (Developer Certificate of Origin) where you assert, and usually an email or maybe a digitally signed document of some kind is enough to just say “You have this code. Here’s my DCO, and now we don’t have to sign anything or have an agreement.”

[20:20] So I think the world is moving more toward DCOs. There are still CLAs out there. Some projects have important reasons why they need a CLA; for example, apps that are gonna be distributed in the Apple App Store but are free software under a copyleft license, there are various things about the Apple build and production process that get things in the Apple Store where the terms that the project has to agree to with Apple are not compatible with the GPL, so the developers who contribute all have to sign a CLA with the project, where that gives the project enough of an exception to the GPL to be able to sign this agreement to get the thing in the Apple Store, but otherwise the project is under GPL.

There are some cases where CLAs are still necessary and many other cases where projects still use them and people generally agree with them, but I do a general move away from complicated or onerous CLAs and towards simpler, more lightweight things like DCOs. I think - although I should stress that this is not legal advice to anyone - my business partner and friend James Vasile, who is a lawyer, has observed some of the same trend and clued me into it, so I should give him some credit for keeping tabs on this as well. Does that answer your question?

Cory Doctorow: Sure. As I mentioned, I’ve got an eight-year-old and she was born in London in our flat in a pool in the living room. She likes to hear the story of her birth, right? So we would tell her the story of her birth… My wife told her she came and yanked on my arm and shouted, “Story arm”, that I would tell her a story. So she runs up whenever she is bored and she yanks at my arm and yells “Story arm!” And being a writer, I like to iterate when I tell stories, so I’d tell them a little differently every time. I’d start a little further back or go a little bit further forward.

What I realized was that the interesting part of her birth story was the stuff that led up to her birth, and not like the stork stuff, but the stuff about how my wife and I met and became best friends and lovers and a couple, and got married and decided to have a baby. You know, all of that stuff is unique; the stork stuff is the same for everybody. What your parents did to make you is almost certainly something I can guess at with a pretty high degree of accuracy, right? But how your parents came to make the decision to do that and make you - everyone has a different version of that.

[20:08] The open source version of how we got here, we talk about the licenses and we talk about the packages and milestones, but there’s like this really strong social component to how we got here, because around the same time that the open source movement was starting, it was also around the same time that the open web movement was starting, that we were sun setting these proprietary network architectures - whether those were the ones the phone companies ran… You know, AT&T circuit-switched services-centric network or the big commercial services like Compuserve in AOL… They both kick-off around the same time, and yet the open web has collapsed, the open web is almost dead.

We are in a desperate and dire moment for the open web, and the free and open source software movement has soared; everything, including the things that are closing down the open web is built on free and open source software. So that is an amazing thing, and the speech kind of interrogates what the difference is, and how one soared and the other sank, and what we can learn from the free and open source movement to keep the web open as we try to open it up again.

I think the thing that the free and open source software movement had going for it is this thing called the Ulysses Pact. The story of Ulysses goes that Ulysses was gonna sail into siren-infested waters and anyone who heard the song of the sirens would be tempted irresistibly to jump into the sea and the sirens would drown them. So normally, when sailors sailed into the siren sea, they would fill their ears with wax. But Ulysses was a hacker and he wanted to hear what the sirens’ song sounded like, so he had his men lash him to the mast so that he could hear it, but he couldn’t get loose. So what he used was his strong self, the moment at which he was strong, to predict that in a future moment he would be weak and to take countermeasures to prevent himself from giving into that weakness. We use Ulysses pacts all the time - if you go on a diet, you should throw away your Oreos away on night one; not because you’re like incapable of resisting temptation, but because everyone sometimes has moments of weakness, and the strongest thing you can do is to recognize that you will have a moment of weakness in the future and take a countermeasure against it.

In the free and open source world, our Ulysses pact is the irrevocable license, because the failure mode of free and open source software, having founded a free and open source software company, I can tell you is that there are moments in which it feels like your survival turns on being able to close the code that you had opened when you were idealistic. There are moments of desperation when that happens.

Of course, it’s ridiculous, because if you’re making anything substantial under free and open source software, you’re building it on other things that other people have opened and can’t close, and if they were to close off their code your project will collapse. So every one of us wants to be the only one who can revoke a free and open source software license, while all the plumbing that we built on top of stays open. Because the licenses are irrevocable, because you can’t close it once you opened it, you generally don’t even get the pressure from your investors or from potential acquisition suitors or from other parties who can otherwise lean on you and put a gun to your head - they don’t even bother, because there’s no point in shouting at you to close the code, if they know that it’s not course of action that’s even open to you.

[23:44] So even though the same desperation that led us to close the web is present for everyone who’s ever made an open source project that succeeded, that desperation can’t express itself in the same failure mode that the web has had. So my talk is about how we can build a Ulysses pact for a newly opened web around two principles that will keep the web open even in the desperation of its founders, even when the pirates who founded it become admirals.

The first principle is that any time a technology or computer gets an order from its owner that conflicts an order that’s been given to it by a remote party, the owner should 100% of the time without exception win. The owner always gets to overwrite remote policy.

The second one is that any true fact about the security of a system that you rely on should always be legal to disclose, under every circumstance. My pitch is that these two principles should the principles that we become zealots for; that if they’re not calling you an unrealistic idealist about your adherence to these principles, then you’re probably not trying hard enough. So my pitch is that the people who care about building an open web to be the nervous system of the 21st century, to have an Internet of Things that’s not an Internet of things on fire that spy on you and ruin your life, is that we need to like take these principles and cherish them as much as we cherish the core principles of free and open source software, and weave them into our licenses, into our professional codes of conduct, into our membership agreements, into every single piece of what we do, so that there’s never any question that this will come about.

There have been lots of times when the governments have tried to pass laws that say “In order to make software, it has to be closed” and the fact that there’s all of this critical, open software has meant that those laws died every time. Because you’re going back to them and saying, “Well okay, but what you’re talking about is throwing away all the infrastructure on which the digital world is built. What are you planning to replace it with when you pass your dumb law?” You know, reality asserts itself, and so if we can create a reality on the ground to assert itself when governments contemplate stupid laws that say that remote parties can override local parties, whether those are crypto backdoors, or DRM, or lawful interception overrides, or any of the other things that have been the parade of horribles of the 21st-century, then we can make a difference.

Olivier Lacan: I feel like – I wish open source developers would basically crank up the empathy on their open source end users the same way that iOS developers sometimes do with their paid customers… Because yes, people don’t pay, but if you keep such a well-curated list of all the changes, first of all it encourages people to do the same, it makes it easier for people to jump in and contribute…

For instance, I think Haml was an example… I was waiting for a feature in Haml or a feature in Redcarpet, in one of those things, and I was about to start working on it, and I thought “Okay, so it’s not listed in the readme, I can’t find it anywhere”, and I went through the changelog real quick and Boom, there it was. Footnotes were added I think recently in Redcarpet and I was super-excited because for my blog - just little footnotes, and you could do that very easily… And I didn’t have to go any further.

Of course, you can improve your feature descriptions on your library, but it’s also great for people to just jump in and see, “Wow…!” Yeah, when you jump in a project, you always assess “How well-maintained is this project?” What better way to see how well-maintained a project is than show “You know what - every month, every week, every year we have these incremental changes, and this is all the things we carefully change and add.”

Scott Ford: Yeah, because there are a lot of problems that we’ve seen over the years that many teams have been facing, and I do think there’s a market for building solutions to help teams solve those problems themselves without having to hire an outside contractor or an outside team. And so there are aspects that I think could be productized. And we’ve gotten started a little bit on one product… We’ve been working on it for a couple years, don’t really have – we’ve got like an alpha demo that we’ve shown to people and I’ve gotten some feedback on.

[01:12:00.27] We’re still kind of working – we’re hoping to have a beta out; probably first quarter next year is kind of realistic for having something that people could actually sign up for and give us better feedback on. That’s called Freshly. It’s around analyzing dependency freshness, and looking at how fresh or out of date software dependencies are, like third party dependencies; most of them open source dependencies. And really assessing the quality of an application or a project from that perspective.

We also wanted to be able to assess at multiple levels of the – you mentioned, Adam, that you’re not a big fan of supply chain as a term for this ecosystem.

David Kanter: Yeah, absolutely. I think you’ve sort of hit the nail on the head, which is that we want to be providing and drive the industry forward with these datasets. I think of our speech project as ImageNet for speech in shorthand. That’s the 30-second pitch to someone in ML. And as I look out, there’s a lot of other areas that could use this sort of public dataset. Now, my colleague [unintelligible 00:26:00.20] who’s a professor at Harvard - he did a study with his graduate students, and one of the things he found is that even at companies that have huge amounts of internal data, like a Google and an Alibaba, most of their research studies were done with public datasets, because that gives you a level playing field to assess accuracy. And you get reproducibility, and all of these things. But we wanna push forward with datasets in all these different areas, because that will ultimately democratize access to this technology, and thereby benefit everyone.

So that’s the vision… And as you say, the People’s Speech is our first dataset. Late last year we actually started sampling it to partners. So two of our members have speech teams, and - you know, if there’s anyone out there with a speech team that’s interested, this is… I think it’s about ten years’ worth of audio.

Allan Jude: Yeah, they’re forks of the same original code from the very, very early ‘90s. And so being 35 years later, a lot has changed in between there. So they are part of the same family tree, but the kernels are quite vastly different. And kind of the goals of each of the projects is very different. So NetBSD is really designed around portability. Especially back in the early days of Unix, that was one of the big things. Some people had like a PDP, and then they had faxes, and all these different computers that were completely different architectures. Before the invention of C, every time you had a different kind of computer, you had to rewrite the operating system to like the assembly language of that computer. It wasn’t like we have now, where the x86 and ARM are the two main kinds of computer, and we don’t have to deal with very many others. There’s PowerPC, and MIPS maybe, and a couple things like that. But we’re not dealing with hundreds of different architectures. There’s three, four popular ones.

So NetBSD wanted to maintain that, and works on all of those weird architectures that we just mentioned, and a bunch of ones you’ve probably never heard of, and a bunch that only existed in Japan, and never shipped anywhere outside of Japan. So NetBSD’s focus is really that portability, and being able to run on anything.

[00:35:57.26] A lot of their ethos from the beginning was “There were these bigger mainframe type machines, and they don’t make an OS for them anymore. We can be the open source OS, so people can still use to run these.” And that has a lot of value, but it doesn’t have quite the same mainstream appeal as running commodity servers with regular applications.

But NetBSD did pioneer a very interesting thing called package source, PKG SRC. And this is a package manager that’s not specific to one operating system. So it’s supported on various different versions of the BSDs, Linux’es, Solaris and its derivatives, Lumos and so on, and it allows you to get some kind of basic applications on stuff. People who are like “Oh, I need modern Python on HPUX”, PKG SRC is a way to do that. And that came out of the NetBSD project.

And then OpenBSD has really had a focus on security, and is kind of like a research operating system where they’re trying new concepts and ideas to make more secure computers. So they invented a bunch of common things we do. They’re the maintainers of OpenSSH, that every operating system uses as a way to securely connect to a remote machine - that’s maintained out of the OpenBSD project, along with lots of other applications. They have their own BGP server, and they’ve maintained a BSD licensed version of rsync, and lots of other applications that you might actually use.

But they’ve also pioneered a lot of concepts. They have a system called Pledge, where an application can say, at the start, after it’s done its initialization where it maybe needed root access for something, it can pledge that it will only use these system calls from now on. And if it ever tries to do more than that, because maybe somebody used a buffer overflow to try to make the application do something it’s not supposed to, it will actually – the operating system will say “You promised you wouldn’t do that, and we’ll stop the program.” And this allows you to kind of sandbox a program.

FreeBSD has something – it’s not exactly an alias. It’s called Capsicum, and it’s a lot more fine-grained, but it means it’s a lot more difficult to retrofit existing applications to that more locked down sandbox. And it works very well if you’re writing a new application and design it from the beginning to support this compartmentalization. But Pledge is a very easy way to retrofit existing applications to make them safer.

They also invented a couple other concepts. They relink the kernel, all the different parts of the kernel together in a different order every time you boot, so that everybody’s machine is different, and an exploit won’t be able to just work on every OpenBSD 7.4 machine.

Christopher Hiller: Yeah. I mean, I’ve I think I’ve had good success with Copilot. I haven’t tried to use ChatGPT directly for coding, but it’s cool that they have – I don’t know what they do now, but it has some sort of understanding of the project you’re working in. And so I would be like - I’d select this class, and be like “Okay, refactor this class in this particular way”, that would be tedious to do manually, and it does it. And it uses my types, and everything; it tries to keep the same style. And with unit tests and testing in particular, it knows I’m using a weird, a non-popular, just kind of an obscure assertion library. But it knows all the assertions I’ve written before, and it’s able to figure out what to use. So it’s not like I’m asking it to write unit tests and it’s going and using try syntax, or [unintelligible 00:21:45.06] syntax, or something… Because it doesn’t; it knows what I’m using in it, and it does it correctly four out of five times.

Lucas Fernandes da Costa: So the first thing I advise is that we look at that problem from when we set that objective; from the first moment we start that project, I will start looking at what our metrics are… Not only our metrics, but like getting qualitative assessments on how we’re performing against that. Not only when we see we’re not going fast enough, but I would monitor us throughout the whole process, so that I can see as early as possible when something goes wrong. I think that’s the first thing.

Now, let’s imagine, as you’ve mentioned, that things have gone wrong at some point, right? So we’re in the middle of the project, something has gone wrong. So first, I’d try to make a qualitative assessment and look at my cumulative flow diagrams to see whether my assessment makes sense with the metrics that I’m seeing. Maybe you’re seeing a huge increase in tasks in the testing stage. Maybe that’s the problem; maybe things are queuing up in tests, maybe things are queuing up in deployment. And then, if that’s the case – imagine things are queuing up in deployment; you see a huge increase in the size of that bench. Well, then it means you’re accumulating lots of tasks to do that deployment all at once, right? Because that’s too painful; you’re letting things accumulate, so you can do a large batch transfer to production. Maybe that’s a problem. And then when you start adding up loads of things - well, you have a larger delta for errors. So that’s one way in which the system can be broken.

Now, there are plenty of ways in different parts of the system that things can be broken, so I’d look at a cumulative flow diagram to see the size of the batch to see if anything’s queuing up, to see if there’s any particular parts of the process that are painful, and try to get some more predictability, try to make things more uniform, so that maybe we’ll see, “Well, we’re not going to be able to hit the deadline with all these features that we initially set. so we need to make a compromise.” Right? Because one of the things that people get wrong, I think, is that they can simply turn the dial and things will go faster. But every system will only produce what it can produce. If you just tell people to work harder and do more, is not going to make any difference. No one cheers for the CPU to process numbers faster. Systems produce what they can produce, regardless of whether they have a goal.

Egon Elbre: One of the recent things I finished was a project called Lensm which allows you to browse source code, and to compile the assembly side by side, and see how Go translates into assembly, essentially, and do this interactively. a nice way to visualize it when you’re optimizing things.

Kris Brandow: There’s no general answer to that question, I think. It’s highly context-dependent. I think a great example of this would be Kubernetes in the current day. Kubernetes is amazingly popular, everybody is using it, it’s sort of becoming a standard of its own… And you kind of have a choice when you start using it. You can either keep your applications as they were before, running in whatever they were before, and just make them run on Kubernetes. Or you can start leaning into Kubernetes, build operators, tap into the API, get all this information, but then you’re tying yourself directly to Kubernetes. And it’s kind of hard to figure out which one of those things you should do, because Kubernetes has only been around since 2014; that’s about seven years… So seven years from now is Kubernetes still gonna be a thing? And if you lean into it very heavily, then you’re making the assertion that “Yes, I think that it’s still going to be a thing in seven years. I think we’re gonna benefit from having done this integration.”

If you don’t, maybe you’re saying “Well, maybe there’s a possibility that we could hook into something else.” But I think usually what winds up happening is that people don’t make a decision either way and you wind up with something in the middle, and in this case, being in the middle is bad, because that means you’re still stuck with Kubernetes at the end of the day, but you can’t actually move to something else, because you’re too tied into some of the things that Kubernetes is doing.

This also happens with cloud platforms. People are like “Oh, we wanna be multi-cloud” or “We wanna be able to switch away from AWS, and be cloud-agnostic”, and then you actually sit down and you’re like “Well, here are all the things that you’re doing that make it so that it’s actually gonna be a giant pain to lift and shift to somewhere else”, as they say.

So I think it’s one of those things where you have to gather as much information as you can, and then make a decision in the moment that makes the most amount of sense. And then realize you’ve made that decision, kind of more or less live with it, and then realize that at some point in the future when you have more information, that you should come back and potentially revisit that situation.

So if you’re building something today, I would say if you’re already using Kubernetes, lean all the way in. Use operators, use all these great features, because that will make your life a bit easier, it’ll make your current way of operating a bit easier. But just remember that some day down the road, maybe 4-5 years from now, those decisions you made won’t be making as much sense.

[36:05] You’ve gotta continually evaluate and reevaluate that decision that you made, which I think is kind of the theme that we’ve had this whole time, around just like “You have to bring in the requirements, you have to bring in the information, you have to gather all of that stuff.” And on that, I wanna raise the prospect of build vs. buy outside of just the code realm, and kind of bring it to our process and project management ethos and the things that we do… Because as we have been saying, we have to gather all of this information, we have to evaluate, we have to track time, we have to do all of this.

In my experience, I’ve very rarely seen people making explicit build vs. buy decision around this; it usually starts out as a buy. Someone’s like “Okay, we’re gonna do Agile. Or we’re gonna do Scrum. We’re gonna do X, Y or Z, and we’re gonna use these methodologies”, and then it morphs into some sort of built thing that doesn’t really look as much like the thing you started with… And new people come in and they’re like “Well, this isn’t how I did X, Y, Z, Agile, whatever, at my last place.” So it seems like quite a mess. So I’d like to hear your opinions on “Do you perceive this as a problem as well?” And where do you fall maybe on how we can start resolving this, or at least making it easier for people to make an explicit build vs. buy decision around this stuff?

Joakim Kennedy: I think I started coding in it about 5-6 years ago, something like that. It’s been quite a while. But it’s always been sort of like a side project; I’d maintain a couple of projects around it… And then as that activity started, it started to bleed into sort of like a professional side, where those two things kind of collided. And the interesting thing was when I started to look under the hood, looking actually at the disassembly, or looking at the assembly produced, I felt like my coding got better, because now I understood better what actually was happening. It’s like, you realize the difference between “Well, if I pass this data in as a slice of bytes, or should I pass it in as a string?” and you know the difference in just the size for the function arguments, and things like that.

Christopher Hiller: Yeah, sure. Mocha doesn’t come with an assertion library. Pretty much every other framework does. So every other framework is gonna have some way to make an assertion; Mocha doesn’t. And I’m sure from day one people are like “Well, why don’t you add an assertion library?” I said “Well, that’s kind of a design choice.” You wanna let a user choose, essentially. And it keeps the project smaller.

So there’s stuff like “Mocha doesn’t do X out of the box”, and it doesn’t. Well, it’s because a lot of people don’t necessarily need X out of the box. And if they want to do X, it’s no so hard to get it working. You can get Mocha working with pretty much anything that I’ve seen. Somebody’s got a way to do it, and it’s more of a design philosophy. It’s not a batteries-included project, it doesn’t really need to be; there are plenty of people who are happy with that. I’m happy with that. If you wanna do snapshot testing, you can use Mocha to do snapshot testing; it just doesn’t do it out-of-the-box.

Nick Janetakis: When it comes to hosting videos, you don’t necessarily need to develop your own course hosting platform… There are these other platforms out there, like Thinkific and Teachable; I’m not sure if you’ve heard of them before… But they’re basically like a course-hosting platform as a service; they’re a SaaS app that you sign up for, you pay whatever ($50-$100/month) and they give you the whole enchilada. So people can sign up, they can create an account, they can put in billing information, credit cards or PayPal, and then they can go consume the courses that you’ve created, because those Teachable and Thinkific sites, if you’re the course instructor, you get an admin back-end where you can assemble the course however you see fit. You create your table of contents, upload the videos, etc. So they’re just like an end-to-end SaaS for building courses, and distributing them as well.

But you know, when you’re a developer, you just think in your head that you wanna develop things, and I feel like… [laughter] I think developing my own course platform has been something I’ve wanted to do for a long time, but I still do freelance work now, so it’s very hard to balance the time of creating the actual course I wanna create, and then develop the course platform, and then do freelance work, and then do what I can to contribute to open source, based on projects I’m doing, writing blog posts, the whole social media thing, personal life - there’s a lot to take on.

So I actually did start building my own course-hosting platform about nine months ago, and this is kind of how I got introduced to your codebase… So I determined I wanted to use Phoenix with Elixir. Right now it’s quite popular as a technology, but it’s not on the same level as Rails when it comes to tutorials, and things like that, so it was a little bit tricky to get started, especially if you’ve never used a functional programming language before, which I haven’t done… And then I just ran across the Changelog codebase, and it’s kind of funny, because one of the courses that I wrote - and I’m not trying to pitch my courses here, but “Build a SaaS app with Flask.” So it’s a Flask course where we actually build a real-world SaaS app; accepting monthly payments, and custom admin dashboards, and Stripe integration, and all this good stuff… And it’s like a real application, with like 4,000 lines of Flask etc. and I was very, very fortunate to find your project on GitHub, because it reminded me of that type of project. It’s not just a simple to-do app, or “Here’s how to make a blog with comments.” You know it’s a real app, running in production, really good test coverage, really good everything; best practices… And for me that was heaven.

I basically used your platform as almost my sole learning experience for getting used to Phoenix and Elixir. Sure, I read the documentation, but it was more like I read the docs to get the super-duper basics, and then it was like I’m just looking at your code, and then occasionally harassing you on Slack if I get really stuck…

Bryan Hughes: Yeah, and this was like the early days of the project… I think the project was less than a year old at this point, but I was like “This is awesome!” I did stuff in like a four-hour workshop that would have taken me two weeks in Assembly, which is what I did in college. That kind of drew me back in, so I got involved as a collaborator on the Johnny-Five project; I maintained Raspberry Pi support for it. The Raspberry Pi was the first platform supported that was not part of the initial set that Rick Waldron created, too. So I guess that’s kind of my biggest claim to – well, I don’t know if it was my biggest claim to fame, but a claim to fame, in any case…

Erik St. Martin: Yeah, and I don’t think we consider that cheating. I think it’s the one thing that makes all of our lives easier, all the time.

To everybody’s point, we use only a couple of tools every day, so I think we’ve gotta start stepping outside the box. For me - and I know I’m gonna pronounce this wrong, because I don’t think I’ve ever heard anybody pronounce it, but it’s actually a project called Radare, or Radare2, rather. It’s not something I use every day, but it’s something I play with. They call it a reverse engineering framework, but really it’s kind of a disassembler and debugger. It kind of does some of the same stuff GDB does and all that jazz; it serves kind of the same role that IDA Pro does, but it’s completely open source, and there’s Go bindings for it, which is awesome. So you can actually script the disassembly and searching and patching of code using Go. I’ve only tinkered with it a little bit, but it’s something that I’m looking forward to playing with more. It has ASCII control flow graphs of the assembly, and all kinds of fun stuff. I’d like to see what I can do about pairing it with Delve, one day, when I have infinite amounts of time.

Alex Russell: I mean, I think that’s true when you are operating your system, and it’s not operating you. A lot of teams that I’ve worked with are in this place where they have bought, even though they didn’t realize it was a transaction going on – they did the starter project thing, they took the next thing, they took the Angular starter kit, they took the whatever, and now they’ve inherited a great deal of complexity… Complexity that they now own, and are the one and only stewards of. They’re plugged into the community for dear life, because that is complexity that they probably don’t fully comprehend, certainly not in every area, and each additional place that they have been sort of granted that complexity in a stack without an affirmative choice, and a measurement, is a place where – they’re taking it on someone else’s word that this is the right choice, which is often a thing that has to be re-excavated later, and investigated.

So the thing that I think I’m most frustrated about with the product side of the world is that we haven’t done a good job of creating a class of product managers that are willing to do bake-offs, that are willing to actually try to ask the question, “Okay, what is the evidence to say that this is actually improving my developer productivity?”

In aggregate, if I build a product, and teams get dead in the water for a month before or after launch, the wind in the hair feeling at the first month - you know, when you had two engineers on it, not 20… Is that faster? Is that better? Are we actually going faster?

So I’m extremely disappointed right now, with the web development community’s sort of poor characterization of its very bold assertions. Maybe they hold water, but the fact that no one’s looking and no one’s asking, it suggests to me that we’ve entered a kind of evidence-free zone, and you just – you don’t get out of that except by accident. So we have to be a little bit more intentional, I submit. And this is where, you know, when we’re working with teams to try to get them out of the ditch - because there’s so many products in the ditch - the most important thing that we wind up doing in this sort of technical coaching kind of thing is to help the teams learn what choices they have embedded inside their stacks, and then learn to operate those choices. Because in many cases, those choices are just latent. They’re not things that they are thinking about in relationship to the other parts of the system, so that they can make trade-offs about them. Because when the system is just kind of there, and everything you do to it does something incremental to it, it’s kind of spooky action at a distance. And that’s a really bad feeling for the team. And so to get a team into a place where it can approach mastery of its own system, you kind of have to walk through a very long period of acknowledging what you don’t know about the system that you’ve inherited.

[50:17] And these are all complex systems, right? A web browser is an incredibly complex system. And so part of this process is very often also teaching teams about how browsers think about their content, like how browsers actually work… Because once you can understand the choices that you’re making in context of like how browsers think about the content you send on the wire, you can start to, with a little bit more confidence, evaluate and then maybe reevaluate some of your choices incrementally. Because the thing that I’m never going to do is say “Blow up your thing and rewrite it.” Maybe if it’s actually so bad… But almost always - and in one of these consulting kind of situations, it’s going to be a “Here’s how we’re going to make the next increment of steady progress towards getting this thing out of the ditch. We’re gonna put some planks under it; we’re gonna put it in low gear, we’re gonna get a tow truck, we’re gonna pull it out gently, we’re gonna inspect it for damage, and then we’re gonna see if we can actually steer it without the axle being bent”, right?

There’s just a series of concrete steps you go through, and they’re all very painful when you’re in that position, because nothing that you’re doing at that moment is going down the road. So when people tell me that they’re having a great developer experience, I kind of want to see it in data.

Ivan Kwiatkowski: Well, they don’t really represent themselves, because threads as a concept, they are a fundamentally runtime object, right? A thread is going to be a unit of execution that is going to run some code. And when you have a single thread, which is the case for a lot of programs, then you can just follow what’s going on in the code linearly, and then you figure out what is going on. When you have several threads, then this is really an order of magnitude more of complexity that you have to wade through, in the sense that as a reverse engineer, you not only have to think about what is going on in the program or in the function that you are reading, but at all times, you have to think about the fact that there might be another thread running somewhere that might be doing things that are affecting what you’re doing right now, or what the current function that you’re reading is doing. And so you do not have the luxury of having all the information that you need in a single place. The functionality, the intelligence of the program ends up being spread over different units of execution, and you have to keep everything in your head to have any hope of understanding what is going on. So this is really a very heavy mental tax that is imposed on the reverse engineer. And of course, the more threads there are, then the more effort you have to go through to try to keep track of everything that is going on.

One good example of this is a Go program that I mentioned in the previous podcast, it’s called Stowaway. This is an open source project that is used to do various proxying operations as a pen testing tool. You can create tunnels, SOCKs proxies etc. and probably pipe them with each other. I’m not exactly sure… But what I’m sure of is that when I was reading the program’s assembly, it felt very, very miserable, because it was obvious that many things were happening at the same time, which of course is going to be the case, because when you have some network program, then packets can arrive from any end of the various terminals. And you can also have many tunnels running at the same time.

[32:05] So you have all these things taking place at the same time, and trying to figure out exactly what does what is extremely difficult. And if I hadn’t been able to figure out that this was actually an open source project for which I was able to go find the code, then probably I might not have been able to figure out everything that the program was doing at all, because there was just too much to work through, and too much to remember, because my memory is actually quite limited, as is the memory of any human being compared to a computer, really.

Brian Kernighan: Right. Well, I guess the proper way to describe it is “Present at the creation, but not responsible for it.” I was a grad student at Princeton in the mid-to-late ’60s, and I got a summer internship one year at MIT in Project MAC, which was basically building Multics, a very large information utility, so called… And then the following year, 1967, I got an internship at Bell Labs in Murray Hill. People there were still working on Multics; I had nothing to do with that. But I spent two summers of internship at Bell Labs, I had a great time, and so I went there permanently in 1969, very early 1969, and found myself in the same group as Ken Thompson and Dennis Ritchie.

[04:22] Ken at that point – I guess the right way to say it is they were suffering withdrawal symptoms, because they had been working on Multics, which was a very big, very interesting computing system, and they had really, really good computing facilities, but then very early in 1969 Bell Labs withdrew from the Multics project because it was clear it wasn’t gonna produce what it was supposed to produce, at least on a timescale that the Labs cared about. And so this left Ken, Dennis and a variety of other people fundamentally with a taste for a nice computing environment, but no way to satisfy it, really… So they spent a fair amount of time basically doing what you would call paper designs, sketching things on blackboards, and so on. But as part of that, very early in that year, I think Ken Thompson found what is traditionally described as a little-used PDP-7. So the people who’d been working on the Multics project at Bell Labs withdrew, as I said, in 1969, and the people who’d been working on it, Ken Thompson, Dennis Ritchie and some others were left without a very nice computing environment; you know, very large, powerful computers that they sort of had all to themselves… So they spent a lot of time basically doing what you might call paper designs, or blackboard designs of what an operating system might be like, what they could do… And they also – Ken Thompson found what is traditionally called a little-used PDP-7, a machine that was pretty obsolete already in 1969. Nobody was really using it… And he used that as a vehicle for experimenting with file systems. At some point, this famous comment that he made, that he had three weeks when his wife took their son off to California to visit the relatives, and in three weeks he put together what was basically the first Unix system. A proto-version of Unix, because you already had the file system you’d need an exact call, a shell, an assembler, things like that.

So three weeks and we have the first version of Unix. And that was in 1969, so you could argue then that 2019 would be the 50th anniversary of Unix. The Unix history book that I wrote, that you mentioned - basically, I put that together as an attempt to capture some of what I remember and what lots of other people remembered of what was really kind of a golden era in computing, and certainly the early days of Unix, very interesting; the evolution of it over the next 20-30 years is very interesting, at least to me…

George Neville-Neil: I would like to see Rust proven to be better than way. I would be very happy to trade in C for a safer C. The reason C never gets supplanted isn’t the safety issue… Because it turns out that most people who are producing products for you don’t care about your safety… But don’t tell them that. What matters is performance. So no one has come up with something other than hand-coded assembly, which is portable and as performant as C. Now, Rust is getting very close.

So if Rust can keep the safety qualities and be as fast as C, then yes, we should chuck as much C out as possible… And all the people who work with me on the FreeBSD Project right now are sharpening knives, and once the pandemic is over, they’re gonna come to my apartment and cut me into little ribbons. [laughter]

But yeah, I definitely think that C is… There are projects in the world right now… There’s the cherry work that’s been done at the University of Cambridge - Robert Watson, who I’ve worked with on various things - trying to change hardware, so that hardware is safer, and we can actually use pointers without foot shooting. And a change in hardware is gonna take a while, so we’ll see how that works. I think it’s very promising.