Adam Stacoviak: Yeah. Here on the air I wanna mention – I think it’s a redirect, rockylinux.org/contributing. So it’s wiki.rockylinux.org/contributing. It actually has a lot of details there… And even to be able to reach out to get on a particular team - community team, brain team, development team, documentation… So if you have a desire to play a role on any level, this page will probably be an ever-changing page, but for now it’s gonna direct you to the right kind of places… So that’s a good place, I would think, to start. Worst-case, rockylinux.org is a good spot to land.

Greg, thank you so much for sharing the story with us. I think that’s what’s the most fascinating to me, and I think Jerod will concur, about doing this show - we get to hear these stories and really see the future, where things are going. You’re very much helping to lead that in many ways, and we appreciate you sharing that story with us. Thank you.

Yaron Singer: Yes, that’s a good question. So what we do is basically we test the models. We have a process of stress-testing the models, and we do that either implicitly – if you just install the AI firewall, if you just put the line of code, we basically do that in the background, we do that implicitly. Or sometimes, you know, there are companies where we start from stress-testing, and then we kind of graduate to an AI firewall.

Stress testing - what that basically means for us is we run a series of tests on the model. You know, some of the tests can be “How does the model respond to distributional drift? How does the model respond to unseen categoricals?” Just all these sort of different scenarios and different inputs, and then we’re measuring the response of the models to all these bad things that could happen. And as we’re measuring them, we’re getting a sense of – we’re basically training our own AI firewall.

So by understanding how different input can affect the model in different ways, when new input comes in, we know whether that input is going to lead to some sort of prediction error, some sort of prediction change.

Let me give you a silly example. Suppose that your model – you have some sort of AI model, and maybe that AI model is trying to predict whether somebody’s gonna earn above $100,000 next year. So whenever we take that input, we check all the different features and when we sort of see how the changes in different features is gonna affect the prediction of the model. So maybe for example we look at how does age affect the prediction of the model. And now suppose that data comes in and somebody has accidentally replaced age with the year of birth. So age is maybe from like 36, age was changed into 1985, or 1986, or whatever.

[32:22] That’s basically some sort of human error, an error in feeding in the data, and by that point, the AI firewall is trained and it knows that “A change in this feature can really affect the model prediction”, and certainly understands what the right distribution of age is, and understands that age needs to be something between probably – it’s seeing data from 10 to 90, or something like that… So it understands whenever it sees a big number, like 1985, there’s something wrong here. And then what it can do is basically it cal alert and it can prevent that mistake from happening by even replacing 1985 by the mode of the distribution, or something like that, to kind of output a better prediction.

Gerhard Lazu: Yeah, that was appreciated in so many ways. I think it’s going to lead to so many other things, this idea. You take – I wouldn’t say a negative, but something that could be a negative, turn it into a positive, a playful positive, and then it leads to many other positive things. So that was really nice.

Jon Thornton: Yeah. So, I'm going to be speaking at The Lead Developer Conference in New York City. The conference is April 7th and 8th, and I think I'm going to be speaking on the 7th.

It's a great conference. I've been as an attendee before; I've never ever spoken there, but… Anybody in a technical leadership role or people who aspire to be a technical leader, it's a great conference.

Dan Kohn: Yes, that was probably my biggest success being involved in CII, was to find David and recruit him in. He and I were really the code developers on that project. He did really the lead on all of what the 66 criteria should be, and everything… But at the time, he actually hadn’t done any Rails development before, and I was working for a Rails development company. So I put together the original scaffolding for it, and the CI system, and then helped along the way on the software development. And again, it’s an extremely satisfying process to have started that from scratch and then have well over 1,000 badged projects.

In fact, not so coincidentally, one of the criteria in order for a project to graduate within CNCF is they do need to get the Best Practices badge.

Kendall Miller: So pseudo sales is always going to be a part of my job, right? I’m always going to be talking to people. It’s always going to lead to sales conversations. There’s a lot of times I end up at a new company and somebody in my network pings me and says “Hey, that’s kind of interesting. Can I talk to you about it?” and it turns into a sales conversation. I don’t like being the sales guy, because then I have to hassle my network, check in, follow up… Usually, I want to hire somebody else that’s going to do those kinds of things, so it doesn’t have to be Kendall hassling somebody. I don’t want to hassle. I don’t mind if a salesperson who works with me or for me is hassling the hell out of one of my friends; that’s their job. But it doesn’t, if it’s affecting my relationship with that person, that’s a problem, if that makes sense.

So what do I want to do? I mean, that’s the messiest part. That’s the hardest part for me every time, Jerod, is because what excites me is just solving business problems. I like getting in, finding out that there’s things that are a mess, and just going and helping solve it. To give you an example, in the last two days I’ve written job descriptions, I’ve come up with prospecting lists, I’ve interviewed people for marketing roles, for operating roles, for sales roles… I’m interviewing for a CEO right now…

[00:28:10.17] I touch every single part of the business, because I’m comfortable in almost all of it. I’m not going to go review a PR. That’s the one spot that I’m probably not going to be useful… Although at a previous company that I was running I would about every year go issue a PR, just to scare all the engineers… You know, there’s always something small, like a typo or something in documentation…

Adam Stacoviak: [01:05:54.12] And I installed Windows, and I’m just so sad for Microsoft, that they can’t get that right. They have the largest installable base of a computer user on the planet. And that’s their best effort. It’s just – I’m just sad for them. That’s just – it’s a mess. They installed so many softwares that are just not necessary, and it’s just disgusting. Maybe it’s their fault, maybe it’s not their fault. I feel like they can solve the problem; they’re not solving the problem. But they’re not.

I don’t believe there’s a default, free password manager in Windows. I googled it… PCMag disagrees. They say that there’s other ones. So they haven’t selected like this default installed for Windows, but… You know, somebody’s got to do this, and who’s going to lead that effort? It can’t be 1Password, because they’re of service. They’re a software company trying to make money. I mean, I think giving away 1Password for free is not very smart… Although it could be the Xerox of 1Password, or I guess the Xerox of password managers, is that they could give it away for free to everybody to a certain limit, and attract a lot of people. And they’re already on all the platforms, so maybe that’s a good way for them… But there is no LetsEncrypt for password managers out there where it’s just free to everyone and accessible.

Adam Jacob: [01:22:14.17] Yeah, yeah. And thinking about it now, and talking to you about it - like, I’m welling up now just thinking about it. I don’t think I’m going to cry, but I might. If I do, great podcast material. But yeah, of course I was giving it to myself.

Of course I was. And look, in the years since, I’ve been in therapy, and learned a lot, and I’m improving… But it’s not easy for me to talk about, like most people… There’s some people for whom they can talk about how they’re feeling, and it’s like very difficult for me to talk about how I feel. If you ask me what’s happening, I’m gregarious enough, I’ll tell you how my day was, I’ll tell you like a list of facts about my day… But I’m not going to lead with how I feel at the end of the day. I’m not going to be like “Oh, I feel anxious. I feel stressed. I feel sad. I feel whatever.” And so it’s not like I was expressing to other people, anyone else, even including my wife, what the burden I was holding was. That was all just inside, and I was just white-knuckling it sort of through all of those moments. And yeah, so there was a lot of internal pressure that had sort of built up there that needed release. I think part of why the speech was good was because that same energy came out in the speech. I used that as fuel. And because I had allowed that to happen when I gave the speech, suddenly all these emotions I had not been processing and hadn’t been holding onto I couldn’t keep in anymore. And even though I couldn’t verbalize them, I could certainly experience them physically by just weeping, sobbing, you know?

Nestor Maslej: Yeah, I think there’s two things that you’re speaking to here. I think first is the fact that if you look at public opinion data, at least in a lot of countries like the United States, Canada, France, people are very bearish about AI. When kind of asked “Do you feel that products and services using AI have more benefits than drawbacks?”, in those three countries and also in a lot of other Western countries the overwhelming majority of respondents seem to disagree. They don’t think that AI is more beneficial than disadvantageous.

[00:39:57.29] Yet in a lot of other let’s say developing countries like Indonesia, Thailand and Mexico there is much more bullishness. People are very excited and very hopeful. And we don’t necessarily know why that’s the case. That’s obviously something that’s very important to kind of unpack and continue thinking about as this technology develops, and as it rolls out even further.

Now, you mentioned this kind of point about economic advantage. I think what we’re referencing here is that there is now a lot of new studies coming out in really different industrial sectors, whether it’s kind of looking at call centers, whether it’s looking at legal work, whether it is looking at computer science work that shows that workers that use AI tend to be more productive than those that don’t. At the minimum they’re able to do tasks faster, and at the maximum they’re not only able to do tasks faster, but they’re able to submit work of higher quality.

So what kind of explains the disconnect? Well, I mean, I still think we’re kind of in the early days of AI integration; these kinds of studies that have looked at AI’s positive impact were quite microscale. I don’t necessarily think businesses are kind of using this technology en mass yet. And I think second is the fact that it’s hard to necessarily know where we might go, even if AI is productive… Because if you’re working 40 hours a week, and all of a sudden you need to be working 20 hours a week because of AI, you could either use that 20 hours to do perhaps different projects that will lead to more money and more value for your employer, or maybe your employer decides “Hey, we just don’t need you for those 20 hours and kind of scales back what they asks of you.”
So I think the kind of jury is still out on whether the integration of AI technologies is going to lead to widespread automation or augmentation, and I think when you kind of look at narratives of fear, or if you at least try to understand why different people in different countries are as frightened by this technology as they are, I think a lot of it comes down to this element of just kind of uncertainty about what it’s going to do to their jobs and their livelihoods.

Quinn Slack: And I’ve been reading – I love reading history of economics, and inventions, and so on… And I’ve been reading about the oil industry. The oil industry got started when someone realized “Oh, there’s oil in the ground, and this kerosene can actually light our homes much better and much more cheaply than other kinds of oil, from whales, for example.” And initially, oil was all about illumination. Make it so that humans can stay up after 6pm when the sun goes down. And that was amazing. But that’s not how we use oil today. Oil is just this energy that powers everything; that powers transportation, that powers manufacturing, that powers heating, and so on. And there were people that made fortunes on illumination oil, but that pales in comparison to the much better use of oil for our everyday lives. And now, of course, you have renewables, and you have non-oil energy sources… But for a long time, we saw that that initial way of using oil was actually not the most valuable way.

[00:46:14.09] So seeing that this just happens over and over, that a new technology is introduced and you’re not quite sure how to use it, but you know that it’s probably going to lead to something… And that’s how we always felt with code intelligence, and that’s – us getting new Intelligent Automation is so exciting for us now.

One of the really exciting things we’re seeing is because – so many people are shocked that these LLMs, you speak to them humans. They seem to feel much more human-like than what we perhaps anticipated AI would be like. We think of AI from movies as being very robotic, of lacking the ability to display empathy, and emotion, and thought processes. But actually, that is exactly how we see LLMs. I’ve seen some studies even the show that LLMs can be better at empathy than a doctor with a poor bedside manner, for example. And for us, this is absolutely critical, because all this work we put into bringing information about code to the human brain - it turns out that AI needs that same information. That AI - well, the human, if you started a new software engineering job, you get your employee badge, you go read through the code, read through the docs, if there’s an error message you’ll look at the logs, you’ll go in team chat, you’ll join meetings… That’s how humans get that information. And AI needs all that same information. But the problem is, you cannot give AI an employee badge and have them roam around the halls and stand at the watercooler. That’s just not how AI works.

José Valim: So I think that’s a very good question. I gave a whole talk at Elixir Conference Europe, in 2022 perhaps, on this topic. And what I said - and usually, when I go give a conference talk, I usually give Elixir updates when I’m at Elixir events. “Hey, here are the things that we have been working on.” Or I usually talk about a project that I’m involved, Livebook, or stuff with machine learning… And then I started the talk saying “Oh, these are the things that I’m excited about in the future of Elixir machine learning, and stuff. But before I talk about those things, I need to address what I call the elephant in the room.” Because if you go to an Elixir event, or I think if you do a poll in a forum or something that, if you ask “What is missing in Elixir?”, most people are going to say it’s types. A type system is the thing that is missing in Elixir.

So if a lot of people feel that’s the thing that is missing, I have to understand that. I have to understand why they feel it’s missing, what do they think they’re going to gain… And one of the things that I say in this talk - I don’t remember precisely the arguments precisely I used at the time, so I may butcher my own presentation… But one of the things that I mentioned in the talk is that usually discussions about types, they tend to get so polarized very quickly. And it’s very common for you to hear things that I don’t believe to be true, if you try to look at it.

I remember seeing people saying “Well, if I was using a type system in this particular language, it would have caught 80% of my bugs.” And then I’m “I don’t think that’s –” I feel what the type system can do for you in that particular scenario was being overvalued. I don’t think it’s 80% of the bugs. And then when you go look at it, when there are some studies trying to find a correlation with that, they can’t find something that precise or that clear, that says “Yes, a type system is going to lead to a program with fewer bugs.” I don’t think we have found the universal proof of that. And I even think this discussion as a whole can be harmful… Because you’ve probably heard people saying “Hey, well, if you have types, you don’t have to write documentation”, and I don’t agree with that. [unintelligible 00:19:55.05] That doesn’t tell me how that things behaves.

So then people say “Oh, if you have types, you can write fewer tests.” And I also think that’s very harmful. I like to say “Well, if you have types, you can write fewer bad tests”, because I don’t think types would replace 99% of the tests that I actually write and I find important. For example, going back to the same logical or. The types are not going to find bugs in the implementation of logical or if you make it always return false.

So I gave this whole presentation which was kind of preparing people why we may need a type system in Elixir, but the presentation was much “Well, types are not all these things that you think they are.” So very good salesman here, right?

Bryan Cantrill: And I think it’s so interesting, because – like, with so many things about the series. It’s like, it’s funny, and it’s light, but it’s hitting on something really, really deep, where you have… And this is an absolute problem in Silicon Valley, where management is like “Yeah, I just wanna make you happy.” And it’s like “Well, that’s not actually not the way you lead.” As Steve and I will know, leadership is not always going to lead everybody happy… And if you try to lead everyone happy all the time, what you end up doing is actually just creating a mess.

Anna Pobletts: Yeah, so from a business or developer perspective, I can kind of see both sides. I personally think it’s kind of worth going fully passwordlesss, if you can, and still having those fallbacks of magic links or OTP codes, which are still a little bit better than passwords, even if they’re not great. They’re familiar to users enough, I think. But I also think it’s important to give people the option to just add on to whatever they have right now. I think that’s faster, it’s easier, and if we want to incentivize developers and businesses to implement passkeys, we have to make it really easy. We can’t make them lift and shift their entire authentication infrastructure. I just don’t think that’s reasonable. And so I think it’ll be a lot of sort of like Home Depot has, where you have a password option. And maybe you even always register with a password, and then you can just add a passkey on top of that as a faster sign in. I think still getting people used to that user experience is what’s going to lead the charge.

On the Passage front we kind of support both of those options. We have a product called Flex, that is basically that; it’s like, we’ll do the passkey management for you, but you still handle all your user management, your password management. Whatever other authentication you want to do, we’ll just add it on top. And so I think it’s a lot easier for websites Home Depot to go do than it is to maybe make that full switch if they’re just not totally convinced of the benefits yet; they get to see it in action first.

Feross Aboukhadijeh: And there’s a lot of problems with the CVE system, but the fundamental problem is they’re all theoretical problems in your app. And not to mention the severities are all really inflated on the reports. So everything is basically critical or high, because if you use it in the exact, correct way, it could be really, really bad. But it’s probably not used that way; the vast majority of these are just not going to affect you.

So I don’t want to downplay – I mean, obviously, there are very significant CVEs, that can be a big deal for you, but just if you just look at the kind of hundreds of warnings that you get on an npm audit, how many of those are actually affecting you, or going to lead to your application getting compromised - it’s a very small percentage of those.

[13:47] So that’s the key Original Sin, or whatever, of a lot of the security tooling, which is why we’ve focused almost entirely on supply chain attacks, and malware, and stuff that basically if you have one of those in one of your dependencies, you will not be upset that we told you about it. You will not see it as like an interruption, or like “Why is this tool annoying me?” Like, that’s the stuff we’re looking for. So it’s pretty different.

Sean McClain: [20:19] Yeah. 100%, this is going to lead to personalized medicine. So let’s just fast-forward 10 years from now, we see a success rate go from 4% to 50%, let’s say. And at this point in time, we can go to the FDA and start changing how clinical trials are done, and actually have it be personalized. And it’s able to actually occur as well from a cost perspective, because you’re fully in silico, and 50% of the drugs you’re designing actually work, which means that it’s going to get new drugs to patients a lot faster. You’re going to be paying for less of the drugs that have failed. So the cost per drug is going to dramatically decrease, and that all together is going to enable personalized medicine.

And what’s going to be really interesting is I believe that health insurance is going to completely get disrupted as well, because the cost has dramatically decreased, as well as the speed it takes to get drugs to patients, and it’s fully personalized. Think of it almost like a SaaS model for healthcare that you have for a lifetime, and you’re able to get the drugs that are specifically tailored to you. That’s the future that’s going to occur within the next 10 to 15 years. So maybe when you’re 60, you’ll have personalized medicine delivered to your door.

Suz Hinton: I know that’s built into Facebook and built into Twitter, because it’s built by nerds… You can actually use J and K to flip through tweets and posts… That’s cool. I already know how to use voiceover on Mac, the screenreader, because I am a big accessibility advocate, and that to me is just how I write non-broken software - you make it accessible and you test it. So to me, I think I’m much more productive in that, because that’s what it was sort of designed for… Whereas I think that a lot of these things are kind of this weird kind of shim, and it doesn’t make a lot of sense.

VS Code feels really bad in Vim mode for me, I don’t know why. I think it’s just because it’s just nowhere near as responsive. So you’re hitting J/K and you’re just like “It doesn’t feel the same.” It feels like someone’s kind of pretending and pulling the ropes behind the scenes. So I kind of just abandoned that, and I was like “If I’m gonna use VS Code, I’m going to lead into the VS Code paradigm.”

Justin Searls: [48:18] And another thing that we can think about, too - this kind of gets back to minimalism… I had a really fun experience at a client a few years ago, and they were just doing tests for the first time. It was a group of 40 people, and they were like “Well, we’ve never written a test before. That’s why you’re here. Teach us how to be a good organization”, I guess. And one of my projects there was helping a small group of three QA engineers. They had a lot of technical knowledge, a couple of them knew Ruby, a couple had used Selenium before, and they were asking me, “Okay, so what kind of integration tests should we write?” And I could have put them down the path of just like “Oh, well, see what the app does, look at the criteria, through the specifications, and for each story make sure that you’ve got a thing that clicks through the browser and does it.” And you know what - that’s going to lead to the same outcome, of like two years from now you’re just gonna have hours and hours and hours of redundant-looking things, where like 90% of the actual activity is signing up new users for the 80th time in a test suite…

So I said instead – I went up to a whiteboard and I drew a 5x5 grid, and I said “Okay, there’s 25 boxes in this grid, and each box is a minute. So you have a budget of 25 minutes, and your build is never, ever going to be slower than that, or you’re gonna start throwing away tests. And I want you as a QA department to work with the business to figure out what’s the most important stuff that you need to make sure works before a production deploy”, and that way – they were also adopting something akin to continuous deployment… That way what they could do is guarantee that the business, that testing isn’t going to hold up a deployment for more than 25 minutes. And it also, again, forced a value-based decision criteria for like “Could we make this test faster? Could we make this part of the system faster? Is it really important for us to test this kind of musty admin page, when we could just do that manually, and get stuff out to production faster?” Making those kinds of ROI-based decisions on “Is this worth automating?” was really productive for them.

Brett Cannon: Oh, very much so. The key catalyst to this was that Python introduced what we call “Assignment expressions are the walrus operator”, and it was a very controversial decision. It introduced new syntax, where there is definite benefits, but it was one of those things where it’s a marginal benefit in several cases that you might scatter about in some places, but it didn’t introduce a new capability to the language that was not somehow doable without it being there… And at this point, people very much like Python’s size, because we have a phrase that “Python fits your brain”, and whenever there’s new syntax that doesn’t open up a new avenue of abilities that weren’t possible before, people very much worry that that feature is going to lead to a potential cascade, or that will be the feature that finally pushes some other aspect of the language out of their head.

[56:25] And because of the amount of negativity he got over that PEP, he just finally said “Forget it. I don’t want to have to fight that hard for one of my decisions ever again.” He just went “Enough is enough. I just don’t want to be that kind of focal point.” And I think the best way to point that out is if you look at what the hierarchy used to be, it all funneled to him; all the pressure of all of the development more or less funneled to this one individual very decisively. And now that he’s no longer doing that, the pressure is alleviated not only among all five of us – it does not mean there’s five more BDFLs, so we have five times more capacity, as some people have suggested… But also because the PEP specifically we have to get consensus, it also falls a bit more on the overall development team to have to come to an agreement and participate and make decisions as well.

That also helps alleviate some of the pressure; it’s like “Well, we tried to get consensus, but no one could come to a consensus, so we’re just gonna let it go”, unless we feel as a council something is very critical and we need to make a decision, and we really think this is for the benefit of the team, and thus we have to potentially make that unpopular decision that not everyone’s on board for, because we honestly think that it’s important enough to make that kind of call.

Austin Clements: Well, I’ll give sort of two answers to this. One is as I’ve been sort of closing out my previous role as Go core tech lead, one of the problems that I have been trying to solve ever since I started on the Go team was garbage collection. It’s just – it takes a lot of resources to do garbage collection. And a long time ago, I did this micro architectural analysis that basically showed - you know, the performance of garbage collection is like down here, and there’s some sort of like theoretical sort of speed of light limit on garbage collection, that’s way the heck up here. And there’s no way that we could actually get garbage collection to be that fast, but surely, surely we can close this gap more than we do right now. And like I said, I did this analysis years and years and years ago, and every once in a while I still bring up the slides that I made from that analysis, and I’m like “We still have this giant gap.”

So I’ve been experimenting with this new garbage collection algorithm that – you know, as tech lead, you draw on all these ideas. And so this draws on ideas from lots of people on the team, and I’ve sort of found a way to glue them together in a way that we haven’t thought about before… And I’m calling this new garbage collection algorithm Green Tea, because I happened to make a lot of progress on it while I was in Japan, going to cafes and drinking matcha every day. See, it all connects.

So I’m still doing lots of experiments, I’m still not completely sure that this new algorithm is a win, but it sure has been a lot of fun to work on… It is very unlike our current garbage collection algorithm. I mean, from a user’s perspective, it still does the same thing, but it’s been a lot of fun.

And if nothing else, one of the technical things that we’ve been thinking about recently has been how to get SIMD - that’s single instruction/multiple data - into Go. And we’ve been thinking about SIMD for years now, and we actually have various internal sketches of proposals and so forth, but we’ve never really made that much progress on it. And I believe that one of the reasons for that is that nobody on the Go core team has really done that much SIMD programming. We just don’t have that much in-house expertise with actually doing SIMD programming… And one of the things about this new GC algorithm is that it’s built around a whole bunch of like really tight computational kernels, unlike the current garbage collection algorithm… And that means that for at least some of those kernels you can really dive into micro optimizing them, and in particular using SIMD. And so this has been a great excuse for me to actually learn and get experience with actually writing SIMD code. And a lot of the ideas I had about like “Oh, we can expose these beautiful APIs that will make SIMD all nice and stuff” have gone completely out the window, because it’s a complete mess once the rubber hits the road.

[41:55] So that’s sort of the thing that I’ve been itching to solve for almost 10 years now, that I’m kind of trying to go out on from being the tech lead of the Go core team. And going into being tech lead of the overall Go team, one of the things that I’m sort of itching to solve is along the lines of what I was saying earlier, about how programming should be fun. I believe that – so you know, several years ago Russ started up what we called at the time the Go 2 process, although as we know it’s still Go 1… But the key part of the Go 2 process that Russ put out there was asking people for experience reports on the big things that really caused friction in real production systems. And I think that process worked extremely well.

So we got these top big issues with the production readiness of Go systems. And we’ve been tackling those over the past several years, and we’re still not quite done with that, but we’ve made really good progress on that. And I think one of the things, however, that that process has missed is the small stuff. Obviously, these big sources of friction and things that cause production problems are very important, but also, engineers are writing lines of code. It’s all made out of lines of code. And there are all these small things on an individual line of code where sometimes you’re like “Why is Go making me do this? This is silly.” And like I said at the very top, I do not want to come into Go and start tearing down the house.

I believe strongly in Go’s stability. But I also believe that there are sort of small-scale things we can do that will make Go more fun, and ultimately more productive, by focusing on sort of the micro as much as we have focused over the past several years on the macro.

Angelica Hill: I mean, as you were talking, I think what came to mind for me is do you think that those growing pains are related to the growth of the Go community? I mean, what I’m thinking of is just logistically, when the language was small, there were less people in the community, so it was easier to hear those voices. Whereas now, we’re a spooling, a huge global community… I would assume it’s – well, in fact, I would know it must be harder to keep your pulse on that community, especially when, as you noted, Kris, there are people who just, they sit, they use Go, they’re not going to engage in Gopher Slack, the Reddit… There are a number of people who want to be active voices in the community, but there are so many others that are not engaging in those conversations, but are really active Go users.

So I wonder if part of the issue, or part of the disconnect is somewhat inevitable, given one team, one leader, say - I know we’re talking about kind of being [unintelligible 00:24:09.27] on the Go team, it must be really hard to keep your pulse on that full community, and making sure everyone’s happy. And I would venture that it’s impossible to make everyone happy. I’m just intrigued.

[00:24:24.19] And then also just going back to the title of “tech lead”, you are the technical lead. That does that not then imply that you are accountable for the technology, its resilience. For me, it feels - and maybe this is just my personal view - more like it’s, I don’t know, a product manager, or someone else’s role; everyone on the team 100% should have that pulse on the broader community, but I just wonder, just thinking on the granular… I have always seen the role of tech lead to be leading the technology. I think they should be, and to be a great tech lead I think you need to have your pulse on the community… But I wonder whether ascribing that to the lead - I think it’s a group effort. I think it’s a community effort. The Go team in general - that should be part of what they do, and I think they do… Looking at the Slack, they engage in conferences, they look at the issue tracker… But also we as a community, as individuals can just encourage people who care deeply, but maybe aren’t looking at decisions, or being part of the conversation. I don’t know, I’m just speaking aloud here.

Raza Habib: Absolutely. So maybe it might be helpful to have a kind of imagined concrete example. So a very common example we see is people building some kind of question answering system. Maybe it’s for their internal customer service stuff, or maybe they want to replace an FAQ… Maybe they’re trying to build some kind of internal question answering system to replace something, or an FAQ, or that kind of thing. So there’s a set of documents, a question is going to come in, there’ll be a retrieval step, and then they want to generate an answer. So typically, the PMs or the domain experts will be figuring out what are the requirements of the system, “What does good look like? What do we want it to build?” And the engineers will be building the retrieval part, orchestrating all the model calls in code, integrating the Humanloop APIs into their system… And also, usually they lead on setting up evaluation. So maybe once it’s set up, the domain experts might continue to do the evaluation themselves, but the engineers tend to set it up the first time.

So if you’re the domain expert, typically you would start off in our playground environment, where you can just try things out. So the engineers might connect a database to Humanloop for you. So maybe they’ll store the data in a vector database, and connect that to Humanloop. And then once you’re in that environment, you could try different prompts to the models; you could try them to GPT-4, to Cohere, to an open source model, see what impact that has, see if you’re getting answers that you like… Oftentimes early on it’s not in the right tone of voice, or the retrieval system is not quite right, and so the model is not giving factually correct answer… So it takes a certain amount of iteration to get to the point where even when you eyeball it, it’s looking appropriate. And usually, at that point people then move to doing a little bit more of a rigorous evaluation.

So they might generate either automatically or internally a set of test cases, and they’ll also come up with a set of evaluation criteria that matter to them in their context. They’ll set up that evaluation, run it, and then usually at that point they might deploy to production.

So that’s the point at which things would end up with real users, they start gathering user feedback… And usually, the situation is not finished at that point, because people then look at the production logs, or they look at the real usage data, and they will filter based on the evaluation criteria. And they might say “Hey, show me the ones that didn’t result in a good outcome”, and then they’ll try and debug them in some way, maybe make a change to a prompt, rerun the evaluation and submit it.

And so the engineers are doing the orchestration of the code. They’re typically making the model calls, they’ll add logging calls to Humanloop… So the way that works - there’s a couple of ways of doing the integration, but you can imagine every time you call the model, you’re effectively also logging back to Humanloop what the inputs and outputs were, as well as any user feedback data. And then the domain experts are typically looking at the data, analyzing it, debugging, making decisions about how to improve things, and they’re able to actually take some of those actions themselves in the UI.

James Snell: And this is critically important… So I’ve had people raise the fact, it’s like “Oh, IBM used to pay you to contribute to Node. Isn’t that like taking over?” It’s like, well, no. When I got involved, when I was at IBM and I got involved with Node, we very consciously made a decision… At that time, I was the technical lead for all things Node within IBM. So I was kind of the gatekeeper. And when it started coming in where IBM wanted to get involved in the project and started contributing, we had folks within IBM saying “Hey, I want to get involved with that” and I was like “No. Wait. We don’t want to overwhelm the project, we don’t want to bring too many people into it.”

[12:22] We want to take a very careful and cautious approach to getting involved with the project, so that it doesn’t just go from one company, one corporate master to another corporate master, and then we end up with the same issues. Very, very intentionally, just like hold off; we’re only going to be doing things here that are for the benefit of the community. We’re not going to come in to try to do a corporate takeover of the project.

And that was critical at that time… I don’t think Node would have survived if it just went from one corporate master to another. The guidelines that we put in place for “No, it’s not going to be governed by one commercial interest”, that was absolutely critical for the project to survive.

Amal Hussein: Like updates along the way… Yeah, I would say – so typically, for teams I’ve worked on, we have kind of like a definition of ready, and a definition of done. And so a definition of ready is like, you know, “When is this bug ticket, and/or feature ticket, when is this ready to be worked on?” For bugs, it’s when we have enough detail to be able to reproduce it etc. For feature tickets, it’s when we have acceptance criteria, when we have maybe designs… Maybe there’s a number of things that go along the way. And typically, if you’re fixing a bug, it’s good etiquette once the bug is fixed to just like tag the person who filed the bug to say “Oh, bug’s fixed now, and it’s shipped. It’s going to be in the next release, or it’s already out in production”, or whatever. Or it’s merged into main… That’s good. You want to have them be able to like retest it and revalidate it, and make sure that it’s like actually fixed. So that’s one.

And in terms of when you’re working on something that’s like a product feature, or just an enhancement, typically you have a product owner, a product manager, or someone on your team that’s really driving that direction and strategy, that is going to kind of give you that kind of product acceptance, so to say. Maybe it’s your designer, maybe it’s your product manager, maybe it’s your tech lead, whoever that is; someone else is validating the thing, to say “Yeah, this works.” And so typically, you’d review that at the end of your sprints, or whatever iteration cycle you use to say like “Ta-da! I did the thing that I said I was gonna do, and it’s done.”

Lucas Fernandes da Costa: Yeah, absolutely. So essentially, I don’t like philosophy because it – basically, Scrum is a way of insulating things, and limiting the kind of like work in progress. That’s what Scrum is kind of about, right? When you have these two experiences, you’re limiting how much you’re going to do in that time, and it kind of makes things easier to manage. But you can achieve the same effects if you’re just doing Kanban. And that’s kind of like the premise of this blog.

[10:01] So what I do in this blog is basically model an engineering team as a queuing system. So if you imagine, the engineering team is in the center, processing tasks; tasks are coming in on the left, and they’re coming out on the right. Essentially, what happens is that if tasks come in more quickly than they come out, they start to queue. And when things start to queue, you have more and more lead time, and things take increasingly longer to be done. So one of the common mistakes managers do is to start more things when they see that things won’t finish, because they think, “Oh, we’re not finishing enough, so let’s start more things earlier”, which then makes everything take longer on average.

So what I do during those blog posts is basically monitor the system as this queue that I mentioned, and then if you plot on a graph where the X axis is time, and the Y axis is basically the quantity of tests that you’re either getting done, or that are entering the system, you can see that the difference starts to increase the larger the difference between the rate of things coming in, and the rate of things coming out. So the greater that difference, the longer the queue is going to be, and the longer the approximate average cycle time of things is going to be.

So once you start looking at those, when you started looking at throughput, which is the way things come out, and also the way that things come in, and you match those, they are always going to have the same distance between the two bands, and you’re going to have more uniform cycle times, because you’re not going to allow queues to form too quickly. And when queues form - sure, then you can, you know, rally the troops and get everyone to actually gather the time down. Because otherwise, that difference is gonna accumulate, and queues are gonna get longer and longer and longer. And then the longer the queues get, the more terrible cycle times get, and you get even less predictable… Because if you imagine that things are queuing up, and then everything takes increasingly longer, you’re not predictable, because everything’s going to take longer and longer. Whilst if the rates match, cycle times tend to be uniform… As long as you’re finishing what you start; that’s a very important assumption.

Gary Bernhardt: Well, I will give another caveat here, or really repeat the same caveat that I said at the top, which is I run very small companies, I’ve never worked for Google, Facebook, Amazon, any of that; I’ve never worked on a very large system, I’ve never worked in a very large organization… So you know, anything I say, you have to read it through that lens. But my goal is to understand every aspect of the business, horizontally. So by vertically, I would mean like your specialization; probably everyone listening to this is a software developer of one sort or another. We are all specialists in that vertical section of the work, software development. Horizontally I would mean like can you think about the marketing, or at least an approximation of how a marketing expert would think about it? Can you think about design? Can you think about finance? Can you think about the business operations? I’m not good at most of those things, but I want to be able to think about them enough that I can make at least reasonable trade-offs across the whole business.

And a lot of times what I hear from friends who do work in larger organizations is that other parts of the business are basically setting constraints that the software teams then have to implement, and that can lead to tremendous inefficiencies, when you end up building things that don’t really matter. So you have the vertical component of a specialty, you have the horizontal component of all the different specialties that you want to be able to work across, and then you have time, which I guess in this analogy is depth, although it starts to get weird to think about… But all of these things evolve over time. and the thing that really matters is you want like the integral of complexity over time; you want to minimize the total complexity over a period, a long period, like ten years, and most of that complexity is not going to be in the first six months, it’s going to be in the maintenance. And many times, most of that complexity is not going to be in the code.

[52:03] One example I really like to trot out here, because it’s so horizontal, is suppose you’re building a billing system, and you’re using third-party to process credit cards. It has two modes. One mode is every single time a charge is run, we make a deposit to your bank account. We deposit the money immediately. The other mode is every day we do a deposit of all of the day’s transactions. You might not have any preference, or there might be some reason that the per transaction thing is preferable to you first from a software perspective. But what does the bookkeeper think about this? The bookkeeper wants one transaction; they do not want 5,000 transactions a day. And you know, it’s an easy to understand example, because even if you’re not a bookkeeper, it’s clear what it means. Obviously, it’s more difficult to do that in more subtle cases, but that kind of thinking, being able to think across domains other than your own area of expertise is so, so valuable. And it’s very valuable to an employer, if they’re smart, if they see what they’re getting.

Woody Zuill: [35:54] But we need to learn to do it. So we don’t automatically have these skills I think in programming because we’re not expected to ever really collaborate this way, except for in meetings. And there’s a big difference between meetings and working as a team. It’s the same thing as like, there’s a difference between, “We’re going to move all the furniture out of my house. All my friends will come over to help me move”, and talking about moving, right? We could talk about who will do what, and how we’re going to lift these heavier things, and so on. So we’re not going to take the couch apart, except we’re going to take the cushions off… So we’ve got to lift that up. At least two or three of us will lift it up. We’ll talk about it a little bit, but that’s not when things are going to happen. It’s in the doing when we’re going to go, “Wait, this won’t get through this door” we’ll discover those problems.

So anyways, we’ve been trained to really work alone - or we’ve allowed this to happen, I don’t know - where maybe sometimes working as a team could help. The way it happened for us originally I think is worth expressing here. I was hired to work with a team that was doing poorly, and the team lead had been asked to manage the team, to become the manager of the team. And she called me and said, “I don’t want to be a manager.” I’ve heard that from a lot of developers over the years, “So I don’t want to be a manager.” And I worked with her for some projects, maybe 10 years earlier or so. She said, “I think you would be a good manager for this team”, because I really wanted to start doing agile stuff, which I had a reputation for.

So I came, I reviewed what the team was doing, I looked at their code, I talked with the other managers, I talked with the director who would make the ultimate decision to hire or not, and I decided I’d like to work here, as long as we had some freedom. And I lined out four things. One is no interference. You’re hiring me because you have a team that isn’t working well. And clearly they’re not working well because the managers don’t know how to manage. And I did actually used those terms with them. I figured if I was that rude to them and they still hired me, I would have kind of a chance, because they’d know upfront that I don’t think very much of them.

I told them, no interference. I have the right to cancel work. If they’re working on something and they’re not able to do it, why are we trying to do it? And even bigger, they had two projects that were about a year late, and I think being a year late is excessive by about a year. And so if we’re working a year late on something, we should cancel that project, put our efforts on something that’s delivering value. I’m not going to do estimates. We don’t need to get into the estimates stuff here, but I needed assurance that they’re not going to ask me or my team for estimates. I made them sign a document that claimed that.

And then we’re not going to do projects. You’ve probably talked to people on your podcast that think of things as products rather than projects. A product is better than a project, because a project usually indicates that we’re going to gather people together, figure out what to do, do it, then disperse the team. Whereas a product is an ongoing thing. And most software, including at this company - that’s a product. It’s being used and it has to be supported for decades, or at least for– in their case, they had some stuff that was around for decades. It’s got to be changing all the time. What happens when Windows XP is no longer supported? You’ve got to do something. What happens when VB 6 is no longer supported? Things are going to change.

So anyways, this is what– I laid down those rules. And what I told the team was, “You’re going to figure out how to manage yourselves. I’m here to block the rest of the organization from you, but you’ve got to figure this out. I’ll guide you through it, I’ll help the best I can, but you’re going to make most of the decisions.” We’ve made a lot of decisions about how we were going to accomplish this. They had a lot to learn. How are we going to learn?

We set aside three hours every week to sit together and study together. And we would pick the topics, the team would pick the topics and how we did it. They happened to pick to do a coding dojo, which is very similar to mob programming. We essentially have five or six people in one room, all working at one computer, and we switched the person who’s at the keyboard and the person who’s guiding them. Because in mob programming, we have this concept of – I call it a navigator; who navigates as we go through it. As we rotate, we are just really switching out the pair at the computer. I’ve found this to be an effective way to learn things, and the team had decided themselves to use this technique. We put all the techniques we could try, and that’s the first one we tried. I think if we hadn’t done that, we probably would’ve never landed on mob programming.

We did that for almost six months. We were learning how to recognize what bad code looks like, recognize what it looked like so we could work on it and improve it and no longer write it. We were learning about how to decouple things, what cohesion is… All these sorts of things.

[40:29] At the end of that six months, we had a project that one of the developers had worked on before I got there that I was going to cancel, but I was really leaving that decision to the team. And they came to me and said, “Boy, we’ve got to get working on this again, because our second deadline is coming up really fast.” So they took a look at the code, they came back to me and said, “This is a mess.” Now they had learned how to look at code and see the problems in it. And we went into a room. I said, “What do you want to do about it?” They said, “Let’s show it to the whole team and see what we need to do. Who’s going to work on it?” Now, remember, we haven’t yet worked as a team. We have been learning how to collaborate over this time. We went into the room, we looked at the code for just a few minutes, and somebody said, “Why don’t we just start refactoring it?” And there’s this concept of Read by Refactoring, where instead of reading the code to understand what it does, you just start refactoring it. And as you refactor it, you’ll get a really good understanding. And now you have code that’s easier to read, expresses itself, easier to work on. We started doing that.

After two hours, we had gotten pretty deep into this thing, and somebody came to the room and said, “Your time’s up in here. We have this room reserved now.” And somebody on the team, with the same kind of excitement in their voice, they said, “Let’s just go find another room.” I hope you can see this, the power of this. What they were doing was saying, “Let’s turn up the good on this right now. I don’t want to quit. This is really effective.” It was in the tone of their voice.

We went, we found another room. In a company in the size where we were at, we had about 50 rooms, and most of them are always– at the last minute, you can’t find an empty room; we found one. We went there to work. And as soon as we got there, somebody said, “Well, why don’t we just find rooms for the rest of the day?”

At the end of the day, we had this tradition of end of the day retrospective where we would ask each other, “What went well today?” And everybody said, “Boy, this working together was great.” And I asked them, “What was good about it?” Well, I was there the whole day, I knew. But I wanted to hear it.

We got a lot done. It was fun. We learned a lot and it was really high quality. Any one of those is enough for me to want to do more of it. And so I asked them, “Well, what do you want to do about it?” They said, “Just find rooms for tomorrow, and whoever can come, they’ll come.”

So this is how we started doing it. We spent six months learning how to be together. The coding dojo that we followed has very strict rules. The only person allowed to speak is the one who’s navigating the person at the keyboard. The person at the keyboard is not allowed to do anything, unless they’ve been given a clear instruction of what to do. We were learning how to explain ourselves, we were learning how to listen to someone else and do what they asked, and we were also learning to keep our mouth shut; because until it was your turn to talk, you weren’t allowed to talk. Those are all good skills to have if you’re going to work with other people. It’s just like, if you’re going to learn to play soccer or football or whatever, you’re not just going to go in not knowing the rules, and you’re not going to go in until you’ve learned the skills, so you’ve got to practice the skills. And that’s what we had been doing; and we’d really been doing it by accident. We didn’t know this was leaning up to this. But after that day, how do we turn up the good? Let’s just do this again tomorrow. And for about five days, we just kept saying, “Let’s do it more”, and then somebody said, “To make this easier, let’s find a permanent room.”

It only took us a week or two for this to change into our way of working. Now, you can’t do that at your company, probably, whoever is listening to this, in that same way, but you can see the hint about how could we go from working all solo to working as a team? But this doesn’t get into the idea of how do we convince anybody else that might be in control of our time that this is worth doing?

Darren Murph: Yeah. So I’ll preface that by saying if you’ve been listening this far, go to allremote.info. At the very top, you’ll see a link to download the GitLab Remote Playbook. I was the lead author on that. It encapsulates all of our best learnings on how to go remote and how to thrive as a remote worker and a remote team. So when we’re referencing the Remote Playbook, that’s where it’s at. Feel free to share it far and wide; totally free to download, like anything in our handbook.

To answer your point about people gathering for other reasons - it’s so true. Not every industry is ideal for remote. This is definitely not an A/B comparison where I just think every company in the world should go remote right away. If you’re in a high-touch industry, if you run a hospital, if you run an auto garage, these things can’t be done remotely and probably shouldn’t be attempted to be remote. But what I’m saying here is for anyone that can work remotely, if you just started doing that tomorrow, everything else in life that shouldn’t be done remotely just becomes easier to do.

[52:17] For example, the sporting event that you want to drive to - well, now there’s less traffic on the road, so getting to the sporting event is easier, and it’s a more enjoyable experience. The marketplace on Saturday morning, where you wanna go get some local produce - that’s easier to commute to, and when you see people there, you’re gonna relish that and savor that, because you haven’t been burnt out on having to see people all week, face-to-face, in an office. Now it’s a slightly different, more appreciated experience.

It’s interesting, I actually feel like this global isolation that we’re in is going to be a boon for community, coming out of it… Because it has made people realize “I’ve been taking relationships for granted. I can’t wait to just be able to hug people again. I can’t wait to engage with people in ways that I’ve been taking for granted. How much of my community have I been ignoring and deprioritizing?” I think coming out of this, we’re going to really reassess that, and kind of check our identities at the door. How much of our identity has been tied to this physical building that you’re walking into and out of every day, and is that really healthy? I think those are gonna be great conversations to have.

Bill Dally: Sure. I’m a hardware engineer who’s been working on both hardware and software for AI in recent years. My first experience with neural networks was in the 1980’s, when I took a course from John Hopfield at Caltech, and was building Hopfield nets, and things like that. I was on the faculty at MIT for 11 years, where I built a research group that built a number of pioneering supercomputers, I collaborated with Cray on the design of their Cray T3D and T3E, and then moved to Stanford in 1997, where I continued to lead research on high-performance computing and special purpose processors for numerous tasks, including graphics.

I first got involved with NVIDIA in 2003 when I was hired as a consultant to help with what was called internally the NV50; it became the G80 when it was announced… And particularly to help on the extensions to the G80 that enabled CUDA, the ability to run general purpose computing programs on GPUs.

[04:16] I really got to like the folks at NVIDIA, particularly Jensen, and he convinced me to join full-time in 2009. So since 2009, I’ve been building NVIDIA Research, the research organization at NVIDIA, and myself doing research on numerous topics… Most recently on some of the path planning algorithms for our self-driving cars, and on very efficient AI inference.