Salvatore Sanfilippo: Basically, they started the company and approached me immediately after starting the company in Germany. We were in Germany, there was some kind of conference, a NoSQL conference of some kind… And they asked if I wanted to join the effort earlier. But I was in a too good situation, basically taking money for free, and working on my system, so I said “It’s okay for me if you do business with this thing. The license allows it, so…” But eventually, it was the most obvious thing that trying to make money with Redis would also pay me.

Chad Whitacre: I mean, it’s networking, man. This is all the social – this is social networking. This is like herd mentality. I mean, what company is not an AI company today, that three years ago we weren’t talking about it at all, right? Like, for better or for worse, humans are herd animals, companies are herd animals, and that’s kind of what we’re trying to work with here. When you talk about sustaining open source, I see there’s three levers that we can pull. Number one is commercialization. So you build a company around your project. Open source itself is not a business model, but over the past decades we’ve come up with business models. So commercialization is one way to sustain open source, to subsidize an open source project.

On the other end is taxation. So Sovereign Tech Fund is doing this. They’re spending German taxpayers’ money on critical digital infrastructure.

Suz Hinton: Yeah, I think I did a mix of them. So I did CCDC, which is the Collegiate Cyber Defense Competition. I did that with a team, and that was just the defense side of what you just said. So they do hire professional red teamers, and they’re a team from every single college that’s participating. And there’s eight of us, and you have to lock down – they give you an incredibly vulnerable network. The gist of the story is they’ve just sacked the entire IT team, and they’ve hired you on as the new IT team, and you have to basically audit the whole system, find out how it’s vulnerable, lock it down… So it’s the same as what you were saying, but we don’t have to attack anybody. But you spend the first day just auditing, trying to lock things down. They interrupt you with business requests constantly. So you’re emailing the CTO, he’s like “Oh, I want you to look into crypto as a product. Can you give me a report on crypto by the end of the day?”, or something. And so they’re constantly interrupting you and trying to simulate a real business environment where you’re just fighting for your life.

And then yeah, like you said, if they just find one vulnerability, which they will, all of a sudden you’ve got choo-choo trains on your console, and then certain other boxes are boot looping, and you’re just like “Oh, my God… It’s an actual fire right now.”

And so that was a very stressful one that I did, but the others were more about – they’re trying to give you experience with everything in cybersecurity, so there’ll be an encryption section where there’s puzzles. They’ll give you a bunch of encrypted text and they’re like “What does this say?” And it’s more about answering the questions, and completing as many of the challenges as possible. And they’re just smaller toy challenges. And they’ll also challenge you to actually get into a box, for example, and then yeah, find the flag and report what the flag was. So I’ve done a big mix of them.

And then there was the reverse-engineering one, which was NSA, and that’s totally different again. Yeah, it’s been a variety. I think I like the ones where I can just sit and tinker… But the cyber defense one - I really feel like I leveled up, especially in Linux. We spent months practicing, and running password reset drills, and things like that… And being able to audit – and we had this big notebook we were all throwing notes into for each other, and we were on a Zoom call for the entire weekend, talking to each other… It was very high stress, it took me a few days to recover. But I really feel that it forced me to level up, and I’m sure you felt similarly, Jerod.

Daniel Stenberg: Sure. It’s actually pretty easy. So the curl project is completely separate and standalone. So I don’t do business with the curl project. I support curl stuff. But I do that separately. So I sell curl services. There’s an American company called wolfSSL. And my primary curl business is just support on curl. A little bit an insurance thing. I sell a number of issues per year, and I have a guaranteed response time. So my customers, mostly actually big American tech companies, they have paid me a yearly subscription, basically, and they file issues, and I help them when they have issues. So I make sure that their curl use is uninterrupted and works well in their products.

[52:25] Usually, companies where curl use is deemed important enough for them to do this. Even though I usually, of course, try to tell a lot of my potential customers that it’s much better if they pay me to do that, rather than spend time for their developers to try to figure out how to fix curl… Because I probably do it much faster and much cheaper than they having to waste engineering time on figuring out how to do things, figuring out even just how to, or just finding or fixing bugs even more.

So that’s what I do. And then of course, in addition to just supporting, it’s also more feature development and contracts, more working closely with the product development teams really, and how they use curl, and debugging their applications using curl. Because very often it’s not a problem with curl, but maybe in how they use curl, or in the area between; it’s hard sometimes. And having an NDA and contract in place makes them feel safe to share their code with me. They wouldn’t dream to sometimes submit extensive explanations in the public bug report, because they’re scared that they’re never really special source code would be something special for the rest of the world to see. So it works pretty good. Still a challenge to sell support, so something that is free, because it’s free. So why would I pay you when it’s free…?

Adam Stacoviak: What’s cool but functional source license too coming from Sentry is their business name is not actually Sentry. They DBA as Sentry. And I know this because we do business with them. And their business name is actually Functional something or other, but their name, the business name is actually Functional something. I should look it up right now, because I feel like an idiot. But it’s got the word functional. It’s not Sentry; their business name, their LLC is not –

Adam Stacoviak: So you may have heard me say this on a podcast potentially, if you listen frequently enough… I don’t say it too often, but I’ve definitely said it enough, where I think our listenership knows about it… But I call it my micro moments, essentially. And so when you work from home, versus a dad who has to go to “the office”, somewhere that’s different, you will go potentially drop them off at school, if they’re old enough, maybe like 7:30 in the morning, go do your day, stop at a coffee shop, go to your office, go to lunch, be with coworkers, leave, be in traffic, get home. 5, 6, 7 o’clock at night, whatever. Some version of that is true for most people who leave and go to work. But for people like you and I who get to work from home, we’re blessed with these micro moments. You get up in the morning, you’re hanging with them, you get some playtime in the morning before the work time begins… You take a break at 10 in the morning to disconnect, and go get some coffee, or refill, or water, or do some yoga… Whatever you’re doing whatever for disconnecting for a moment, where you just don’t walk through the living room and just like put your blinders on and ignore. No, you probably stop for a moment. I get a snuggle anytime I can get my kids; every moment I have my kids is priceless. Absolutely priceless. I just love hugging my kids. They’re amazing. And so I call those times my micro moments. And if I was away from the home, working somewhere else, I couldn’t enjoy those micro touches, those micro noon things, or having lunch with them when they’re home.

[00:16:23.13] My one son’s in school five days a week now, my other son’s in pre-k three days a week. So I still get Tuesdays and Thursdays micro moments with him… But you kind of get the point, right? Like, that’s the blessing, is the real – I guess really the advice is when she knocks on your office door, unless it’s absolutely crucial for you to say “Don’t come in now…” Like, I used to have my son come into meetings. Like, I would be in a meeting with – I can recall before having meetings with like Sentry, or Fastly, or whatever our sponsors were… And I’m like on these phone calls, and I’m like “Hang on a second, my son just knocked.” And he comes and says hello, and he like maybe waves at the camera…

Hey, this is my life, right? If you want to do business with me, you want to do business with all of me. And all of me is my ethics, and my family… And so I would never deny my son coming in and bombarding, unless it was just like something that just didn’t make sense, like a podcast. If he came in right now, I’d be like “Bud, you know you can’t come in now. Like, now’s not the best time ever.” But we also do edit, so I could edit it out.

I just would say “When she knocks, answer.” Let the office be somewhere that she is welcome. You know, that it isn’t “Dad’s working. Don’t bother him.” It’s more like “Wow, I can be curious, and I can explore, and I can ask questions, and I can fiddle with whatever is available”, whether you’re working with hardware, or working with software… “Just be curious and ask me questions.” That would be my advice to you.

Jerod Santo: Yeah, exactly. And they’re doing that almost purely for cost savings. I would say that’s their primary motivation. And the cost savings are significant if you are in that group of people who have a stable business, with known growth trends… Because the cloud provides a lot of things, but the main thing it gives, especially for startups, is that dynamic scaling. You know, scale it up, scale it out, scale it back down again… That’s really powerful for companies who don’t know what their server demand is going to be. So that’s one.

And then there’s also this movement of taking cloud APIs to either your own hardware, or to collocated and cheaper VPS-style infrastructure. So now you’re still – everything we learned about the dynamic scalability of cloud APIs and AWS and others have given us, what if we could use those same APIs, but build them on our own hardware, for instance? That’s another movement. And that one’s about – I think it’s also about probably privacy plus cost, but without losing a lot of what the cloud provides. It’s trying to have the best of both worlds, I guess.

Justin Garrison: [47:50] I feel like there’s a fine line here for - when you mentioned the Dependabot model as being a good thing, it’s not my general experience. Because it’s just like “Oh, it’s just sending me more work to do”, at some level. Like, “You know what? I don’t have time to focus on that thing, because I have something else to do.” And the more automation, the more work you’re putting on someone. And in a lot of ways, best practices is very expensive… As we were talking about already - timeouts, and pod disruption budgets, and all these things are like, you don’t have to worry about them originally, because you’re like “I’m just getting started.” And then you mature a little more, and you’re like “Oh, I’m gonna get used to doing these things, and I’m gonna get a little bit better at them.” And I’m teaching my kids how to do things, and they’re not doing that right the first time, but that’s okay, because they’re still trying, and they’re still putting an effort. And then later, I’m gonna correct them a little bit. But at some point here, especially with infrastructure - and infrastructure has such a large impact on an organization, downtime… Companies can go out of business with major outages. This is a very critical piece that you can’t just automate everything, because you’re going to affect someone up the stream that you don’t know what they’re using or why. Or even a customer, of like “Oh, I didn’t know you had that API. That’s crazy. This is a surprise to me. This is amazing.” But that automation fine line of how much is too much, how much are we actually going to put on developers and just say like “Just accept these.” Because 90% of people do. “9 out of 10 developers accept our PRs, so you should, too.” It’s like the sales pitch of like buying a toothbrush, or something. Like “Hey, you should just do it, because everyone else did.” And at some point, you get numb to the automation of like “I’m not learning anything from this. It’s not actually helping me get maturity out of it. It’s just noise to be able to say, “Yeah, sure. Whatever.” I’m not the one that fixes it either, because at the SRE side of it I don’t know if developers are getting paged for that. Or if the SRE team fixes and debugs everything, because they watch Git a lot closer to what a developer might be doing at like a global scale. How does that problem show itself inside of a company that is building something for other people that are also trying to automate on top of you, and as a product that has a hosted service that needs to be up and available? Because I’m sure your availability needs to be higher than your customers’.

Allan Jude: …that BSD is harder-core… But for a long time, it was definitely if you were doing business with your computer, if you were making money hosting something, you wanted to be doing that with a BSD, and then Linux was this kind of – it’s what the kids are doing, but it’s just a toy. And that has changed a lot. It’s been 20 years, and more than that, and IBM spent a couple billion dollars making Linux better… And then separately, since then, bought Red Hat. But… I think part of it is also the assumptions that BSD makes – I guess the best way I’ve heard it phrased is Unix doesn’t stop you doing something stupid, because that would also stop you doing something clever. But it does mean that there are no guardrails to keep you from shooting your foot off.

Dan Moore: And what you have to realize as a new dev is that your manager wants to help you, unless you’re dealing with a toxic manager. And I’m gonna set that aside. Let’s assume you’ve found someone who actually wants to help you, but they’re busy. And so first of all, asking for a one-to-one, which is - for your listeners who might not be familiar with it, it’s just like a weekly, bi-weekly regular meeting. And you could do it with 10 minutes - that’s probably a little short, in my mind; it shouldn’t be an hour… 30 minutes is kind of the sweet spot. But it’s a regular check-in. And you as the new dev should manage that. Your manager will come with things, but you should think about what you want to communicate to your manager, what you want from your manager, how to communicate your challenges, and not just your challenges, but like what you’ve done to address those challenges. And you’re building that relationship with your manager so that when tough times come, or when they have new opportunities, they think of you. So you’re opening up those communication lines, and the important thing is - again, not all new devs are right out of school, but when you’re in school, the curriculum is fed to you. When you’re in school, the teacher has a responsibility to take care of you. And a manager is not the same as a teacher, and you need to be very aware of that.

[00:36:09.29] Actually, one of my other popular posts is “There no adults in the room”, which was – it was a little disturbing to me when I realized it, but I think… I came out of school and I had the idea that like someone knew stuff. You walk into business, and people know how to solve these problems. And that may be true in kind of a relatively static business, with a really good lineage… All I can say is I’ve never been in one of those companies. All the software companies I’ve been in, people are discovering new problems, and solving new problems all the time… And that basically means you look around and people may have more experience than you, but you’re all the adults in the room, and you all need to take responsibility towards solving those problems.

Cory Doctorow: And one of the things I talk about in this book, when I’m getting into the theory of change and how – like what is the actual path that we get to from here to there, one of the levers I think we can pull on is government procurement. Because the government is the biggest buyer of many technologies, and even on technologies where they’re not the biggest buyer, they’re such an economically significant buyer that without their business, a lot of firms would fail. And as just a matter of like sound public administration, governments could and should say “If you sell us something, and then we try to buy like an add-on, or a plugin, or a maintenance or a part from someone else, you aren’t allowed to sue them.” Like the IPR that you have to sign when you go to the W3C, where you say “If you have a patent that overlaps with a standard you’re making, you’re not allowed to use that patent against people [unintelligible 01:03:54.09] the standard”, we could just say “If you want to do business with the public, you can’t constrain the public’s ability to maintain and extend the functionality of the product you sold them.” And this isn’t even a new idea. So Abraham Lincoln only bought interoperable rifles and tooling and ammo from armors that would agree to use standard sizes… Because it’s embarrassing to be the commander in chief of the Union army during a civil war and to show up at Gettysburg and say “Sorry, boys, war’s canceled. The sole supplier for the bullets isn’t opening this week. They had a big booze-up and they’re all too hungover to make bullets. Go home, I’ll call you when they’re ready.” So this is just good administration. And yeah, companies are gonna go “Well, how dare you constrain our ability to exercise our rights?” And the correct public response is “Oh, you don’t have to. You just can’t sell to us if you’re going to. Because yeah, your shareholders’ priorities are important to you, but the public’s priorities are important to us. And if you’re too emotionally fragile to be a supplier to the US government, find a line of work that’s like more suited to your delicate sensibilities.”

Chris Benson: Yeah. And as a non-expert, but the Defense Production Act… Listeners may recognize that, because they’ve been hearing about it a lot over the last year, because it’s been a point in the Russian invasion of Ukraine, it has been repeatedly cited as a mechanism that the US government can use to increase production to support that effort, our allies, the Ukrainians. And so you may have heard that before. And from that I know, as a non-attorney, non-legal mind, that it gives the US government broad powers on requiring commercial companies in the United States, or that we do business with, to meet a certain set of criteria, and there are things that they can be told, “You must go do this, because it’s in the interest of our national security.” And so it’s a fairly sweeping thing, is my understanding. Since it has been referenced here, I agree with you; you not only have the executive order, but you also have reference to a fairly strong point of law from Congress.

Nithya Ruffnithyaruff: We do GitHub Sponsors… We support foundations like the OpenSSF, Apache Foundation, Linux Foundation, Project CNCF, that sort of thing. And we have lots of developers who are behind the scenes actually contributing to projects. It’s never enough, because all of us consume a lot, so we have to keep working on that. And most businesses, not just Amazon, is challenged with business justification; why should I dedicate five engineers to doing this work? Because there’s so many competing needs. Customer needs, and product development needs, and so on and so forth.

[41:52] So we work hard as an OSPO and open source marketing team, who’s downstairs at our booth, to work with businesses, to educate them on why they should be involved, why they should contribute back, what’s the business case for setting aside people to do it… So those are the ways we help the business do more with open source. But we have to have a good business decision and argument, because businesses know business, and they need the return on investment or justification for why they should be involved.

Kris Brandow: [laughs] I do have some nuance for the preemtive over-engineering as well, because I think the thing is that people – once again, especially with business requirements, I think that the unknowable unknowns are really known knowns… But I think the things that people want to pretend are unknown unknowns or knowable unknowns are actually known knowns. [unintelligible 00:57:16.10] example of this, and like one of the things I’m really annoyed about constantly is things like content negotiation for HTTP. There will be a point in the future when you will want to change the structure or format of what you’re sending over the wire, or what you’re receiving over the wire, and if you don’t build content negotiation from the beginning, it’s going to be nearly impossible to put that in later. So you really do build yourself into a box with something that you will want to change. I mean, we see this all the time with like API version 8, where it’s impossible to get rid of an old API version, because you haven’t built yourself a way to easily move forward.

[00:57:53.15] And I think a really good example of this is what is happening with Qwik as a protocol, where they’ve already revved to version 2 of Qwik, because what they’ve realized, the problem with TCP was is that they never revved the version. So when they needed to rev the version, they couldn’t rev the version, because everything had ossified into knowing this very specific image of TCP. And if you changed it, everything would break. So with Qwik, they’re like “No, no, we’re going to intentionally rev the version, we’re intentionally going to make a new version of this.” Not adding any new features, but it just changes some bits in the image, so that middleboxes and things can’t ossify around this specific thing.

So it’s like, you know in the future you will want to change things. That’s a given. We will want to make advancements where you want to change how Qwik works, change how all these protocols work, so we put in place the things now to enable ourselves to do that in the future. So I’d say that’s one bit of preemptive engineering that is extremely important to do. That’s also the thing that engineers never want to do. If you build – like, not even many of the HTTP frameworks don’t even properly do content negotiation. I actually was looking this up, and it’s kind of hard to find a library in Go that will properly do HTTP negotiation as far as what the spec says you should be doing. But it’s a very important thing to do to give yourself the flexibility to move in that future. Even if it’s just around optimization. If you have one API call that is a lot of boilerplate data, and you don’t want to be sending this giant JSON blob, being able to just be like “Okay, I’m just gonna make a little binary protocol, and we’ll do some content negotiation. If you can accept this, then I’ll send it to you.” That’s a very easy way to be able to pick and choose the things that are taking up a lot of bandwidth and reduce their bandwidth a lot, in a way that’s backwards and forwards-compatible. But if you don’t do that content negotiation in the beginning, you’re gonna have stuff that’s just like “Nope, I expect Jason, and I will spit JSON back at you, and I don’t care what you tell me, I will always just do this.”

Landon Gray: Oh, goodness, no. That’d be an awful way to start. Yeah, so for me - when did I first become interested in coding? My parents, they owned a small business, and I remember one of their favorite pictures, and actually one of my favorite pictures too, is me sitting in front of the computer, the first computer that they bought for their business, with a Twizzler in my mouth. And I’m sitting there, kind of looking away from the computer, staring at whoever’s taking the picture.

Adam Stacoviak: [12:18] Yeah. Because when I came to DX, or the domain getdx.com, I was thinking “Okay, is this a company that helps me ensure that my developer-facing dev tool has a great experience? Or is it something else?” Because I’ve been in product management; not really engineering management, necessarily, but product management. And so I’ve interfaced with software developers; I’ve never had to measure necessarily exactly how efficient engineering is personally, but I’ve had to measure how efficient we are at building products… Which is kind of the same, but not quite the same always, because that interfaces with business, that interfaces with marketing in some cases, that interface with product alignment, and product-market fit… Which is not necessarily just simply “How do we build this thing? How efficient is this?” Simply just the developers, the engineers building this thing. And so I’ve never really been in the camp I’m measuring specifically productivity. Why not just call it developer productivity though, or something else? Like, why only murky the water any further? Who named this thing, Abi?

Jerod Santo: Sure. No, I think you’re well-positioned right now. And speaking of bets, just to maybe pat myself on the back and give you some more credit… I was just coding on our Elixir app today, and have been just over the last couple of weeks… And thinking, “I made this bet more than seven years ago on Elixir, and Phoenix for our platform that we run our business on.” And yeah, we’re a small website, with very limited scope, and all these things… But it feels pretty good to just – and I’m still productive, I’m still enjoying it, and it’s still working for us, and I still like the language… I mean, seven years is a long time to be working with one thing and not to have – I mean, of course I get wanderlust. I kind of wonder “Well, what if we rewrote this part in this thing?” or “Maybe this was better”, but all in all, José - I mean, thanks again for creating these tools, because I’m just like a happy user, seven, eight, probably a decade later - I can’t see us changing anytime soon - just writing Elixir code and running our business with it. So that feels like a good bet.

Cory Doctorow: Yeah, Google Wave, right? Google Video… All of those. Meanwhile, their ad tech stack, their server management stack, their mobile stack, their video - all of that stuff, docs, all of it, bought from someone else. Even the satellite stuff, bought from someone else.

So the poultry packers, they gobble each other up, and now there’s three of them, and they have divided America into three territories. They don’t compete with each other. If you have like a cable modem, you know how this works. There’s only one company offering your cable modem. Except if you’re a poultry farmer, there’s only one company that will buy your birds. And they determine everything about how you grow those birds. So you have to buy the chicks from them. They tell you what your coop has to look like; they give you the floor plan for it, the blueprints. They tell you when the lights go on, when they go off, what spectrum light your light bulbs can emit. What food you can give them, what schedule they’re fed on, which vets you can use, what meds those vets can give them… Everything except how much you’re gonna get paid. That you find out when you bring the chickens to market. And because they have regionwide insight into the poultry production of you and all your competitors, they give you money that is calculated to the penny to be just enough to roll your loans and do it again next year.

They do things like A/B splits, where they’ll be like “What happens if we give chickens less food? Will they still get as big?” And they don’t tell you that you’re in the experimental arm. And you go to market with sickened chickens, right? “What happens if we don’t give them this medicine?” and your chickens all die, and they’re like “I guess you’re on the hook for that.”

When people speak out about it, they are struck off. So if a poultry packer refuses to do business with you, and you grow chickens, and you’ve got a million dollars worth of debt for your physical plant, you’re dead, right? There’s nothing you can do.

And so farmers who speak at like state congressional hearings, state legislative hearings get struck off. And there was one guy who spoke out on a state regulatory proceeding, who after he got struck off, he was like “Well, I guess the one thing I can do is I’m really good at fixing these standard coops that the poultry processor requires.” So he went into business doing that. And they told every farmer that if you hire this guy, we will not buy your chickens either. They just destroyed him.

Matthew Boyle: Exactly. And I think for me, that’s the most valuable piece of all of this is. If you can get it right, your system models, the business space, the problem, and you can reach a consensus on sort of various language, it means that you can have really great conversations about how things actually work, and exactly what needs to happen next with business colleagues, without having to do sort of this weird translation in your head between what you wrote in code and what happens in the system. Like, if you write your code well, and it’s structured, you can say, “When you press that Convert button, a lead becomes a customer, and after six months, we convert that customer to whatever, and then if they leave, they count as churned.” All of those words have specific meanings to our team, and if I’ve used those words and that language in my code, it’s very, very easy for me to kind of have that conversation and everybody know what I meant.

Marcin Kurc: There were many, many different things that we had. And people used to do that, right? And now you cannot – “I’m just not gonna do business with you unless you adhere to certain frameworks and certain certifications”, and so forth. And from a reliability perspective, we’re really getting close to a very similar approach. “Tell me how you architected your systems, how I can trust you that you did the right thing. You built this on AWS - that’s great. But is it multi-region? I need to see some data that really gives me a good idea, or comfort that I can make a big investment.” Because an enterprise is not gonna go there and say, “Oh, three months later we can just switch all of our systems to something different.” That doesn’t happen, right? It’s a big, big investment.

Matthew Lanham: Absolutely. So the INFORMS stands for the Institute for Operations Research and Management Science. It is my favorite professional organization. So it’s been around for many years, and we used to call it Operations Research and Management Science, but now we refer to a lot of the stuff that we’ve done for years as analytics and data science. Within INFORMS, there’s also a certification program called the INFORMS Certified Analytics Professional, or CAP. So I’m a CAP, I was one of the first CAPs many years ago… And basically, the whole idea with the CAP is you don’t have to be a technical person to get this cap, or you don’t have to be just the business person. It’s really for everybody, and the whole idea, how it came about was the people at INFORMS worked with business professionals from all different areas of analytics and data science operations research, to identify what are the key kinds of tasks that you would do as a professional. And what they did is they came up with basically seven domains; business problem framing is the first one, than analytical problem framing, knowing your data, methodology or approach selection, model building, deployment and lifecycle management.

[06:16] And if the audience hears those things, they’re probably thinking, “Hm… That sounds a lot like CRISP-DM, that we all heard about in school, or at some point following a process. And that was the thing, is a lot of times when we’re working on these problems, you’ve got to follow a process. And they kind of extended that CRISP-DM framework, and there’s just a lot of tasks within there that we hope people are aware of, and think about when they try to develop solutions in practice.

Jerod Santo: We’re just trying to come up with business models for you. No, it’s 30%, Will. It’s 30%. Come on, you have to be an oil baron, an app baron.

Adam Miller: Yeah, a lot of people wanted to buy bikes, so I guessed that one totally wrong early on. And so 2020 was just a great year. And I felt very lucky. A lot of other businesses were affected by COVID, and it lasted a lot longer than a few weeks, like it did for us. So I felt incredibly fortunate, and because of that, I wanted to really do a good job and take advantage of our good situation as a business with really high demand.

And so 2020 was pretty great, until about fall 2020, and all the inventory we had or the inventory that was being made started to – we got that stuff out the door, and lead times started creeping up. They went from 30 or 60 days to 90 days. I remember when derailleurs and stuff - those lead times went to like 120 days, and we all freaked out and thought, “Oh my God, how are we going to run our business with four month lead times for these parts? That’s just wild.” And in the course of the next year, and even now, a lot of those lead times are between 500 and 700 days to get parts, which is just crazy. We’re placing orders out through 2025 for certain parts right now…

So kind of zooming out, we had to figure out how to run a business and make a plan, and figure out what we were going to sell and order all this stuff… I mean, order products that haven’t even been designed yet, in some cases. We had to grow up really, really, really quick, and so I think as a business we grew up by 10 years in the course of a year or two.

Adam Stacoviak: When you look at the market then, currently, any business with hundreds or thousands of engineers, if it’s when, what’s in place currently? When you look at the market of addressable servicing, I suppose, what is it being used? If it’s FireHydrant at 2%, or 5% - I have no idea where you’re at, but what are people using today? How are they getting by? If it’s when they’re clearly having incidents, things happen, how are these teams managing these things? Is it mostly internal? Is it mostly just ad-hoc? Is it nothing at all? Is it maybe a small sliver using FireHydrant because you’re newer to the game and you’re still gaining market share? How do you break it down?

Montana Low: Yeah, I mean, it’s kind of a long-winded story. It’s definitely not the first time that I’ve taken a stab at machine learning infrastructure and trying to make things simpler. I joined Instacart about seven years ago. I had been a chief data scientist prior to that, and mostly it was all self-taught; I didn’t really deserve the title. But at small startups you get to pick your own, and that’s what I wanted to be when I grew up, so to speak.

Anyway, when I joined Instacart, it was a really exciting time. There were a couple dozen engineers in the company, and we were getting large enough that we needed to move out of a monolithic Rails app into more of a distributed architecture that would be horizontally scalable. One of the first projects that I did when I started there was pulling all of the product catalog data out of the single Postgres database that we had, moving it into a new Postgres database, but then fronting that with Elasticsearch, so that we would have this horizontally scalable, catalog system that could power the whole eCommerce website as we added thousands and thousands of stores to Instacart platform.

[04:16] And that was really exciting, that was really fun. I learned a ton. I had worked with natural language processing and search prior to this. I had some experience with Lucene and distributing that, so it was cool to get some new technology and to really leverage that and to start unlocking data scientists and how they could impact the product in a more direct way.

But data science was very nascent at Instacart at that point. Fast-forward a couple of years, I got to sort of lead several SWAT team initiatives around the company to pull out more systems into more distributed architectures to help stitch these things together. And as our team grew, we brought on a VP of engineering, Jeremy Stanley, who’s a brilliant data scientist; one of the best people that I’ve ever worked with. And he sort of put out a call for help of, “Hey, if anybody can help us get some of these models that we’re building on our laptops to actually impact the product somehow, we’d love to talk.” I got to work very closely with him to help figure out how we would productionize a lot of these systems, and to help build a lot of the tooling that data scientists need. If they’re going to use Python, should they be using Conda? What does a pip install actually look like? How do you get that into production? The whole nightmare of dependency management and lifecycle management of models when they’re not just built once, but they have to be rebuilt continuously, with new data as it comes in. And then you have to get the feature data to the actual model, but it can’t be the data that’s coming out of your snowflake warehouse - at the time we were on Redshift - because that’s too slow and latent.

We were learning everything and building it on the fly, and it was chaotic, but fun. Actually, we published a lot of that work, and a library called Lore, which was Instacart’s open source platform. Now, as the ecosystem evolved around us really quickly, over the last, five or six years, things have changed at breakneck speed; there’s a new platform library company coming out every day that’s doing something really cool… And so we grew that internally, but it didn’t really make sense to keep a lot of the stuff that we had built, because original libraries built better embedded solutions; they actually built the bridges, and we could take some of our tape and glue out of the middle and things got simpler.

But fast forward another couple of years at Instacart, the original system that we had built with Elasticsearch as the heart of our data architecture, our data infrastructure - it really became like the beating heart. If anybody had any data, and this included all of our machine learning feature data, we would just shove it into the Elasticsearch document. And then anybody who needed data would just get it right out of the Elasticsearch document. And so our documents grew to several 100 fields, and many of these fields themselves were nested JSON documents; they could be tens of kilobytes of additional payload data, and so our Elastic document size blossomed.

And Instacart, I think, is somewhat unique in a couple of its constraints. One is the real-time nature of the business. Instacart is not like Amazon, where if it’s not in stock, Amazon tells you it’s not in stock, and it’ll be two days late. Instacart, can’t do that. If we say, “Sorry, we’re not gonna be there in 45 minutes; we don’t have your entree that you’re planning to cook for your family”, your family is gonna go hungry, and that’s going to be a really, really bad customer experience.

[07:52] And so everything at Instacart is built, from the product, from the machine learning, it has to be rapid, and online, and responsive. It can’t be an offline 24-hour batch job that we get around to eventually. And so I think that that is a really challenging technology problem. It’s a really challenging business and product problem as well. At the same time, Instacart is a platform for hundreds of different grocers throughout the country, that have tens of thousands of stores, all that have different inventory. So it’s true that we have one product, one box of Captain Crunch cereal; it has an image, it has nutrition facts, it has this universally true data about it. But then it also has little facets of data that are specific to every single store, like what is it actually being sold for in that store? Is there a manager special that day? Is it in stock on the shelves, or did it just they just sell the last box? And so if you think about this from a data architecture perspective, it’s a pretty classic - you have two tables, one is your product, one is your offerings, you join those two tables together, you denormalize that data into Elasticsearch… Easy. Except, we actually have a million products, we have 10,000 stores; you multiply that together, you get ten billion. And so all of a sudden, this is an incredibly large Elasticsearch cluster, and it’s growing very, very rapidly… Because Instacart was at the time expanding into whole new verticals beyond grocery. It was basically all of retail. And it’s like, “Oh, now we have like this whole other dimension, and we want to join whole other things… How are we going to scale the cluster?”

I remember seeing a graph plotted of like our Elasticsearch capacity increase per node added to the cluster. There were some diminishing returns there. You don’t get perfectly linear scaling when you add nodes to a cluster. At the same time, - well, that curve is asymptoting and flattening. There’s another curve that’s coming up exponentially, which is Instacart’s growth rate, both in terms of writes to Elasticsearch in times of timely ingestion into the system… And this is another thing that Instacart had contractually agreed to providing updates to the website on behalf of retailers in very short amounts of time.

One of the things that I’ve heard that Walmart for instance does is they have a green/blue deployment for Elasticsearch, and they will spend 24 hours filling up their green cluster with updates, they’ll flop over to it, all traffic gets that, and then for the next 24 hours, they will refresh their blue cluster, and then they’ll flip over to it. So you can just rebuild your cluster every night, flip back and forth between the two, and that way you avoid a lot of the incremental update penalties you get in a Lucene index, in this inverted index world. That’s not a strategy, for example, that Instacart can employ, because of the tight time constraints.

And so we were all sitting around, kind of pulling our hair out, trying to figure out how we were going to out-scale the business with our technology, and getting a little desperate, honestly. I think Postgres was not the first idea, but eventually, we did decide that, fundamentally, this is a joint problem. If we could do the join at read time, rather than index time, then that would potentially eliminate a huge amount of work, because many of the documents that we were joining and indexing were actually never read before they were reindexed again. And so we could actually, by not doing those useless amounts of work, we could reduce the amount of work in the system substantially.

And so we built a prototype for this system of what would it look like – you know, people have been sharding Postgres for decades; it’s something that people know how to do. It’s a little finicky, you have to get it right… But even more recently, with things like TimescaleDB and Sidus Data - they make sharding a lot more manageable, a lot more tolerable. And so we started looking at some of those options and we started to look at the – Postgres also has these full-text search capabilities built into it. They don’t nearly have the bells and whistles of Elastic, but the basics are there.

[12:14] So I started talking with our NLP guys and our search engineering team, and saying like, “What are we actually using in Elastic? What machine learning functionality in Elastic do you have?” And like “Oh, we tried, but it fell over, so we couldn’t do it. We couldn’t actually use a lot of it. It’s too much load on the cluster, it’s already on fire…” And so what we learned is that most things happen at the application layer, and most things are like joins between various microsystem data stores, feature stores that have gotten kicked out of Elasticsearch because they were creating too much load on the heart of the company. And then we would join those all at request time at the application layer; sometimes that would take eight seconds for our P90. It could be quite slow. And sometimes what we would find is like we would do this eight seconds of work, and then at the last step we would find everything, all of our high candidate high quality results were out of stock, so we would have nothing to show, because we had to implement several constraint layers upstream. When we really got into the nature of the system that we had built, that was this distributed machine learning beautiful beast, it was not it was not a pretty picture. It was a very complicated picture.

And so we just said, “Well, we don’t really have any other options. We’re going to try to do this in Postgres.” We stood up our prototype, we had it running… We were shadow-testing its search results against Elasticsearch, what we were getting back and forth. We were finding lots of data ingestion, bugs, bugs that had been in our data pipeline going into Elasticsearch for years; we discovered several of those… Because we had to rebuild the pipeline in parallel for Postgres; it was a totally different pipeline. Obviously, we found several bugs in our Postgres implementation as well; when you’re doing a second system rewrite, that’s never a fun thing. I don’t typically recommend people go that route.

But things were looking okay, until the pandemic started. We had plotted out the intersection of those two curves looking something like a year out that we would have to figure this thing out, and to kind of experiment and prototype. And we went through that year of growth in about a week, the first week of the pandemic. I remember getting paged the next Sunday - everybody does their grocery shopping on Sunday morning, so if there’s a new load issue, it’s going to be Sunday morning when we get paged. And so I remember getting paged, and Elasticsearch was timing out 30-second requests… We had stopped indexing, so we were in danger of not meeting SLAs unless we could get indexing going again, and traffic would go away.

We did all kinds of things. We thought about putting up a Stop sign on the website and saying, “Sorry, Instacart is full. You have to come back another time.” Luckily, we never had to actually deploy that. Luckily, we were able to scale our way out of the pandemic… But it was a lot of work.

So while we were in the middle of this incident, we said “Well, we’ve got this other cluster over here. We think that the results are about the same as the cluster that we’re using, that’s kind of dead right now, because it’s just timing out, a hundred percent load…” So we just flipped the switch, put all the load against Postgres, and started using it. Of course, it immediately went to 100% CPU utilization, and also caught on fire… But we were able to find a few missing indexes for some long tail queries that we hadn’t really optimized, and within a couple of hours, we got that cluster to a point where we could actually serve traffic again. And so that was that was really exciting, to really get the system bolted down to a couple of months after that… But for the most part, we had sort of shifted what was the primary system and what was the secondary system. Elasticsearch from that point, going forward, was really the backup to this new system that we had. And we had a couple of incidents with the new system, as we started throwing more and more data into it. Because after we did the original optimizations, we got down to like 4% CPU usage or something in the Postgres cluster. And it was it was vastly underscaled compared to our Elasticsearch cluster. I mean, it was just tiny compared to it is. It was really amazing.

[16:22] But at the same time, I mentioned all these other features, stores, model stores, everything else that we had… And all of those, whether they were Redis, or whether they were Postgres or Cassandra, those were all melting down as well. Those were not horizontally scalable systems; we learned a lot about scaling every system we had, whether that was RabbiMQ, or Redis, or… If you can name a database, we probably tried it at some point at Instacart.

So we had lots of fun, but our solution in this case was basically like “Figure out what your database is that has the most CPU usage, pull all the data out of it, and dump it into this new, horizontally-scalable Postgres cluster that we have.” And so we just did that over and over again, and we barely kept ahead of our doubling week-over-week growth curve for the next eight weeks. Like I mentioned, sometimes we missed optimizations, and we didn’t really have the time to vet and test the system that we were building like we should have, or could have… But I think that we did the best that we could with the resources that we had.

And we spent at least a year after that iterating, adding more, really unlocking some new machine learning for our search team that we could now do. And we didn’t get as far as I really wanted though, because at the time there was – there is a library called Madlib, which is an Apache Foundation project; I think it’s ten years old, it’s been going for a while… But there were some constraints at the time. They were locked to a specific older version of TensorFlow, I think. My memory is fuzzy, I didn’t get a lot of sleep back then. But we weren’t able to actually take a lot of our deep learning models and put them into Madlib and run them inside of the database to eliminate some of the microservices… So we actually kept quite a bit of the microservice architecture, and kept building around that. But it kind of bothered me, because we were able to clean up so much of the distributed system. I felt good about – the system that we ended up with was much better than the system that we started with. And it was kind of full-circle for me, coming from like – you know, I joined Instacart, and I was all about distributing everything… But by the end, I was all about pulling everything back into one fairly monolithic system.

So that was kind of eye-opening for me, about the complexity, both organizational, but also technological, that these systems can develop, and how powerful it can be if you can simplify the system. For example, when we were on the Elasticsearch pipeline we had a dedicated infrastructure team, we had a dedicated catalog team just to the pipeline. We have a dedicated search team, dedicated machine learning engineering - all of those resources… And we had upstream of that catalog data acquisition specialists that would get new kinds of data to do new kinds of products and services, or add new features to the website. But it took multiple quarters of planning and execution from the – like, you can sit a few product managers around the room, they conceptualize… Like, “Hey, we want to add this feature to the website.” Like, “Okay, we’ll go source the data, we’ll get the data into the pipeline, we’ll ingest it into Elasticsearch, the search team will start using it, and then they’ll display it. And they’ll start work this quarter, the search team will get to it the next quarter…” “Oh, wait, we don’t have the data in the right format. Let’s circle back for another quarter of this whole process.” And so it was really, really problematic.

Kris Brandow: I mean, I think generally most companies are going to sponsor something that GopherCon wouldn’t have a problem with this type of benefit or something. I mean, at least I hope they wouldn’t. I think if they do have a problem, I think that’s like a signal that, hey, maybe this is a sponsor that you might not want to have, and that’s another– once again, when you want to be inclusive, when you want to grow a diverse community, there are sacrifices that have to be made, and sometimes that’s leaving money on the table from places that are not going to be supportive of the things that you value, right? So if you do value making it known that community members that have, say, multiple partners are welcome at the conference, and you have a sponsor that’s like, “Well, I don’t want to support that, so I’m not going to give you money for that”, that’s a choice you have to make. Do you actually support that, or were you just saying you support it, until it came push came to shove and you’re like, “Well, I have to go leave this money on the table”, so now it’s like, “Okay, I actually don’t support this type of thing”?

So I think truly if you as a conference organizer - do support this sort of thing? And I think it applies broadly to the Go community as well, on things that aren’t – anytime it comes with money; even deciding to work at a company… It’s like, when it comes down to money, that’s where you really see what your values are at the end of the day. So it’s like, are you willing to just let that money walk out the door and support this thing, or have to figure out more logistics? Because yeah, there is associated costs with– if you really are giving someone use of a space, that’s another room or another space you have to get, there’s more cost for Wi-Fi, since you have more people on it. Obviously, if you’re giving them food, that’s like half the cost of your ticket right there, so that’s a substantial cost. So it’s like there is cost associated with doing this sort of thing.

I hope that with the sponsors of things like GopherCon or GopherCon Europe, that they would not have a problem with calling out this type of arrangement. But yeah, I don’t know. But yeah, if it was me personally, like if I was organizing a conference and a sponsor said, “Hey, I noticed that you have this benefit for people’s partners”, and having this discounted ticket for them to use the Wi-Fi and get some food, I think it’s fine if you say your partner or your spouse, but if you allude to the fact they might have multiple partners, “I’m not okay with this”, I’d say, “Okay, well, here’s your money back. Have a nice day.” I don’t want to be doing business with someone that’s drawing those types of lines around– at the end of the day, what types of relationships are conference attendees allowed to be in? Why should you, as a sponsor, get to dictate that sort of thing? So yeah, I guess that’s my rambly point of view on that.

Johnny Boursiquot: I have a very painful memory of a project which started out green, quickly became brown; or it started browning, but we were moving so fast that there was “no time” to go back and face it. We just kept rolling. I think we’ve touched on some of this over previous episodes, but we had a proof of concept; indeed, a brand new project, greenfield. New GitHub repo, and everything. Brand new stuff. And we sat down with the customer, identified business processes, and this is what this thing needs to do, this is the problem domain, and all the good stuff, all the due diligence, the architecture… We didn’t wanna do too much architecture upfront, we didn’t wanna do big design upfront (BDUF); we’re trying to avoid all that, we’re trying to be agile… Gosh, that terms sort of gets under my skin now, but - we were trying to do a little bit of design, a big of development, a bit of design… So basically trying to do the right thing; trying to not run before we can crawl.

Despite doing all these things, a year later we ended up somewhere we didn’t originally think we were gonna end up, despite all the requirements gathering, despite all the sitting down with the customer… Because again, in time, they kind of changed the scope of the project, they changed the nature of the project, they had a pivot in there… And this wasn’t like a startup or anything; this was like an existing business, with a new business unit that’s trying to do things. So this was like a well understood problem domain.

They didn’t sound like they were looking for product-market fit, or anything like that, which are things that cause a lot of change, a lot of churn. But the conditions for this were effectively perfect. When you establish a business, who wants to build a new piece of software to basically take over manual processes? There’s nothing more clear-cut than that. Yet, a year later we ended up in a place that nobody on the team was happy with. Unfortunately, somebody went home and over the weekend came up with the “idealized” version of what the data model ought to be. Kind of surprised everybody the next with the new thing. People weren’t surprised in a bad way, they were surprised like “Oh, you did that. We’d been waiting to do these things forever”, or make these changes, or change these class names. All the things you wanna do as a developer, like “Oh, this thing doesn’t really mean that anymore. We’ve overloaded the term, so let’s come up with a new term to represent this. Let’s come up with a new name for these processes”, whatever. You’re just refactoring the whole thing with reckless abandon.

What ended up happening - we now had two models. [laughs] Because clearly, we couldn’t just replace the old with the new that the developer had done over the weekend. It was unproven, it was all the things you’d expect. So we were like, “Hey, let’s change the tires on this 18-wheeler”, or 16 – I don’t know how many tires trucks have. But “Let’s change the tires. Let’s change some of the tires on this thing as it’s rolling down the highway”, right? We just tried to change too many tires all at once, and now we ended up having this monstrosity of some parts of the application referring to the new data models, and new class files, and new things, and then some of the old ones referring to the old ones.

[19:53] So now we onboarded a new developer… Worst-possible scenario, right? We bring in a new developer. Now they’re scratching their heads, wondering, “WTF happened here?” And we kind of didn’t have a good answer. It was like, “Well, the old stuff was kind of old and brown and really kind of – that’s not what the business was about anymore, and the new stuff is kind of an attempt at reimagining that, but we just haven’t had a chance to make the old stuff go away, so that we can only have the new stuff. Now we have both of them.” To me, that is the worst possible scenario. It’s like saying “Give me all the negatives on both sides, and let me have that.”

So every time we talk about greenfield, brownfield, I just think to that project. It’s okay to have brownfield, it’s okay to have legacy stuff, it’s okay to have cruft, it’s okay to have these things, but the naive thing, especially if you lack the benefit of time on your side as a professional, the naive thing to do is to think that you’re gonna come in and over a weekend just re-envision the whole thing, and just come as “Hey, folks… Look at what I came up with.” What, you think none of us had this idea? You think none of us wants to just go home and over the weekend just remodel the thing? What, you think we’ve just been lazy? No. This stuff is not as straightforward as just swapping a few class files.

So I think we need to learn – to me, if you’ve only ever worked on greenfield stuff without the benefit of having sat down and grown with the project and see it in operation, experience the pain of it, get woken up at night by a page… All of these things that give you that well-rounded perspective on these business systems that we’re writing. If you’ve only ever done greenfield stuff and you don’t get to see it through, and you move on to another greenfield stuff - you’re lacking, to me… Not through a fault of your own, but you’re lacking a certain aspect of understanding how – basically, that’s the stuff that takes you from coding to engineering. Engineering is where you add time to the mix, to understand how do you keep this thing running over the longevity, over the long-term.

Peer Richelsen: It’s a bootstrapped, organically-grown organization. And being bootstrap has pros, but it also has a ton of cons. For us as a VC-funded business, with employees who have equity in the company, millions of dollars in the company, who want to make this thing take off… It’s just a different sense of incentives. And incentives, at the end of the day, drive everything; the good decisions and the bad decisions. Culture is very important, and we try to keep our culture very developer and community-focused and humble.

And I think it’s also interesting the difference between not being bootstrapped, because it gives us, with the VC funding, just focus on building… I mean, that’s the classic playbook - aggressively building market share focused products, and not taking what’s… Like, we’ve turned down some enterprise deals, let’s say, where we would really need to stretch our codebase and focus on an entirely different set of problems. Even though it may make 200,000 a year or something, we turned it down, because we did not depend on it immediately to keep the lights on.

So I think there’s some unfair advantages and it would be not smart to play them to your strengths. So for us - sure, revenue is great, and it’s important, and a key KPI for us, but so is market share and developer adoption. So if we can grow faster and really build a great product for these people to use, we should be in good shape.