Justin Searls: But you know, we’ve got the next round of investors that we’ve got to impress. So that’s why we’re ordering all these foosball tables, and these massage therapists, and just like stocking it… And you may as well have had casting of just stock video people in the office, or something… The amount of money that was getting pumped into VCs, that was just silly money, that had no care at all what the ROI is. In fact, there was a long time there I would talk to founders and they’d be like “No, we’ve been not only not to chase profitability, but it’s seen as a red flag if you become profitable, because it means you’re insufficiently investing in growth.”

[01:11:54.21] Like, that was the environment that I sold in for the entire run that I was doing that role at Test Double. But the thing that I wanted to sell, which I had learned in the – I graduated college in 2007; I was fortunate to get a job, and then held onto that job for dear life during the 2008 financial collapse…

Hard knocks, right? And I had to learn a lot of these skills because everyone in 2008, 2009, 2010 only cared about “Show me the ROI show, prove your worth, prove your value.” And I got all those skills, and then around 2011, 2012, they stopped mattering… But that’s why I’m excited. That’s why I view this optimistically; the people who actually give a s**t, who really, really sweat the small stuff, who care about quality, and who care about just the credibility of stuff, like we’re talking about with Apple. They want to do a good job and they want to make great stuff. That is aligned with making money. That is super-aligned with demonstrating an ROI for shareholders. And now it’s just finally the time where it’s valuable again. And it’s been so frustrating to watch all of these winners and losers seemingly be so arbitrary, and just a popularity contest with the investment class. Now we’re closer to the ground and we can get real traction if we choose to.

Aiden Bai: Right. I remember I was in a Twitter Space with Theo a month ago, or something; or two months ago. And he was talking about Million. And the one thing I remember he’s been saying is “Million is a very difficult problem, with not as much ROI as it should have.” Just like projects like Bun. You’re rewriting the entire JavaScript runtime to make marginal – like, okay, in the grand scope of things, it’s just so much performance optimization. And so the reason we kind of labeled Million as an experiment that you can try is because it is. We’re taking a big, big bet on rewriting the virtual DOM, and making it compatible with React, and making an entire compiler for it… But I think it’s a bet that we should take.

Obviously, there’s a lot of technical constraints, like deterministic returns, or whatever. And our hope is the compiler can solve that for us, where we’re able to statically analyze our component, and make it work. It’s why Solid can work. Solid uses static analysis to optimize their JSX. But yeah, it’s definitely a huge challenge, and sometimes most applications aren’t apps, or web apps, and most applications aren’t data grids either. And so the ROI there isn’t super-high, unless it is super-high for you. But for us, it’s okay, because if it does help that whatever percentage of people, that’s worth it for us.

Kate Bradley Chernis: It depends… Because I’ve worked for a lot of nonprofits as well, including United Way Worldwide, and National Disability Institute, and the Walmart Foundation… So like I said, there’s only two objectives on social, and this doesn’t matter if you’re a nonprofit or for-profit, or government; it’s click or share. That’s it. Where you click to, or what you’re sharing - that depends. Ice Bucket Challenge - hello. Right? How human was that? They did a great job, but they raised money at the same time.

[34:08] Now, you and I both know that they have a grant to spend this money. And oftentimes, for a nonprofit, while sale isn’t the objective, a lot of times it’s just make some noise, to be honest with you, because the people giving them the grant want to see that visibility online a little more. But everyone has an objective, it’s just a matter of breaking it down.

When I was working with the Walmart project, it was fueled by the nonprofit foundation, and this was so boring. I mean, I had a great time, but it was – there was the IRS, Walmart, National Disability Institute, Bank of America, and AT&T, and United Way Worldwide, and they were working together… There was a free tax prep website that United Way Worldwide had built in tandem with Walmart. It was the first free tax prep online available. And we were trying to help lift the poor out of poverty through income tax credits and financial education. These people will maybe make $20,000 a year, so a $2,000 EITC credit is actually life-changing. Now, that’s the boring part here, and there’s acronyms galore… But we got the project with my method 130% ROI year-over-year for three years, and their ROI was taxes filed. That was their objective. So we still wanted people to click and share, but once they clicked, they’d start to file the taxes. So that was the whole… And by the way, one of the things we learned was how to take a national message and - this is what I was working on with them, or one of many things… To localize it. And the localization came through local hashtags. The cities, the college campuses, wherever we were, you know…

Nick Nisi: [laughs] Your reasoning is totally sound. My only pushback, I guess, on that - and it’s not a lot of pushback - would be yeah, it’s just yourself, but it’s also yourself from years past, that you have to remember all of these things… And it might help with that. But I also understand, the ROI isn’t very apparent going back and adding them after the fact. So I can see that.

Daniel Whitenack: Yeah, that’s awesome. And speaking of learning, you already mentioned the surveys that you’ve been doing… And there’s a previous survey that’s already published. I know you’re working on the next survey that will come out… But people should look at this survey - we’ll definitely link it in our show notes. There’s a lot of really interesting stuff in the survey, and some of the highlights… I’ll just call out a few of the highlights here. And then I’d be curious to know what stood out to you, Demetrios, or Chris, whichever one. So some of the highlights, just at a high level… Company use cases like text generation and summarization are useful, but participants are going deeper and exploring a lot of other ways to use LLMs, like data enrichment and data labeling augmentation, generation for subject matter experts, and other things…

[32:15] The use of large language models in organizations is still unclear due to some high costs, and an unclear ROI. They talk about hallucinations, the speed of inference with LLMs being potentially a blocker for certain types of use cases… You talk about the infrastructure, we already talked about that stack a little bit… Some of the things around augmentation and consistency of the models… So those are just some of the highlights from the survey, which people can go into a lot more detail on. But I guess one question I would have is, as you were building this report, first off, is the whole report generated using an LLM? Because I didn’t see that highlighted in the report.

Jerod Santo: I think so, but it’s hard to tell them that, you know…? It’s hard to make those decisions on other’s behalf. I don’t know their context… By the way, event-stream is used by Electron, and a few other very large projects, even though it was a seven year old library that the author no longer recommends using, so… I think everybody should be more aware, and I think events like these bring [unintelligible 00:15:45.18] decision-making to the consciousness of the developer community, and I think that probably will have positive ramifications.

[15:56] I know that in my life there’s been a bit of a process with regards to dependencies where I moved from very dependency-heavy, or always look for a solution online first, always find a third-party package first, mostly because I didn’t have the skills to produce what I was trying to… So in that case you’re just trying to get something done, so “I’m gonna pull in somebody else’s code because I actually don’t have the skills to write that code.” And then as I gain skills and experience over time, moving away from doing that, to being more critical with which dependencies I pull in, and really weighing the ROI of “Is this worth all of the baggage that it brings with it?”

So you kind of have these two extremes - you have the “Not invented here” syndrome, which basically says “I’m gonna write every single line of code, and if I didn’t invent it, I’m not gonna use it”, and we see where that can be problematic and bad… And then on the other hand you have what turns in dependency hell, which is like I’m basically just gluing together everybody else’s code to get a job done; I don’t have very much code of my own.

With most things, I think the prudent decisions are somewhere in the middle, but it’s difficult to tell other people that as well, as you’re just learning it for yourself.

Jerod Santo: It’s now time for Sponsored News!

Join the Open Source Pledge

David Cramer and the team at Sentry have launched Open Source Pledge to raise awareness and call for teams and orgs to support the open source they’re using.

We don’t think it’s the only solution, nor do we think it’s the only way to give back, but we do believe giving cash money to maintainers is an appropriate way to show your thanks, to recognize their hard work, the value they create for you. Maybe, just maybe, we’ll do our small part in encouraging the maintainers to keep putting up with us in the enormous ecosystem we rely on.

After reading David’s post, you may have two questions about this pledge:

1. What does it take to join the Pledge?
2. What’s the ROI your company gets from joining?

If so, head to opensourcepledge.com to learn all about it! And thanks again to Sentry for sponsoring Changelog News.

Abi Noda: Okay. The topic is, as the title implies, how to measure anything, and in particular, how to measure things that are seemingly unmeasurable. So when we talk about what is the dollar ROI, or interest rate, or cost of technical debt and poor developer experience - just a few minutes ago we were essentially calling that unmeasurable. In this book they talk about how anything – if you want to measure something in terms of dollars or cost, that you can really do that with anything as long as you take something intangible, like technical debt or developer experience, and then you correlate it to something objective, or something monetary.

So an example of this would be the DORA metrics and the DORA report, which I know you guys have followed. So what they essentially did is –

Emily Freeman: Yes. And we have learned so many things that I wouldn’t have even thought of prior to starting this business. But think about at a company – I’ll pick on the large companies who struggle with vendors. A cloud, to onboard a vendor to one of the major tech companies - six-month process. It takes forever. So to do that with 20 influencers is impossible. So for us, with the larger companies, that’s a huge selling point. It’s like, “You write one check, we handle the rest for you.” And then two - yeah, it takes an entire… This is a full-time gig. So unless you have a resource for a full-time role to actually scope out influencers, maintain those relationships, make the placements, look at scheduling, look at the type of content, you’re not gonna be able to do it, because we’re just experts. I mean, at this point I know so many of the influencers, where they play, what plays well on their different assets… That is something I could have only learned by doing this 100% of the time. And so yeah, it’s been phenomenal learning how this actually operates, and the best possible ways to get the biggest ROI for companies.

Joseph Jacks: Anthropic’s been beating Open AI now for two months on the benchmarks and performance. So Anthropic is like actually way ahead… But even Anthropic doesn’t have tens of billions in equivalent compute to scale. Like, Meta is literally doing this. They have hundreds of thousands of H100’s-equivalent from the last report that Soumith put out, who’s the PyTorch creator, and kind of leading a lot of this stuff. And they’re probably going to have half a million or a million more… And they’re literally just getting started with like scaling all this infrastructure.

So I take two sides to it. One is I think it’s justified and it makes a lot of sense, in the same way that releasing OpenCompute and React and PyTorch made a lot of sense, to deliver massive derivative benefits to their business, that were measurable. Really huge benefits. ROI - very easy justification.

On the side of LLaMA though, you have tens of billions of dollars now you’re spending. It’s not a few hundred million dollars or maybe a billion dollars. It’s like tens of billions. And so I still see some returns there, and you could see maybe a multiple of that if you look at the enterprise value of the business.

[01:22:09.15] But I don’t see the transformer approach they’re taking as the most efficient and effective way of actually continuing to innovate and scale the capabilities of the models. I think we have much better techniques. I mean, you have to really understand - what is the transformer doing? The transformer is taking in tens of trillions to pretty soon hundreds of trillions of what are called tokens; each token is a couple words, and split apart, and so a token is like a few bits of information, basically… Some tens of bits of information. And then you have hundreds of trillions of these things that you’re compressing and training into a system.

The human brain uses 10 to 30 watts of energy to compress what you could approximate as probably tens of exabytes of data by the time we’re wobbling around and able to like string together the first few coherent words. The amount of energy getting consumed by these transformer-based neural nets is gargantuan. We’re talking about tens of megawatts. It’s crazy.

The human brain is many, many, many, many orders of magnitude more energy-efficient than the state of the art neural nets that are getting produced today. And I do think the human brain resembles something probably roughly approximating a biological neural network of some kind, with electrical signals between our synapses, and all that kind of stuff. I think that’s kind of like the Red architecture. But to do that in silicon, at our current approaches is as if we were doing vacuum tubes in the ’20s and ’30s. That is our current state of the art sophistication in building neural nets. We are in vacuum tube days.

Russ Cox: Sure. I’m actually really excited for Go 1.23. I’ve been getting to write a whole lot of code recently, and getting to use the new iterators and new range loops is just really amazing. I mean, being able to write custom iterators that fit into range statements - it’s really kind of transformed the way I write a lot of my code. And being able to abstract things like scans over databases, and those sorts of things is really, really helpful. So I can’t wait for all that to be live, so that other people can write those sort of things, too.

In the sort of longer term, something that I’m sort of wondering about - I think that’s the right framing - is just the sort of overall sustainability of open source. Because we saw with things like the xz attack that – we’ve known for years that critical things are just dramatically underfunded, and it creates this really easy opportunity for bad actors to come in, and really just sort of do arbitrary damage at these low levels.

I asked a roomful of senior-ish people from a variety of companies how much money they thought the xz attack cost. And the general consensus was “Maybe $1 or $2 million.” And maybe $1 or $2 million to get an SSH shell on every machine in the internet almost - it’s pretty good. That’s pretty good ROI. So we need to figure out ways to make those attacks way harder. We can’t keep doing what we’re doing in open source. So that’s the sort of interesting wonder that we have to figure out as an industry in the next decade.

Gergely Orosz: Well, the interesting thing is that the news has been dominated by layoffs, because they are a big deal whenever they happen, and now they’ve just been happening, initially a few, and then some more… And now every week there’s so many companies, small or sometimes large, doing it. But the companies that are not really affected, they’re kind of hiring as always, and in fact, some are hiring more, are the ones that are honestly just not - they were never the most desirable companies to work at, but I think this might change just a little bit.

I’ll give you an example - there’s a major bank in the Netherlands, which saw huge attrition at the end of 2021, because of this crazy market; maybe also because of my articles, I’m not sure… Because a lot of people from that company went and they took offers at the likes of Twitter, or big tech, or venture-funded startups… They just got a big pay increase, and this bank couldn’t move up with the market, because that’s not how they’re structured. They cannot respond quickly, and they typically don’t. So they lost a lot of people. And they got us contractors, and they have a huge headcount to fill for 2022. I talked with a hiring manager there, and they’re like “This is great. Finally, we might actually be able to hit our headcount, but probably we won’t. We’ll probably still have empty headcount.”

[06:10] So the good news is that there are a lot of companies hiring. And I’m also seeing – I’m running a job board where companies can pay to access either candidates or post jobs, and I vet these jobs, but I’m also seeing growth there with a lot of venture-funded stuff. So there’s a lot of venture funding, especially at early stage, going on. And those companies - they want to hire they; they need to hire good people.

And then there are the companies that are a little bit more conservative. So a good example is iGen. Not many people know about iGen, but if I say Stripe, you’ll know what I’m talking about. iGen is the B2B version of Stripe. They’re based in Amsterdam; actually, they’re headquartered there. They’re a publicly-traded company, and they were trading at about like $50 billion market cap. I’m not sure where they are right now; maybe like 40, or something like that. So they’re big, and they process about as much as Stripe in terms of merchants. At Uber, for example. Partially, we used iGen, and I think they power eBay, and Booking.com, and a lot of the sites that you just don’t see behind the scenes… And Stripe announced that they were doing 14% layoffs. Meanwhile, iGen, because they’re a publicly-traded company, you can see their financials - they’re just passively profitable. For every dollar, they make 60 cents of EBITDA profits; that’s not fully profit, but it’s a very strong signal… This is actually more profitable than, say, Google even, in terms of the percentage amount. And they just announced that they just sent out a memo to their staff that they’re – not that they’re not laying off, they’re actually investing, because their fundamentals are strong.

And the headcount is also very different. This company, iGen, even though the price is about the same as Stripe, and they have a bit lower cut rate, so their ROI will be a bit lower, but they have a lot less staff; they have half the staff as Stripe, and at lower cost locations.

So all I’m saying is, these are some of the companies that I know about who are kind of just doing fine, and there will be a lot of these… But we should acknowledge that it’s not the top of the market. So the companies that used to pay the best total compensation packages in terms of high base salary, great equity, all that - those companies are now struggling.

And then they are still hiring. I just talked with someone who got an offer from Google in London, and it’s a really good package… So there’s all that going on, but I think the noise of all this bad news is really on top of people’s minds.

And then one last thing - if you look at what is actually happening in the market in terms of numbers, it is a correction, but when we look at – let’s say there’s big news that Meta laid off 13% of staff, or 11,000 people; it’s a huge number, 11,000 people. But when we look back at how quickly they hired those people, just this year they hired 20,000 people. They’re 87,000 people right now. So by just laying off, they’re kind of back where they were in March… Which is bad, right? But it just shows how much all companies actually hired. I rounded up between Meta, Microsoft and Google, I looked at their public filings of how many employees they have now and a year ago, and they went from having 400,000 employees, these three companies, full-time employees, 12 months ago, to having 500,000. So they added 100,000 in just one year. And Microsoft is 20-year-old company. Or, sorry, older; like 30-40 year-old. So in the last year, their growth was incredible.

So we are seeing a cutback, but it goes up that crazy - like, that last year when we talked about it. So it seems that it’s a correction, and it is painful, and it’s not happy to see them… And honestly, I think the reason people are just really shocked, most people, and said, “We haven’t seen this”, this whole market reminds me of what we’ve seen in what finance folks have seen in 2007-2008. It’s kind of our version of like a massive market correction in tech. And outside of tech, things are kind of going fine. You talk with other industries and they’re like “Yeah, it’s just like normal. We’re not seeing any panic.”

Erik St. Martin: Yeah, so it’s interesting… We’ve talked about this before on (I think) Go Time and Changelog as well… It came around as a really interesting story. At the time, Brian Ketelsen and I worked together at a company; this is 2011 or so… And we were both using Go, and we were attending lots of conferences… And so early 2013, we kept talking about the fact that like “I really wish there was a Go conference. I really, really want to go to a conference and meet other people who develop Go.” We both live in Tampa, really big in the tech industry for healthcare, which is not, a bleeding edge technology adopter… So it was basically just Brian and I covering all of Tampa at that time with Go. It was still very, very early. And so mid-2013 rolls around, like April-May or so, and some folks in Denver, from the Denver Go meetup group were talking about “I really wish there was a Go conference.” And Brian and I were “I know, this is what we’ve been saying.” And so it was basically “Why don’t you guys do it?” And we’re “Well, how hard could it be? We should. We should totally do it.” It turns out after the fact - really hard. It’s actually a lot that goes into it. We knew it’d be a ton of work, we totally underestimated just how much work it would take. But yeah, so that’s sort of how things came about.

It was really interesting, because we were hoping to just get, 100-200 people together. At the time, we didn’t know how many gophers there were; the language was still super-new… And so we planned to hopefully get like two or three hundred people together, and then it just sort of exploded from there. We had to keep calling the venue back to get more room. We maxed out at 750 people the first year… So that was really crazy for us, because - I mean, anybody in the conference circuit kind of knows, like… I mean, most conferences take years to even hit those numbers. So we were just really, really in shock.

[08:15] It was really a humbling experience though too, like the number of sponsors who came on… And same thing, they just wanted a conference for this technology too, and so many of them weren’t even worried about ROI at the time. They’re just “The ROI is a Go conference starting, right? We need to hire people, we need these events to exist.” And so that was really awesome.

Then like all of these people that we really admired in the Go community were there helping us stuff bags, swag bags… It turns out stuffing 750 swag bags takes a long time. Who knew…? So yeah, it was just really interesting to have – I’m trying to even remember… Steve Francia, Bill Kennedy, Damian Gryski, Dave Cheney… There were so many. I’m gonna leave people out here, which I always hate doing those things… But trying to remember all of the people who were like back there - we’d sit here forever. But it was just kind of crazy, all of these people who were like “No, no, you should be giving talks. Why are you helping the stuff bags?” But yeah, I think everybody was just really excited to just see this thing that happen, and anything they could do to make it an event that would be remembered, they were willing to do.

And the same is true about MC-ing, right? At the time, Brian and I were running around just putting out fires, because there’s so much stuff that happens behind the scenes that people don’t even know, and like things that you need to manage while you’re on site… And so we’d be like putting out a fire and then running up on stage to introduce the next speaker. And Kelsey Hightower came over and was like “It looks like you guys are kind of a little busy here… Do you want me to take over introducing the speakers for you, so that you guys can handle everything?” And so he did, and that’s just like another example of people in the community just trying to pitch in to make it something.

I think that all of that probably sparked the atmosphere that exists there today, where we care more about the community than anything else, and everybody just sort of contributing to it being a great time.

Lukas Egger: Only with my voice, yeah. But we have like four categories in terms of phases, and four categories in terms of risks. It’s almost like a matrix, right? We created this matrix and we call it Definition of Done matrix. And then what we do is we go to the owner of a product, whoever has the need, who has a problem that we are trying to solve. And the first thing is we’re not taking it away. We’re not saying, “Oh, we are here to solve your problem, but stay aside, we’re doing innovation now”, right? No, we’re saying, “Hey, we’re in this together. So this is what you want to achieve. We understand the pain point.” Now we will assess in terms of the risk, let’s say desirability, feasibility, viability, or stakeholder management, how far your idea has already gotten, right? And think of it as a scorecard. Literally, think of it like a four by four scorecard, and you start ticking off how far you got, when the beginning is like, we understand what we’re talking about, and the end is pushing it into production and making it part of the product.

And then what we’re doing is we’re saying, “You as the owner, you tell us where you want to de-risk next. And we often make this conversation a broader conversation, with more stakeholders, or the senior leadership of the organization. We say, “Look, if we invest in feasibility, the next thing in terms of an outcome we would create is this.” Or let’s say viability. And we then ask the stakeholders and the owners, like “We are just helping. Do you want us to try to de-risk viability, or do you want us to try and de-risk desirability?” And we then upfront tell them, “Next thing we would do is – let’s think about it. Let’s say a viability in ROI calculation, or define a first user study” or these kind of things. So it’s almost like an à la carte menu of de-risking activities based on the scorecard, and we say, “Hey, this is how far we believe you are. And you tell us where you need help in de-risking this project.” And that’s how we do it.

And then every time we get feedback from our customers, or we do our analysis, we come back, we add or subtract from the scorecard, we say, “This is the new state of the project. Now, what is the next thing we want to do?” Why are we doing this? A, because we want everyone to have skin in the game and include them. B, we want to be methodical and very clear how we work. And C, we don’t want to be away for months and take something away and then say, “Ha! We have solved it”, but rather we want to have really quick iteration cycles where everyone is excited that we de-risk it… Or if we find out there’s a big impediment that makes it unfeasible or impossible, that we communicate it not at the end and everyone is, “Oh, what happened?”, but they’re part of the journey. So we are making them part of the journey, we’re very clear in what phase and what risk we’re working on, and we’re doing it together and not isolated from the rest of the team.

Nora Jones: Yeah. A lot of orgs index on the quantitative, like you said, because it’s easier to see, it’s easier to measure. And so I think, especially as orgs start to feel they’re having a lot of incidents, they start counting things. They start counting how many incidents they’re having. They start counting what days they’re having them. They start counting how quickly they resolved them… All those things, because it gives them a sense of control. But really, I think, especially in times of desperation, orgs can over-index on them, and it can actually be pretty detrimental to their learning. And so when you over-index on them, you are kind of tanking the learning. And so those numbers are not going to get any better and people are just going to get more upset over them, and you’re going to end up creating this culture that you were trying to actually improve.

And so one of my favorite equations is performance improvement is a combination of error reduction plus insight generation, and it’s from cognitive psychologist Gary Klein. He studies organizations, like what makes them successful; he studies how experts think. And he put it in this book called Seeing What Others Don’t, and it’s about a lot of what we’ve spoken about today. But performance improvement can’t happen with just one of those sides of the equation. It can’t just happen with error reduction. It can’t just happen with insight generation. It needs to be a combination of the two, right? It needs to be a combination of the quantitative and the qualitative. I mean, it goes back to KPIs and OKRs. Yeah, it’s great to have those numbers, but if you don’t have a story behind them, if you don’t have context behind them, they’re meaningless and harmful. It’s the same thing with error accounts and such.

So you do have to get creative with how to quantify some of this learning. Maybe you like, “Hey, how many people are reading the report? How many people are voluntarily showing up to this incident review meeting? How many people are talking in this incident review meeting? Are we getting people from sales? Are we getting people from marketing to come, or is it still just engineers? There are little things you can do to track that improvement journey, too. You can track if people are interested, or if they’re giving you feedback. There’s a lot of things you can track, but I’d really index on the collaboration and the learning opportunities, or if people are having those moments where they’re like, “Oh, I didn’t know it worked that way.” That is like the best ROI you can get from one of those meetings. So there are ways to quantify it and track if you’re doing it well, it’s just not as clear-cut.

Adam Stacoviak: Yeah, it does… Because you can say “Well, that’s $40,000? Okay, we can ROI this this way, we can eke it out that way… Okay, let’s do it” kind of thing. What’s the big idea with Fly though? I know we talked a lot about your journey, and in and around some of the things; we’ve teased out to some degree what you’re doing with Fly, but what’s the big idea with Fly?

John Nunemaker: Yeah, that was pretty cool. That was actually kind of like a – I won’t say accident, too… Because I was kind of maintaining it, because everybody else had kind of moved on, and so I just kept it running, and occasionally would add things to it, and stuff; honestly, in my free time. And I remember we started getting a couple support requests here and there, people wanting to buy it… They were like, “Hey, is it for sale?” And I remember asking, and everybody’s like “I don’t know if it’s for sale. We’d have to vet the buyer.” And that kept coming back. So finally, I was like, “Alright, this isn’t gonna happen. It’s just dying right now. So if we don’t do something with it, it is gonna permanently die”, and I was like – I just like it, again, because looking back, it’s the theme or the thread that ties a bunch of that story together. So I was like, “You know what - I’ll vet the buyer. Are you guys okay if I vet the buyer?” And everyone was like “Yeah, that sounds fine.” “Everyone” being my manager, and other people, levels up. So I was like, “Alright, I’ll vet buyers.”

And then as I was thinking about that over the weekend, I remember I was like, “Why don’t I just buy it? I’m already maintaining it basically in my free time. I wouldn’t be any different, except now I would have ROI if something good happens to it. I’d have ROI.” That, again, comes back to impact. So I think on that front, that’s kind of where – so I was like, “You know what - let’s try it.” Again, I don’t think I’ve thought about it until like we’re talking about it now, but… Common thread - you never know it until you ask. So worst-case scenario, when you negotiate, they say no.

[51:57] I remember I think I offered – I don’t even remember. At that point it was losing like $1,000/month or more, just from hosting, and AWS bills, and stuff, and I was like, “Alright, I’ll give you $5,000.” So basically, this shows that I’m real, I’m genuinely offering for it, and I will take on the burden of $1,000/month, vetting the buyer, all the problems that could possibly arise from selling it, and any possible negative impact of closing it down, or any of that kind of stuff.

I sent that up the tree, and at some point they were like, “Yes.” They dropped the price to like – I don’t know if I’m allowed to talk about any of that kind of stuff, but it was like an order of magnitude lower. They were just like –

Justin Searls: Yes, well - I mean, unfortunately, it is all too common of a situation, and I think that it is challenging to write tests that are not susceptible to several specific things that contribute to what is commonly called brittleness or flakiness, right? The most important thing to understand as we’re approaching this question from the outside in is what is our goal to have this built in the first place? And the goal is probably to have some sort of confidence that things are working. And if we get one green build out of five and no source code changes in the process – like, my confidence is not high, right? And we know this based on kind of intuitive experience that if it passes one time out of five, it means that you have proven the system can work. And you’ve probably also proven not that the system has some sort of fundamental flaw and will break in production, so much as you figured out that there are environmental timing or ordering implications that can cause one or more of your tests to not work.

And I think the first thing that a team should consider when they are running into this problem is to get back to consistent green builds as fast as possible. Because again, if you’re thinking about testing as ROI, if all nine of those people are getting an email every single time that the build breaks, and then say three of those people just independently start racing to go and screw around with timeouts and stuff - that’s a lot of dollars flying out of a window, because of one little tiny thing that might not be well understood. And then no one was no one woke up that day and was like “I’m going to really wrangle all of the flaky stuff in our test suite”, right? They’re all trying to do something else, and this is the thing blocking them from that thing. Not to say they’re half-hearted in their attempt to fix it, but what they’re really needing is a salve in that moment, as opposed to a solution.

So the first thing that I would do is I would lockdown everything. Normally, I don’t like freezing time in the system, right? But I’d probably start with the common quick fixes that I can apply, like “Hey everyone, it is now 2019, August 3rd, it’s a Tuesday, and it’s 11:33 PM, and we’re just going to lock that whole server down that way”, whether we’re using a test tool to do that or Unix. “Hey, everyone, we’re also going to no longer randomize the test order, we’re going to use this particular seed that is known and good, that we’ve seen work.” “Hey, everyone, we’re going to change all of the directory globs that are currently in an unspecified order in Linux, and that’s why Linux builds are failing, but on our Mac where it’s the alpha order, those are all passing.” Like, “Hey, we’re just going to do a sort on all of those glob requests everywhere, so that we’re just loading alphabetically, because it doesn’t really matter…” And we’re going to like go through the half dozen or so quick hit things to just try to get to consistent builds as soon as possible. And hopefully, that’s one person one day, if ever.

Tanner Linsley: Well, I think it was natural, looking at other tools, to do it. Redux has dev tools, and Apollo has dev tools… A lot of things have dev tools, and I also saw – Kent likes to talk about shipping your own dev tools a lot as well. It’s just kind of been around. I knew that I wanted dev tools from the very beginning, because I wanted to be able to inspect this – the lifecycle of react-query can be a mystery sometimes, unless you have the dev tools there to show you what’s going on.

So I knew I wanted them from the very beginning, I just didn’t know how I wanted to implement them. I looked into building an actual Chrome extension, and I can tell you that that was very quickly not going to work for me, mostly because of the time that I was going to have to invest. It was not worth the ROI.

Chrome extensions are a bunch of domain knowledge that I don’t really want to have and I don’t really care about right now; maybe in the future, but I don’t want right now. So I’m like, “I’m just gonna use the tools that I have and that I know work for me. I’m just gonna build a component. It’s a React component, you can put it anywhere you want, and it’s the dev tools. And I’m gonna ship it with a bunch of in-line styles.” I even made a little styled-component type thing that just uses inline styles, and a theme, and everything. It just kind of ships an all-in-one inclusive component. And it worked out great; you can render it wherever you want, or you can just use the React dev tools main one and it just renders at the bottom of your screen type of thing… And it works great.

There are some drawbacks… People are like “Oh, I wish it wasn’t taking up part of my screen. I wish I could have it in an external window.” Yeah, sorry… If that’s really that much of a pain, you can go build the dev tools extension, because I’m not doing that.

Paul Bakaus: [40:11] Yeah. One thing I just wanted to call out is that there’s two separate goals here. One is to really make the creator experience on the web better, to make sure that we can onboard creators to the web for many years to come, and that they’re successful, and that there’s enough ROI - to use a businessy term - for them to stay on the web and to find it valuable… But then the other goal - and this is also actually really why we think Web Stories matters - is to make sure that the user experience of the web is not as frustrating as it is today. And it is frustrating. I think we all notice. But the part of the web that is the content web - you might be landing on one really good blog post, and you click on another blog post, and then have like a newsletter capture, and you have GDPR dialogues, whatever… There’s massive plasters of ads… There’s so much that has been going wrong over the last couple of years. That’s something we’re trying to control with that Web Stories experience on the consumer side.

Nhung Ho: [43:56] Oh, man… [laughter] The operational component of the work that we do cannot be understated… Because it’s not just about building the model, it’s actually taking it to production, dealing with the ETL, grabbing the data, serving the customers… So you need to really make sure that you have fairly robust monitoring in, so that you can be proactively alerted if something goes wrong. Because the worst-case scenario would be a customer contacting you and saying “Hey, you totally messed up my cashflow. Now I can’t pay a vendor this month.” That would be disastrous. So building and monitoring… But you can’t feasibly monitor for every event, for everything… And so doing a sampling and monitoring along the way.

I will also just put in a plug, which is that when you’re building - for us, we have three million customers in the U.S. alone for QuickBooks, and then multiple transaction streams, we’ve got, again, all that data… And we quickly get into the realm of over ten million models being built and served. Start in batch mode. There’s this obsession with building models in real time for everything. Every single team I talk to, they’re like “When am I gonna get the model in real time?” Well, let’s start with a batch. The complexity for a batch model is just much easier to handle. You validate your hypothesis, you validate the use case, and then you can say “Is the ROI enough if we move it to real-time?” That’s one area.

There’s also – the thing that we’ve made an investment in that has really helped is investing in a robust machine learning platform that allows us to do distributed training and distributed scoring… So marrying the best of, essentially, the MapReduce framework. Doing that distributed computing with our models.

The cashflow is not the only use case where we have individual models for users. For transaction categorization, which is where any transaction comes in, we figure out which accounting books it goes into. We have individual models serving in real-time for customer in that case as well. But you also have to figure out when is the scenario where you actually wanna get to that level of complexity.

So for us, we always start with one base vanilla model for everyone to start with, and then see how far we get, and then we figure out “Okay, how do we blow this up in times of scale and complexity?”, and how much headroom is there in order to solve the problem well.

Nikola Mrkšić: Okay, it’s a really good question. I say it’s hard because it’s enticing, it’s fun, it’s a big problem, I expect to spend the rest of my life solving it, and to be maybe a small cog in the wheel of how that ends up being solved. It’s a hard problem on a pure academic level, because it’s that compound movement of different NLP tasks that all need to work really well, and they need to communicate with each other, which is something that we’ve not really yet cracked.

The thing at the center of the dialogue system, language understanding, is not a tax that’s fully well-defined just yet. Think of speech recognition - in most languages, you say something and there’s exactly one way of writing it down. But natural language understanding - what does it mean? How do you choose to interpret it? What are the things you’re choosing to take away from, even like an order, or something as simple as that? So there a lot is left to the interpretation, and that means that it stops really being a science, or even a field of engineering where you have a clear metric to beat… Because really, what we’ve shown over the past ten years, especially with machine learning, is that you give something a clear evaluation, and the sheer force and the intellectual power of the people working on it will crush it.

If you think about question answering, the SQuAD dataset.

[12:10] I remember the leaderboard on the Stanford website, I believe, and - you know, at first I think the scores were pretty low. Then all of a sudden, like 3-6 months in, we got to the point where performance was unbelievable. I couldn’t believe that it’s that good. And it’s because when you define a clear scope for a problem, we’ll build a machinery to solve it.

Now, when it comes to building these voice assistants, we don’t know what machinery to build. The truth is we’ve built a lot. [unintelligible 00:12:36.27] I like to compare these assistants to aircraft carriers. When you think about Alexa - 14,000 people building that thing. And if you think about the ROI, they’re not building it because they’re making a ton of money on it. They’re investing in the future. I mean, Amazon always has a math around it, and I hope they’re right… In any case, it’s really good for us, because they’ve actually indirectly funded a big growth in the area. They’ve allowed us to, in turn, build a lot of stuff ourselves. But we were doing it before they got interested.

The problem itself is hard. It’s a good word to use… Because you have to solve a lot of these different problems, you have to solve things that are not just in the domain of machine learning, but also human-computer interaction.

The voice user experience is something that academics tend to overlook. They don’t appreciate that sometimes just the tone or something like that is much more likely to prolong the conversation and imbue the caller with enough patience and goodwill to go through a conversation. Equally, people who are very good at user experience don’t tend to be mavericks at machine learning… And then kind of bringing it all together to build something - it takes a lot of different personas, people… It is hard. So I guess that’s what I mean.

Erik St. Martin: Yeah. And I think it’s also important to do a couple of things… So the other thing we agreed upon to help with this is that all speaker costs would be paid for. I think that’s important, because if companies are paying for you to travel, you only get people submitting who work for companies that will send them, or have the money to send themselves… So that reduces it. Plus, you also get a lot of companies – this has gotten better in recent years, but it user to be really bad. Companies want ROI, so they send you to speak at a conference, and… Yeah, I think it opens it up to a broader community, I think it allows them to control their own messaging and not be beholden to ROI on the company that sent them, and all of those things.

Justin Searls: [laughs] Well, we named the company Test Double as sort of a joke. If you’re not familiar, Test Double is just a bit of jargon that a fella named Gerard Meszaros coined when he was writing a book called xUnit Patterns about unit testing… And it just means like a fake thing that you use to stand in for a real thing in your tests. I think he was probably referring to a stunt double… And it’s anecdotes like those that you get with my level of experience, but I’m not sure that high-quality ability to write tests is necessarily that much better… But honestly, these days when I think about testing, my primary focus is on making sure that people are well-situated to get a good return on their investment.

It’s still the case that a lot of systems have no tests, it’s still the case that the people who do have tests are sort of just doing it because a testing sounds good, or because somebody mandated that everything be tested. But beyond that, there’s a tremendous value in asking the question of like “What do we want out of our tests? How much are we willing to invest? Will we know we’re getting a good ROI out the other end? What can we be doing to set ourselves up to be even able to answer that question.”

So thinking about the individual purpose of each test, and kind of talking through that and designing specific tasks for particular uses, as opposed to just sort of like a boolean state of “is tested/is not tested”, those are the sorts of conversations that I find really fun and engaging when I’m working with a new team.

Ahmad Nassri: I wanna also reference some history I’ve had in these kinds of conversations in the enterprise… Because in the enterprise, again, there’s zero-risk policies in the tolerance for it. It even kind of makes its way to somewhat logical things that we should replace and we should extend, especially when it comes to cloud providers. I’ve heard this a lot in the enterprises, like “Oh, we can’t trust to put our data on GCP, or Google, or Amazon, or whatever. What if they go down?” What if your data center goes down, how is that any different?

And I hate to use the slippery slope example, but those conversations just keep on coming up again and again. It’s like “Well, I don’t wanna use a vendor, I don’t wanna use a third-party. I don’t wanna use this”, and it’s a rational way of thinking, for sure; I think it’s about trade-off conversations and ROI and total cost of ownership at the end of the day… Because the cloud vendors really want your business; they’re subsidizing half their stuff.

Trust me, we’re not actually paying half the cost of what we should actually be paying for any of these SaaS or cloud services, and that’s working out for our industry very well right now. We should be paying thousands of dollars for GitHub right now, but Microsoft is subsidizing a lot of it. Great. I’m gonna use it and I’m gonna take advantage of it. They’re definitely taking advantage as well, because they’re analyzing every single line of code that you write, but that’s beside the point. My focus is on the business and making the business work.

Same thing with any of those choices that you have to make with vendors, with tools, with third-parties. Are you willing to take the compromise and are you willing to spend the cost lower now, but end up accelerating your team and you can focus on other things that accelerate the team? And then later on, when you’re successful and you have the time and the energy, maybe go back and reimplement those things the way you wanna implement them, but perhaps considering the total cost of ownership you’re actually making it sustainable and making it longer-lasting, and actually investing in a budget that can carry as a legacy well beyond your team and yoruself being there.

Jon Evans: I think we have captured most of the things. I mean, I wanna stress just how important and how useful our partners have been - the Internet Archive, Software Heritage, Stanford, the Bodleian etc. I’m sure I’m leaving someone off now who really shouldn’t be left off. This is inevitably the way when you try and list people… But I think it was really important that we cast a broad net, trying to work with as many of these organizations… The Long Now. The Long Now have been great. Having a conversation with anyone at The Long Now is always a mind-opening experience, even if it’s a relatively simple one. And Silica, another partner…

That’s a long list, but I think it was important to treat this not as a thing that one company is doing for one company, but that a broad consortium are doing, and hopefully has general goodwill. I mean, this is not a project which has an ROI. This is a project which we think is actually important, or could be important. It’s sort of a weird project, in that you hope it’s not really that important. In a perfect world, all this data will be saved anyway, and you just sort of grab it off the internet, and no one will care about it.