Drew Wilson: Yeah. So what happened to me was I had sold my company, Plasso, to GoDaddy. And it was literally a dream scenario. An all cash deal, with the option to jump on as an employee, and get a good paycheck and additional payouts if I stay on. So I stayed on there for two years. But with that initial cash sale, I paid back my investors plus some, because they made money on the transaction… And then paid some of my employees that had equity. I made sure everyone who worked with me had equity, so folks made money. And then I had money. And I had never had money before. As a matter of fact, in order to get Plasso to where it was, I had to take out a bunch of loans on the same day, so they would not know about each other, and they wouldn’t show up on the FICO score thing, the credit report. So I took out a bunch of loans on the same day just to keep the company going.

[00:38:24.08] I was making enough for myself, but that’s it. And I needed these other employees to help sell the thing. And so I took on massive debt. The amount of debt that would take me my entire lifetime to get out of if things didn’t work out. And so yeah, I took on that debt… But then, just as I had to fire my last employee, being like “Sorry, I don’t have any money”, literally, that was on a Friday, that was their last day… I came into the office on a Monday, and there’s an email in my inbox from GoDaddy saying “Hey, we’d love to talk about partnering.” And acquisitions always happen under the guise of partnerships. I didn’t know that. And then three months later, I had sold my company. A little over three months. But to me it was like 10 years of my life. It was just so exciting, and at the same time – it’s like a rollercoaster of emotions, because things are going your way, not going your way, is it going to happen, it’s not going to happen… It’s going to happen… Oh, man.

And I didn’t have anybody else. It was just me. I was the only person doing negotiations. There was no advice giver on my end. No lawyers, nothing. It was just me doing it all. And so it was a great experience. It was a great experience. The company acquisition team at GoDaddy is super-nice.

So it was cool, got through it all, made some money… Then I was like “Well, I’ve never had money before. What the crap do I do with money?” And I’m like “There’s got to be a site somewhere –”

Autumn Nash: That’s like the Wells Fargo algorithm. Were they trying to be harmful? No. But - what’s the saying with the causation? You can have data and it could be honest data, but why is the data that way? So it got to a point where it wasn’t lending money – the algorithm was not recommending people of color and women, and people of different backgrounds to be recommended for loans, for mortgages. And it was because people of color were at average making less when the data was trained. So they were making less and they had more credit issues or different things. But it was judging an entire group of people off of that data. Was that data taken in correctly? Maybe. But why is that bias there? Is it because more people of color were facing other socioeconomic issues?

John Nunemaker: I think I would do it. I just would have to learn more about how people do that. And even in that, again, that podcast episode about like buying five – I can share the link later, or whatever… But it’s like buying five manufacturing companies; in that scenario, that guy was like – they do kind of like a PE model, where basically they’re like “Look, we come in and we run it”, and there’s six partners in the group, and each of those partners will personally guarantee one business. And so right now they have three, guaranteed. And when they do that, they do two things. So one, they charge a management fee to the business based on EBITDA. So they buy businesses that already have good profit margins, they take a slice of that profit margin for everyone who’s in the management group, and then they’ll also do - whatever, first 8% preferred return, in the event of a sale, goes to the investor, and then after that it’s like split based on equity and all those kinds of things. And they bring in basically different people in every single deal.

So I wouldn’t say different people from the standpoint of like “Well, Adam’s in this deal, and Garrett’s in this deal”, or stuff like that. It’s more like “We’ve got five people each putting in like 100,000 or 50,000, so now we have 250k, 500k, and then we’re going to personally guarantee, do an SBA loan for the other half…”

[01:26:13.23] So their debt to whatever ratio is basically like half. Because again, if the business went all the way in half, you’d still be able to run it, which is pretty rare for a business that’s been around for – they stick to 30, 60, 80 years. 30 plus employees, so they know that all the knowledge is not with one owner in a 10-person company, or things like that. So again, a different scenario here, where we have software and leverage and stuff, and it’s less of less of a concern. But I’m like, that idea is really fascinating to me.

The group, the six, each have a share in each of the companies they acquire. One of them runs each of the companies they acquire, one of them personally guarantees… Not necessarily the one they’re running. And then they also kind of – for lack of a better word, they shard their investors across all the deals as well, so no investor is heavily invested in one of them. And I’m like “I actually really liked that model.” I think that’s sick. Because now they’re doing the SBA loans, which means they can personally guarantee, but not actually put up the collateral, so you don’t run out of collateral and run out of money… But I don’t know all that financial stuff yet. I haven’t done any of that, so I’m like, I’m just going to start simple. And I’m like “Look, I’ve got some friends, they had a little bit of cash too, I’ve got the collateral, let’s put it together, make it look a three to five-year payoff.” If it’s an eight-year payoff and we take more cash out, that’s fine. It’s just, ideally, I don’t want to have my house on the hook forever. Or maybe I pay off the loan at some point… Like, let’s say Box Out does sell, for some reason. We’re not planning on it now, we’re not talking to anybody, we don’t have any interest in that… But if somebody came along with the right price, we’d be like “Sure.” That happens, now I’ve got some cashflow, so maybe I pay off the loan, get the lien off my house, and then I loan it back to the business at a lower interest rate, or something like that, so that the lien is basically gone from my house, but there’s still some amount of leverage of using other people’s money to make money, which is kind of an interesting concept, I guess. Again, I’m all a rookie. I don’t know any of this stuff. I’m just flapping the stuff that I’ve been researching for a couple of years, and then heavily for a month or so… So I don’t know, but that’s just what I think in my head.

Autumn Nash: The price of cars now is just wild. But I think that all this is gonna change the way that we have finances, too. Back in the day when I started art school you would buy a version of Adobe Photoshop, usually with like student loans or whatever, your first money, and then you’d use it until you had a feature you needed now, another one. Or your computer completely died, and you had to get a new computer, so you needed a new license. But now you have to pay Adobe Photoshop every month. And artists already don’t make a lot of money… You know what I mean? And then now, it’s just wild. We’re switching how people pay and own –

Jerod Santo: I do think that the lockdowns began, and hiring probably froze for a while, while nobody knew what was going to happen next… And so it makes sense that the postings dropped - yeah, like you said, from 100 down into the 60s, in the matter of a few months. And then just climbed, climbed, climbed, probably as stimulus money was injected, we had the PPP loans… The markets were going crazy, valuations were high, money was easy… We were still at ZIRP, zero interest rate policy… And it just climbed, to where it was just sunshines and rainbows, up until mid ‘22, and then it just drops from there. And of course, we know what happened in our industry, in the tech world; everything tightened up, money got more expensive, layoffs began, hiring freezes began… And it seems like we’re still on our way on the downslope of that. And where does the downslope stop? Who knows…?

[00:23:53.03] What’s explaining this? Some people are saying this is AI already hitting us where it hurts. I think that’s premature. I don’t think anyone’s losing their job because of AI. Maybe on the margins, in the tech world. Obviously, in art and other places there are people who aren’t as valuable as they were, because their job has been somewhat offset by language models and image models etc. But I don’t think any software developers, unless their bosses are pointy-haired bosses who don’t understand what’s going on, are like flat-out not getting hired because AI is so productive right now. I just don’t think that that’s the case.

Saron Yitbarek: It’s really pricey. And back then there were no payment plans, there were no loans… You had the money or you didn’t. And even I didn’t have the money; I had to borrow some of the money from my mom to help me pay for that program. And so I felt like if you didn’t have a bootcamp, it was really hard to find that support system that to me was so crucial to me learning how to code. And I wanted to create a community for people who were learning, and so I started doing these Twitter chats; this was back when Twitter chats were all the rage, everyone had one… And you’d pick the hashtag, or hashtag was codenewbie, and I would tweet out questions every Wednesday night at 9pm Eastern Time, and I would ask “What are you excited about? What are you learning? What are you stuck on?” and people would respond, and kind of talk to me, and really talk with each other and get to know each other. And there’d be a lot of just this love in these Twitter chats, and this support, and just really good vibes all around.

And I did that week after week, and after about six months of doing that, I said “I think we have something here, but I really want to dig deeper into the stories. I really want to focus in on these developers and inspire people in a deeper way.” And I thought “Podcast. Audio. That’s the right way to do it. That’s a great medium for that.” And so we launched The Code Newbie Podcast. And I think it was maybe two months into doing the podcast, that I got an email from a company who said “Hey, I’ll give you 200 bucks if you run an ad on your show”, and I was like “Whoa, I can make money from this?!” So that was really the first time I started thinking of it not just as this fun side hustle I was doing while I was a developer full-time, but actually had the potential to be a sustainable business.

[08:04] So that launched just – my thinking shifted from that, from “How do I make this into a media company?” It’s kind of funny, because I guess content creator was the right term, but I never thought of myself as a content creator, because my content was never centered around me. It was never about me being in front of the video camera, or me talking about my opinions. I wasn’t trying to be a thought leader. It was really about shining a light on other people and bringing people together. So I thought of myself as more of a community/media company than a content creator.

And so we did the podcast, and then over the years launched another podcast called The BaseCS Podcast, which was me and a good friend of mine, Vaidehi Joshi, who did this series on computer science, and we basically turned her blog posts into an audio format, into a podcast. And that went really well, that got really popular. We launched a conference called Codeland that we do every year, we had meetup groups all over the US… And so just trying to listen to the community and figure out what’s needed and what people wanted support on, and trying to turn those into some type of content, some type of product to help people has been kind of the name of the game.

And so I ran that for three years part-time while I worked as a developer, and then three years full-time, and then I sold it a couple years ago to Dev.to, which was renamed Forum, is the official company name… But they run the Dev.to platform. And after I sold that, I thought “I think I want to stay on the business side of things. Let me go to business school, so I can learn business things, and I can learn the proper way to do business.” And I got into Colombia and did the Executive MBA program for two years, and I think it was like six months into the program I was like “I think I just wanna start another business.” [laughs] And so after spending a ton of money on this degree that I didn’t really need to start a business, I ended up launching Disco as part of a class that I took, this entrepreneurship class. And Disco was meant to be audio courses. So it was taking my love of podcasting and my love of audio - and audio at that time, if you remember, Clubhouse was poppin’. They raised like billions of dollars…

Kris Brandow: Yeah, and I think this is also about like kind of the making sure that we have an acceptable bar of what is the minimum that we will do as software people… Because yeah, if the floor is that it takes six months to build something, then the floor is it’ll take six months to build something, and we as an industry need to accept that. Because there’s a lot of – I think we as an industry are still in this kind of… We want to believe that the software we build doesn’t have severe impacts on people’s lives.

[01:06:03.11] I think a lot of people want to be like “Well, I’m not building software for an aeroplane, or for any of these things… So my software isn’t life or death.” I worked at a financial institution that was like a literal bank, and gave people loans, and credit cards, and all of this. And I remember one of my co-workers said to me “Well, what we’re doing doesn’t make that much of a difference.” And I’m like “We’re literally deciding whether someone gets a loan to buy something. That is a pretty major thing. And if our software messes up, they might not be able to buy something that they could have otherwise bought, or they might otherwise need.”

And so I think like a lot of us are like “Oh, no, what we build isn’t that important. No, it is. It has very profound implications on society, and we should be very diligent in making sure that we’re taking the time necessary to make sure that we’re building things correctly, and that we’re not doing that malpractice… Which is a very harsh word, but I think it’s one that we need to start doing… Because I think, especially if you look at the social media companies and some of the kind of egregious stuff that they’ve done in the past, it’s like, would we maybe have prevented that if we had better documentation, if we had better ways of raising our voices and things like that? I don’t know if the answer to that is yes, but it’s certain that the way we’re doing things now is not working, and we need to change something. And the thing that’s been constant since the ‘50s, that has been said, is “We need to document things more.” So I think that’s a good place to start, even if it does cost us a little bit more time. But I think also, it’s a planning thing. When you have more information - sure, this one project, maybe it takes an extra two months. But you might save six months on the next project because you have all this information to look back on. So it’s a thing that also builds over time.

Gerhard Lazu: Yeah, it will be just a few dollars. And I think we have a few things to clean… So I basically enabled Storage Lens, which is an option in S3. And you can dig down, so I’m just going to once again share my screen, I’m going to click around for a few things, I’m going to come back… 2469. Obviously, you won’t have access to this, but if you’re using AWS S3, you can enable Storage Loans and have a play around with it.

So what I want to see is here, extended Storage Lens. Okay, and now it loads up, and we can see where the cost goes. So we can see the total storage, we can see the object count, we can see the growth and how things are changing, and how many more things we’re adding… This was in the last day, today, all requests, like month to month… So you can see we have a 1% change in total storage month to month. So we’re approaching the one terabyte mark; not there yet, but getting there quickly. And if we see which are the buckets that contribute, and I have to remember – where was I? Oh, there we go. So we can see Changelog assets, which are the static ones; they contribute 22%. Changelog Uploads Jerod, they contribute 21%. This is the storage costs. And Changelog.com backups, which is mostly Nightly, they contribute, again, 20%. So they’re like roughly evenly spread. So I’m wondering, is anything here that we can clean up? Anything here that we don’t need?

Alex Gallego: So for context, we were three poor kids from Brooklyn before that. That is the truth. I knew how to write C++, but largely, I was living in Brooklyn, and I was broke. We built a really cool system, it was actually used by some of the largest financial organizations, and then Akamai were kind of in the stocks for it… It wasn’t like a huge, hundred-million-dollar kind of thing. It was a small exit, but it was enough to make a big difference in everyone’s lives, or at least for me. I think there was a small return for investors; it wasn’t like a big return. But we didn’t lose money, we all made a little bit of money, and… Anyways, for me at the time it was important. I graduated with student loans. Even though I had a full scholarship in terms of tuition, I still had to pay for room and board, and get loans for that. So that was like a pretty important step I guess mentally for me to just like “Okay, I don’t have any debt” kind of thing. So it wasn’t like tens of millions of dollars, but I did have the flexibility to move without needing to have a job at the same time.

Cory Doctorow: Yeah, Google Wave, right? Google Video… All of those. Meanwhile, their ad tech stack, their server management stack, their mobile stack, their video - all of that stuff, docs, all of it, bought from someone else. Even the satellite stuff, bought from someone else.

So the poultry packers, they gobble each other up, and now there’s three of them, and they have divided America into three territories. They don’t compete with each other. If you have like a cable modem, you know how this works. There’s only one company offering your cable modem. Except if you’re a poultry farmer, there’s only one company that will buy your birds. And they determine everything about how you grow those birds. So you have to buy the chicks from them. They tell you what your coop has to look like; they give you the floor plan for it, the blueprints. They tell you when the lights go on, when they go off, what spectrum light your light bulbs can emit. What food you can give them, what schedule they’re fed on, which vets you can use, what meds those vets can give them… Everything except how much you’re gonna get paid. That you find out when you bring the chickens to market. And because they have regionwide insight into the poultry production of you and all your competitors, they give you money that is calculated to the penny to be just enough to roll your loans and do it again next year.

They do things like A/B splits, where they’ll be like “What happens if we give chickens less food? Will they still get as big?” And they don’t tell you that you’re in the experimental arm. And you go to market with sickened chickens, right? “What happens if we don’t give them this medicine?” and your chickens all die, and they’re like “I guess you’re on the hook for that.”

When people speak out about it, they are struck off. So if a poultry packer refuses to do business with you, and you grow chickens, and you’ve got a million dollars worth of debt for your physical plant, you’re dead, right? There’s nothing you can do.

And so farmers who speak at like state congressional hearings, state legislative hearings get struck off. And there was one guy who spoke out on a state regulatory proceeding, who after he got struck off, he was like “Well, I guess the one thing I can do is I’m really good at fixing these standard coops that the poultry processor requires.” So he went into business doing that. And they told every farmer that if you hire this guy, we will not buy your chickens either. They just destroyed him.

Jerod Santo: Reading Max’s post, he agrees that tech debt is often a necessary thing, stating “if a growing company isn’t taking on some measure of leverage, then it’s likely not shipping as fast as it should be.”

But there comes a time, and sooner is usually better than later, when you have to pay down the debt you’ve been accruing. But how is the best way to go about it? That’s where Max’s post focuses most of its efforts. His suggested process includes surveying the debts you have, assigning effective interest rates based on how active or dormant the subsystem is and how much it costs to maintain, then prioritizing paying it down based on those rates, much like you’d pay off high interest loans before the low interest ones.

Finally, Max has this to say about the entire process. “In the end the key thing here is to use frameworks like this to remain solvent in terms of our code base and product workflow. With tools like this, we can avoid bankruptcy (literally and figuratively).”

Christina Warren: Yes. Well, so it’s funny, because my background, like going all the way back to when I first got into anything, when I built my first website when I was 12, I’ve always – for as long as I can remember, the two things I’ve loved the most in the world have been movies, pop culture, TV, and technology. Those have always been kind of my two biggest passions. And I was lucky enough when I was a journalist, and the story there is that I graduated from college in 2008, right in the middle of the Great Recession, and I’d always thought I was going to go to law school, and wanted to be either a technology attorney or an entertainment attorney; ideally, an entertainment attorney, because I figured “Well, I could leverage that and maybe be a producer, and work at a studio, or for a TV network, or something, and use my power to get projects I want to get made…” that was the big pie-in-the-sky goal. But in 2008, going to law school and taking out loans didn’t seem like maybe the best idea. It was a little bit – I think that I totally understand the hesitancy of kids who are graduating now, or who are going to be graduating next spring, you know, about the uncertainty in the world… And so I did the very logical thing, which - I’m being ironic here, because it’s not logical at all… And I was “Well, I’ll just go into the very stable career of journalism.”

Brian Kernighan: And I think probably what you want is something that makes whatever pathway is gonna work for somebody readily available. That they don’t get caught out of it for financial reasons, or discriminatory reasons of any sort, and all of these things. What you’d like is this idea of kind of equal opportunity, and people go through whatever process makes the most sense.

You’ve seen from time to time discussions about, for example, whether it’s better to go to college and get a degree in something, or better to go into a trade, like plumbing, or something like that. For example, you can be something like a plumber, get out of high school, go through an apprenticeship, learn a trade, become very good at that, actually make a fair amount of money, and for many people that might be a viable alternative, and perhaps even more satisfying than going off to (let’s say) a four-year college. A two-year college is some kind of intermediate position in that…

I think the main thing is to make it so that anybody can understand that the options are, and find a pass-through where the options make the most sense for them. I suspect there are a lot of artificial barriers that the country would be better off if those barriers could be reduced. And I don’t know what they are for different people. Going to a private school like Princeton, an expensive place - that costs money. So there’s a financial barrier to that. Some places solve that with student grants, other places you have to take out loans, and then you’re stuck with debt for some period of time afterwards.

So I don’t have a good solution. I think it is important to make it possible for people to go in whatever direction seems to work best for them, and to have a clear idea what the trade-offs involved are.

Kris Brandow: So it’s that sense of control, in a way… I think that point you had about startups is very accurate as well. I think people go into a startup and they’re like “This is going to be fun”, and it’s like, you’re taking out this nice 0% interest loan and you’re buying a giant house, or something… But it’s like, no, those payments are gonna come due each month, and if you get behind on those interest payments, it’s gonna take you a long time.

[01:00:14.20] I feel like that’s what happens - eventually, a company gets acquired, or there’s some exit that happens, and then you look around and you’re like “Wow, there’s a lot of stuff that like, we thought we were doing this in such a great way, but all we were doing was just paying off our old loans with the new loans, and it’s like, all of this has now caught up to us, and we have to actually sit down and figure out how to retool this so that we can actually move forward.

And I just think the hours people work at startups is a lot more than people think… It’s a fun experience, I think, for people that are young and really into what they do, but I think it has a number of downsides as well.

Gaëlle Sharma: Well, probably like many other product managers, it was by accident. I had no idea that there was such a thing as a career for product managers. I’d never heard of this. So earlier on in my life, when I was in college, I was really interested in public health, and that led me while in college to work at a community development thing. I was doing a fellowship where I was learning more about the community, what did it mean to do impact investing, and how we could make loans to small businesses in Chicago; that’s where I was located.

I had the chance to have a manager at the time during my fellowship who was giving me some feedback, and one day she said to me “Gaëlle, I think you would be a great product manager and that you would really enjoy it.” I had no idea what that was.

So she taught me a little bit about what it was, and she explained how some of the work that I had been doing was very similar to a product manager, because I had been helping create new products for the community development bank, and working really closely with customers and solving customer problems… So she really kind of opened my eyes to this potential career… And then later on, as I was trying to think about what to do as a next step after the fellowship, I found some product associate positions and I decided to give it a go and apply. It turns out I had a lot of good experience… And that was it. I just kept going further and further into this career path.

So yeah, somebody had heard about it, and saw that I had a lot of the skills or interests that mapped on well to doing a good job in this role, so they kind of like pushed me towards it.

Nhung Ho: Yeah, so there’s two parts. I can speak for my team and how we built models… But I’ll start with how we as a company realize and really reinforced one of our beliefs from before the pandemic and made it much stronger. We have our product called QuickBooks Capital, and so we provide small business loans to our customers. And I would say the magic or secret sauce there is we have a lot more data than a standard lender would. For our customers, we have all of their accounting data, and from that, we’re able to actually offer loans to customers who traditionally would not be able to get loans.

What we saw during the pandemic was that there was a large population who were underserved, in general, who would not have been able to get loans, who we were able to offer PPP loans for, because we knew their data very well, we knew their risk profile, and we could actually underwrite.

I think that really reaffirms our decision that we need to continue to invest, and figuring out ways to continue to serve the underbanked and underserved population - that’s at the company level. And there’s a lot that you can do in the data to figure out whether or not you’re capturing that population or you’re actually doing the right thing there.

On the other side, the more technical side, with our teams, is that we’ve seen just – you know, with these large worldwide events… I like to point back to 2008, where you think it’s like a once in a generation event… And then ten years later you have another once in a generation event. These events - you can’t write them off as one-time anomalies; you actually have to figure out how to fold that in and be able to automatically regress against them, learn from them, and then figure out how to apply it moving forward.

Like you said, we’re at the tail end of the pandemic right now, hopefully - fingers crossed.

Mat Ryer: Well, I’ve found this very interesting story about Citibank, who due to bad UI design accidentally sent out 500 million dollars. So yeah, apparently there was a form in this very complicated, old bit of software, and it’s not very clear when you look at it what you should do, and I guess there’s no confirmation… So somebody actually paid out – they had loans that they owed, and they accidentally just paid off all the loans, to the tune of around half a billion.

And apparently, in New York, according to New York law, if you make a wire transfer by mistake, then you have to pay it back… Unless it’s to pay off a debt, and that was this case. So the judge found against Citibank; it’s ongoing, the case, but… There you go, the importance of good UI design. I think it’s vital, and this just highlights that. Your websites might not lose 500 million dollars, but it still can be annoying.

Chandler McCann: Yeah, so that was very exciting, and I can’t thank Jeremy and the team enough for supporting us to build out this vision… But June of last year we sort of launched our program and opened up our application process, and we actually got applications from ten countries on five continents, which was pretty exciting for our first year of the program and its inception.

[39:52] Since then we’ve been working with six non-profits. Kiva, which is obviously a large lending platform in the non-profit space right now, so helping them predict the likelihood of which loans would go unfunded or not. DonorsChoose, helping provide supplies to teachers and classrooms, Anacostia Riverkeeper here in Washington DC, helping them forecast E. coli levels in the river, given some sensor data that’s being read, to some very interesting healthcare use cases with the University of California San Francisco, the Zuckerberg Spinal Cord Institute, where we’re looking at ways to use OR data to help improve outcomes for spinal cord surgery… And then finally, working with the University Hospital of Mannheim over in Germany, where we’ve been forecasting top World Health Organization causes of death, as well as predicting patient mortality given people that come into the hospital.

So those are some of the use cases we’re working on now, and we’re excited to just continue to broaden the impact to people we’re working with, and keep the program growing.

Michael Geers: …housing loans, for example. And these areas are two different areas. In these companies, most of the time there are also different departments in a classical way of speaking. So doing the cut from the user’s perspective and the mode the user is in when he enters your site works well for us.

Jerod Santo: So, in the post you draw an analogy to debt… Well, I guess the analogy's already there… But in finance, you mention that debt can be problematic, but it's not automatically a negative connotation. If you're in debt to the point that you can't get out from under it - maybe you have student loans, and there's no way of declaring bankruptcy in this country, and you have crippling debt, that's really a problem. But there's also like a smart use of debt. We do it all the time, it's just part of the way that we live life. You point out a home mortgage. So draw that analog for folks. Of course, we know there's a debt analog, but you say with a home mortgage, you're not crying about it, you're happy about it.

Sherol Chen: Yeah, of course. And yeah, Google is not the only company that’s offering these capabilities… But yeah, it definitely starts with what you’ve been saying earlier about how you have a certain language - say you’re a German doctor and you need a tool that recognizes that context; or you’re searching for – cat and food pictures is general, but maybe you’re an automotive manufacturer and you have all this label data of car parts, that isn’t gonna show up in the more general use cases of the pre-trained models.

So what you need to do is – you have this dataset, and you know this dataset is good and accurate. What you would do – and in this case let’s use… I’ve talked a lot about tabular data. So we’re using spreadsheet data. Spreadsheet is something that almost everybody uses today… But in this case, what feature we’re looking to establish within a spreadsheet is the ability to select a column in a spreadsheet, and say “Given all these other columns, can you predict what the value of this column is gonna be?” It could be categorical, it could be numerical…

The example that I’ve been using a lot is – one of the open datasets that we have is a banking dataset that shows different types of banking customers, and things from how often they deposit, to what their income is, to what kind of jobs they have… And then trying to infer which banking customer is most likely to turn on direct deposit, so they’re gonna have their paychecks directly go to the bank. So what you would do is you would find, first of all – we talked about CSV files; of course, this is Google, so you could have your data in BigQuery, and you just give it the BigQuery ID.

You go to AutoML tables, and you would load in that dataset; you would import it, and it will be labeled, it will be either categories, or numbers, or text… And from there, you have the opportunity to look at the different column names. From the column names you can then nullify or select your features. In this case they have things like age, education, balance, housing, loans… Things like that. From there, you look at each of the column types and you select one to be the target. In this case we’re gonna select direct deposit. This is like a true/false, like “Does this person use direct deposit or do they not?”

Then there’s other analytics you can look at. It gives you the opportunity to see the spread, or whether there’s any columns that have missing values or invalid values, things like that, where you can remove – and this is where a data scientist [unintelligible 00:47:22.03] these kinds of tools actually are very useful, because they often have data… They have lines and lines of data, for which some of the data might not even be valid. So being able to detect that automatically would be very useful for them.

From that point on you would just then train the model, and the selection you’d make there is “How long do you feel like training for?” Usually one hour, honestly – these are approximations, and it changes case-by-case. But an example, for this case - I would say for 4,000 lines of input, you could train for an hour and you’ll get a decent result. You’ll get 95% accuracy for predicting whether or not someone has direct deposit.

[48:10] Once it’s trained, you select that threshold. If it finishes earlier – of course it’s gonna detect things like over-fitting, when it’s just kind of reaching diminishing returns, so it will terminate early as well, if it needs to. But at the end of that you get an evaluation. So you look at – there’s the recall, there’s precision, there’s looking at the true positive rates and the false-positive rates, there’s the accuracy… So in this case, this model that I’m just gonna pull up on right now - this model has an accuracy of 91%, which is pretty good.

Again, case-by-case - what is your use case? What is your threshold in production? And it’s gonna differ. It’ll also show you feature importance, how the labels are doing, and you can then – basically, you could deploy it right then and there; you could just deploy it within AutoML and then you can run online predictions, you can run batch predictions, or you can export the model. Like I said earlier, you could export it; it will be a TensorFlow package. You can run the model in a Docker container, you could take that Docker container, you can download it onto your local machine, or you can just not do any of that and just use the REST API, and be able to make predictions that way.

So that gives you the end-to-end – basically, once you load it in and you click Train, and the training completes, you’re basically ready to… And you have to deploy it as well, but you’re ready for online predictions after those few steps.

Jerod Santo: Paid down some of them student loans. [laughter] Very cool. Well, I mean, even though it happened a while ago, I’ve just gotta say congratulations. That’s a heck of an accomplishment, I think.

Paige Bailey: Yeah, and I can hunt down a link for the PDF of that guy as well… But it was awful. I see a question from David, “AI leading to inadvertent discrimination is really interesting” and that is 100% true. Tiny input differences do have massively different outputs over time.

The other thing is that if the models aren’t kind of checked and QC-ed, they just perpetuate the bad assessments that they were making before, you know what I mean? So they will preferentially not give loans to people of color who come from low economic backgrounds, or they will preferentially not allow people of color to be able to be up for parole in a reasonable amount of time… Unless you are actively asking questions to challenge the model’s assessments.

[47:57] As an industry, I think that it’s on all of us, if you’re doing anything with machine learning, to ask these questions if something smells fishy… Again, it doesn’t matter if you have a Ph.D. If it sounds like it a model assessment might be wrong or biased in some way, absolutely challenge the data science team that’s being used to create it.

The entity that I was mentioning before, PAIR at Google, People in AI Research, is a human-centered research and design initiative to make AI partnerships fair, and to make sure that we have tools that will be able to help spot these biases. TensorBoard is one of them, the what-if tool is one of them, the visualizations that happen through this publication called Distill.pub - and that’s supposed to be a link, but I guess it didn’t go… But it’s this beautiful publication that goes into what is actually happening whenever a machine learning model makes its assessment, and how can we really understand the mathematics behind it, because it’s so dang complicated; it’s more complicated than a human could possibly understand… But how can we shine a light onto these decisions.

Brian Behlendorf: Right, and it might seem like “Why bother thinking about this as a problem?” Well, in emerging markets where they had started to digitize land records, in order to give people not only a sense of ownership over the property they’ve perhaps liven on for generations, and the idea was also allow them to be able to take out loans to start businesses, right?

There’s an economist named Hernando de Soto who wrote a book called The Wealth At The Bottom Of The Pyramid, which was all “Let’s help India build their economy, and South America and others, by allowing people to build businesses on top of the ownership in the land that they have.” So there was a big movement to digitize these systems, but it was discovered that this also made it easier for corrupt bureaucrats–

Sandi Metz: Yeah. It turns out the problem has a couple of really great characteristics. One, it’s kind of fun. We can all sing in class while we work on it. Two, it’s a problem domain that everybody knows; we don’t have to teach about bank loans in order to have a class exercise. Three, it seems simple but actually it’s sneakily complicated. Those three qualities combine to make it a really useful programming exercise.

There are a number of ways to do it, I wouldn’t say that I have the perfect solution to it at all, but the nice thing about the problem is the kinds of difficulties that you get into trying to write code for it, provide a scaffolding to talk about big ideas in object-oriented programming. So it’s really easy to talk about the Liskov substitution principle, because there are natural Liskov violations in that song. There comes a time when it’s really easy to talk about inheritance vs. composition because we end up in situations where we have to choose what we’re gonna do.

It’s easy to talk about mutability. There’s a place in the song where having immutable objects is possible, but you could mutate them instead. What does it mean to do that? We’re an object-oriented language, but immutability is a big idea that we should freely steal from the functional people.

The goal of the book isn’t to write the perfect solution to the 99 Bottles of Beer problem. The goal of the book is to take a problem that everybody knows, that you don’t need any extra brain power to understand the domain, and use that problem to explain ideas in OO. And it’s pretty perfect for that.