Gerhard is back! Today we continue our Kaizen tradition by getting together (for the 10th time) with one of our oldest friends to talk all about the continuous improvements we’re making to Changelog’s platform and podcasts.
Matched from the episode's transcript 👇
Adam Stacoviak: Chasing not the nines in that case, right? That’s the anti-nines.
This week we’re talking with Bruce Schneier — cryptographer, computer security professional, privacy specialist, and writer (of many books). He calls himself a “public-interest technologist”, a term he coined himself, and works at the intersection of security, technology, and people.
Bruce has been writing about security issues on his blog since 2004, his monthly newsletter has been going since 1998, he’s a fellow and lecturer at Harvard’s Kennedy School, a board member of the EFF, and the Chief of Security Architecture at Inrupt. Long story short, Bruce has credentials to back up his opinions and on today’s show we dig into the state of cyber-security, security and privacy best practices, his thoughts on Bitcoin (and other crypto-currencies), Tim Berners-Lee’s Solid project, and of course we asked Bruce to share his advice for today’s developers building the software systems of tomorrow.
Matched from the episode's transcript 👇
Adam Stacoviak: Well, I’ll give you the TL;DR. Somebody who was just able to con people with large amounts of money, various bank accounts, he would con one person, then con another person… Credit cards, bank accounts that didn’t match this name, so when it comes to their customer and attaching a bank account to a cash app account, or to a Venmo account that doesn’t match your name - that’s anti-money laundering; that’s what you’re speaking of.
I mention that because this person never really used a bank account or a credit card that was in his name. It was always somebody else’s. So when it comes to these intermediaries - I don’t know how he was able to by-pass this stuff, but that’s what they do. They say “Okay, your account is in this name, Adam Stacoviak. Does your bank account that you’re attaching, money coming in or money going out, match that same name? If not, we’re gonna flag it for anti-money laundering and you have to prove you own it by way of W-2, or some sort of tax form, or–” W-2 wouldn’t fit there, but some sort of tax form that you filled out, that says this account is yours, or whatever. Something. Right? So that’s where that comes into play, this intermediary benefit.
9 out of 10 AI projects don’t end up creating value in production. Why? At least partly because these projects utilize unstable models and drifting data. In this episode, Roey from BeyondMinds gives us some insights on how to filter garbage input, detect risky output, and generally develop more robust AI systems.
Matched from the episode's transcript 👇
Roey Mechrez: Definitely. And I think stability is well coupled with the ability of the model to generalize. Hypothetically, you have a dataset that is the only distribution of whatever, and you can generalize it well. But that’s clearly not practical. You don’t have access to the entire distribution, and you cannot generalize to the entire distribution, obviously, and you definitely don’t have [unintelligible 00:21:46.06] But when you shift into practice AI, I don’t care about generalizations. I want to be good on what I need to be good at, and make sure that nothing that’s out of that good will come into my model. And that’s why I call it garbage in/garbage out. I want to make sure that I have a trained set, I have its distribution, and that’s the data that I’m going to get, how we’re going to achieve that.
I think that’s one of the most important problems that cause this failure rate, the 90% failure rate that I talked about in the beginning. Because AI researchers, data scientists, they know how to build models, and they can achieve this 95% accuracy in the lab, for sure, with a stable, static dataset. The production is not stable, it’s not static, and it’s definitely dynamic and it’s going to shift. Data drifts, data shifts. Something is going to change. And how you react to those changes. And then why it’s important, by the way.
I think while we’re progressing with value around AI, we’re also shifting from, let’s say, statistical applications, which I define as something where… The difference between 80% or 81% or 82% accuracies - okay, nice; better to the business. But there are mission-critical problems. Claim assessment in insurance, anti-money laundering, fraud detection, defect detection in PCB boards, whatever… These are mission-critical. The difference between 80% and 81% is critical. It could be massive to the business. And I’m not even talking about applications where their business requirement is 99%. I need 99%. The model can achieve 90%. What do we do now? And that can be breached. There is a way.
