Attack of the Canaries!

Alright, we are here with Haroon Meer from Thinkst. Hey, thanks for coming on the show, Haroon.

Thankst for coming on the show. [laughter]

And it starts…! Thanks for having me.

Sorry about that. I had to do it.

We’re happy to have you. Owen Valentine - shout-out to Own, longtime listener; he likes to put episode requests in, and he said hey, talk to you. And I said, “Okay.” I take orders around here, and I checked you out, and I thought “This is pretty cool. Security products coming out of South Africa, Canary tokens…” Lots to discuss. Where should we start? I know you have strong feelings on bootstrapping versus VC funding, I know you have strong feelings on InfoSec the industry, you probably have a cool perspective coming from where you’re coming from. What’s most interesting to you?

So I think it’s pretty open. I think we should probably start with what we do. So Canary tokens is probably a reasonable place to start.

Yeah, let’s hear it.

Okay. So effectively, what we do is we build products to let people know when they’ve been compromised. The opening logic is – so in our previous lives, most of our early team worked as pentesters, so breaking into networks. And one of the terrible secrets is that for years and years we’d break into networks all around the world, and nobody knows. Not until you hand in the report. And this hasn’t changed much. So it happens when you’re doing pentesting, it happens when real attacks happen. And so companies just find out they’ve been compromised months after. And so our whole pitch is to try to fix that.

So canaries are an old concept, which are honey pots. And honey pots have gotten a bad name historically, because mostly, they were used by the research community. So people would put up honey pots and say “Look at how many attacks were from Russia. Look at how many attacks were from China”, which is pretty useless for real world activities. And what we do is we say “If you had honey pots on your network, and they were really low effort to deploy, they end up giving you a really high signal.” So you find out at two in the morning that Bob from accounting just tried to log into this machine that really shouldn’t be there. And what it works on is just the logic that the people on your network or in your infrastructure know your infrastructure, but attackers who land there need to situate themselves. And so typically, those attackers run around like bulls in China shops, and they get there. And so they explore things and they touch stuff. And inevitably, they touch these canaries, giving you a really high signal that badness is happening. And that’s all we do with Canary. We make it super-easy to deploy them, so that people actually do it, and then we focus really hard on not generating extra noise. So customers with like hundreds of canaries will get four alerts a year, so that when they get an alert, they know they need to react to it.

[07:50] And so Canary was the first product we built, and then we built Canary tokens, which are the same concept, but much smaller trip wires. And for the broad applicability, for your audience, we’ll give you for example an AWS credential file. You put it on your CFO’s laptop, and when anyone uses that credential file, that API token, you will get a message saying “Listen, the AWS creds that were only on your CFO’s machine - somebody just used it to log it.” So again, you get a really high-quality signal that tells you someone was on your CFO’s machine.

And so Canary tokens are bunches of little tricks like that, that are really hard for attackers to resist, but gives you a really high-quality signal that something is going wrong. And for Canary tokens, we give them away free. And so literally, millions of people have used them, or use them to figure out when they’ve been breached.

It’s a really simple concept. I mean, this is like read receipts on things that you don’t want someone to read, you know?

Exactly right. I’ve done this talk in a bunch of places where I point out that really simple things, done well, are in really short supply. And it’s a whole other soapbox of mine, where I feel like people building products are incentivized without anyone being moustache-twirlingly evil. The world kind of sets up so that every time you speak to them, you ask “Well, what’s new in the product? Well, are you now doing this?” And so nobody is ever incentivized to do something and just focus on doing that thing super-well. Because what they’ve got to do is keep showing new features, so that you think that it’s worth it. And we fight the urge really hard. We try really hard as a company to make sure that we always did simple, always low noise. Our CTO says “Just as reliable as a brick.” Like, people need to be able to build on it, no complications, and know that it will work. And that’s our pitch.

These canaries act as like standalone machines on a network? Give me an example of the footprint. How does this work?

Yeah, exactly right. So version one, we shipped these little hardware devices, which in 2016, when we started, nobody was starting a company saying, “Let me ship hardware.”

Now it’s cool. Yeah.

Yeah. It worked out pretty cool.

Now it’s kind of cool.

[laughs] But part of the reason we did it was because we really wanted it to be easy for security teams to deploy. And security teams still have a problem with “Hey, can I spin up in the DMZ? Can I get a virtual machine?” And this way, we were saying “Take this box, plug it in, and you’re good to go.” And typically, each Canary imitates exactly one host. And so when you plug it in, you say “Listen, I want you to be a Cisco router.” And from that moment on, that device, its MAC address is a Cisco, the services are Cisco. If you NMAP it and get its TCP/IP stack, it’ll respond like a Cisco. And the work that we put in is that if you then with literally just two clicks say “I want you to be a Windows server instead”, that device reboots, and now it’s a Windows server on your network. And now it runs a Windows fileshare, or Windows RDP, and you can enroll it in Active Directory. And the whole point is that it shouldn’t be hard for you; you should be able to go “Make this a Windows box, put it in my AD, enable RDP and a fileshare.” Or you say, “Hey, I want you to be an IBM mainframe, expose TN3270N LDAP.” And you drop them and you forget about them.

We’ve got customers who never looked at their canaries for seven years. And what you’re looking for is in year four, when people break into their network and are logging on, you get this one message that says, “Listen, somebody found this Windows Share. Somebody went into the directory called Exec Salaries, and somebody copied all these files. You’ve got a problem.” And that’s the whole pitch.

[12:07] What about the stack trace after that? Is there – I’ve got more questions, but is there any sort of like… This seems like error monitoring, basically, for an application. But instead it’s sniffing out attackers, finding hosts on your network.

Yeah, it’s literally what we’re looking for. And the way we pitch it as “We want to give you one clear signal that you’ve got a problem now.” And we’ll give you the details related to that attack, but we end our mission there. We don’t then do forensics beyond that, or all of that stuff. And part of our pitch is like from old Unix - do one thing, do it well, play well with others, give you an output so that you’ll work with others. And our major push has always been “Make this quick and painless.” So we obsess about that to ridiculous degrees. It’s got to be quick, got to be painless.

So the repository on Thinketh – not Thinketh…

Thinkst.

…Thinkst, on your org on GitHub is Open Canary. Is this the software that’s running on your hardware?

It’s a version of it. So what happens is we build canaries, and beyond Open Canary – so Open Canary shares part of its internals. But if you downloaded open Canary, you’d have to install it, you’d then need to make sure you’ve got monitoring on it, and reporting on it. And if you subscribe, if you buy our Canary service, you get these devices, but the devices report into your console on AWS. And so from your console in AWS, you get to click on a Canary and say “I’m tired of you being a Cisco. I now want you to be a Synology NAS.” Your device reboots, and now it’s a Synology NAS.

And then I mentioned earlier that version one was these hardware devices. Since then we’ve got VM options, or Hyper-V, VMware, or cloud options, GCP, as your AWS. And the Canary tokens which we spoke about, if you’re a customer, you get your own private Canary token server. So literally, you can mint a jillion of those tokens free, all through your environment. And yeah, it becomes the lowest-priced, high-fidelity alerting you can get.

My lens is sometimes the homelab lens. And so I’m thinking of this from the homelab perspective, because you mentioned Synology, which is awesome, because that’s like mostly – I mean, it’s not only in the homelab; there’s a lot of small businesses, offices using that, for example, so…

It totally makes sense. And part of the joy with Canary, which in some ways we got a little bit lucky with, because we thought it was a good idea, obviously, which is why we started building it… But something that we didn’t really count on until we saw it in action is that with lots of the stuff, that’s why attackers are on your network. And so people sometimes go “Well, what happens when attackers get smarter, or attackers wise up?” And the simple thing is, as pentesters, this is what you do; you go further on the network by finding one more open file share, grabbing one more config file that had a password… And you can’t just not do that; that’s what you do.

[laughs] Yeah, exactly.

If you take the Canary token example, if I get to your CFO’s machine, and if you tell me “These folks are running Canary tokens”, and I now see an AWS API key on that machine, am I going to not try it? Like, potentially, that’s access to your cloud. And so one of the things we’re super-proud of is we do zero outbound sales. All our sales so far have just been internal. And we collect a whole bunch of tweets on a site called canary.love, which is people saying nice things about us. And one of the interesting things is that you’ll see lots of those comments from pentesters and red teamers saying “Well, now when I find stuff on an engagement, I don’t know whether I can use it or not, because maybe it’s a canary token.” Or “Now I’m scared to try this when I find it, because hey, maybe.” And so it’s interesting, because it changes that calculus a little between attackers and defenders.

[16:23] Yeah. I have a couple of questions, both on the implementation side, but also, while we’re talking pentesting - because I’ve done some pentesting as well, and you’re absolutely right; you basically are feeling around in the dark, and you’re just looking to discover… You’re trying to shine light on new areas of the network. And so of course, you’re going to like touch and feel, and like that’s exactly what you’re after. But my question is - and maybe you can’t divulge, but is there a way to like fingerprint a canary without touching it somehow? If I was super-smart, what would I do?

Yeah, it’s a great question. And it’s been our thing from day one. When we proposed this, one of the early anti takes would be “Well, you’re starting an arms race, because now people will start trying to fingerprint you.” And I have a bunch of answers for it. One of them is - at least get into the arms race. Like, right now you’re just getting your tail kicked as a defender.

[laughs] Yeah, true.

Yeah. Get into the arms race. But two, we can, as far as possible, try to identify you trying to identify us. So for example, the early NMAP’s a really good example. When NMAP does an OS scan on you, it has a very observable fingerprint. And so a Canary will be able to tell you “Hey, you’re not just being port-scanned right now. This person is NMAP OS-scanning you.” And depending on how you configure your network, you might say “No, we should never be NMAP OS-scanned. Let’s react to this.” And so it for sure starts this arms race where clever people will try to figure out ways to do it, but we ended up being on a better wicket there because all I have to do is say “Hey, what you’re doing to me is not usual. Because you shouldn’t be talking to me at all.” And so it dramatically changes that calculus, because now attackers have to be careful of everything.

One of the folks who work for us says our entry level package should just be stickers saying “I run Canary.” Because an org that just says “I’m running it”, you just put attackers in a horrible position. Like you said, if you spend all of that time feeling around in the dark, now you’re just terrified that anything you touch is gonna –

Is gonna bite you.

Exactly.

It’s a bomb, essentially. Well, the tripwire; it could be the tripwire, to use your terminology.

It’s like playing minesweeper, you know? But with no information. You’re like “Well…”

Yeah, exactly. You’re right.

For sure.

So on the implementation side, the software side of what Canary does - is it masquerading as these different OS’es and services, or is it actually like rebooting the VMs? Or how does it actually do it?

No, we masquerade. So there have been people who’ve tried deception products who boot full-blown operating systems, and we’ve got – like for many things, we’ve got strong opinions on that sort of stuff. One of the things we feel very strongly about - for years and years as pentesters we’d own networks because of their security devices. So they’d buy stuff, they’d implement it, it’ll be dual-homed… You’d take it on this network, and you’ll hop across to that network. And so we spent crazy amounts of time making sure that you will never attack a canary and be in a better position than you were in before you attacked the canary.

And so we’re not running vulnerable versions of Windows; we’re faking out the TCP/IP stack, we’re running network services that we’ve written in memory-safe languages, on jailed file systems… And again, we’re not Oracle, we’re not saying we’re unbreakable. What we are saying is we’re really conscious to minimize splash damage or blast radius. So nobody’s going to be better off for attacking us.

[20:06] And one of the ways we’re able to do that is by emulating stuff. And then the question is, can we emulate enough of it to pull off the con? Because effectively, you’re trying to con a user. And again, there’s two things to that. The one is we believe we can. So for some of them - we’ll talk good Redis right till we’ve caught you; or we’ll talk RDP till you authenticate and tell us who you think you are. But on the other hand, I think it’s one of those things we make a mistake with when we judge security product sometimes, which is where people question “Well, I can think of ways to defeat this in the lab. What if I did this, and then did a timing attack, and I could tell that you are responding to me slower than a real machine would respond to me?” And while we’re doing that posturing, Snowden is mapping to every share that he can, and stealing every PPT that he can in your organization.

So if Snowden was getting an SMB share, or a Samba share, or a black box version of a Windows Share, he didn’t care. He was just grabbing files. And this still happens on networks everywhere. Attackers want what they want, and almost don’t care what’s underneath it.

Another super-interesting thing with that - we spend crazy amounts of time making sure that our con is complete. And the user never has to know this, but it should completely con an attacker. But I’ve been on pentests where it’s the middle of a Windows network, and suddenly there’s a skull box. And I’m like “What is the skull box doing there?” That doesn’t stop me from browsing its filesystem. Like, more than anything, I browse that filesystem.

And so I think people overestimate how deterred an attacker would be if something looks odd. In actual fact, that’s par for the course. Everyone has a Frankenstein box that they’ve forgotten about, that actually has the keys to the kingdom because they’ve forgotten about it.

Yeah.

And so it just turns out to work in our favor.

Yeah. Just really hard to ignore that one weird-looking box that you think “Well, maybe this is my way to the next stop”, you know?

Exactly right.

It’s like an unsolved mystery, basically. “What’s inside the box?”

Well, especially when you’ve been not making progress for a while, and you’re like “Oh, wait a second… What’s this?”

It’s a super-interesting insight, and one we didn’t have when we started off. So we spent all this energy making sure we could totally imitate stuff… And at some point we realized “Hey, we used to give classes on pentesting at Black Hat for several years…” And one of the things – like, we’ve got slides where we tell people “If you land on a Windows network, and you see lots of Windows servers, and one lone Red Hat box, go for the Red Hat box. These people know how to configure their Windows network, but they had to put up this Red Hat box for their telephony, or for their NAS… And you’ll own that box, because these Windows people don’t know.”

[laughs]

And then when we made canaries, our first instinct was “We’ve got to make stuff that really blends in.” Well, actually, we were teaching attackers “Go for the stuff that doesn’t blend in.” And so again, totally by accident, it puts us in the fortunate position where we say “Listen, just deploy your canaries. Don’t think too hard about it; just deploy it. You almost can’t deploy it wrong. Because if it blends into your environment, that’s cool, and if it sticks out, that’s cool. Just deploy them.” Empirically, it works.

[24:01] What’s the typical ratio, like canaries to real?

That’s a super-good question. So initially, we thought it would need to be high. In fact, we added as an open research question for how many needed to be deployed. And in truth, the number can be shockingly small. And in part, that’s because attackers are bouncing it on networks for months. Until ransomware started giving attackers incentive to disclose themselves early as they ransomed your network, the average used to be more than a year. So attackers would sit on networks for more than a year. And during that time, what they’re doing is trying to find your crown jewels. And so if you have 10 canaries on a large network, but in your DMZ or next to your key servers that’s what the attackers are trying to do, they’re trying to get there.

And so part of the way we pitch the service is take five. Take five to make sure that this is not vaporware. And again, we’ve got kind of a strange sell approach. It’s almost a hyperversion of the PLG that everyone looks for. I mentioned we’ve got no outbound sales team. And so our pitch is you go to our website, you see the price, and you say “Try five of them.’ And nobody tries to upsell you, ever. And typically, what happens is within a year you have a pentest, and those pentesters get rumbled by your canaries; or you catch something. And then when it’s time to renew, typically someone says “You know, actually, we’d like to put these in all of our remote offices.” Or “Actually, we’ve just made an acquisition, and we don’t have time to go down there. Can’t we just send four canaries down there?” We’re like “Cool, we’ll just ship them to that address.” And people can grow their flock – Literally, we’ve got customers now paying us hundreds of thousands, who we’ve never met, and we really don’t have the sales team to sell to. And they’ve just up-sold themselves every year.

That’s interesting. I would expect the sales process to be somewhat difficult, because your payoff moment is like when they get hacked, you know? And it could go, maybe – I mean, I guess if you have an annual pentest, maybe that’s what triggers it. But if you don’t, you could go years without ever providing value, quote-unquote. Visible value.

Yeah, so it’s a good insight, and it’s another place where I think we got accidentally lucky. And by that, I mean, I think if people were not doing pentests, far more people would have been questioning the value of it. And we didn’t have a plan. Like, if you asked us on day one “Well, how –” Like, we hyperoptimize for being silent, unless it’s a real attack. And so how would people know that there’s value? And typically, what happens for us is – so there’s two things, the one is… This sounds super-corny, but I’ll say it anyway; we make sure that the installation is delightful. And so our initial pitch is “Listen, 7.5k, you get five canaries and the hosted console. Just try it. Like, how bad can it be? 7.5k, try this.” And then we got super-lucky, because Slack used us and said something publicly, and Airbnb used us and said something publicly… And so then our message was “Hey, 7.5k and Slack says we’re cool. How bad can we be?” And then we’ve got to make sure that your first experiences with us are delightful, because we’ve got to convince you… And so we put a lot of effort into removing all the suck from that experience.

[27:44] And then we’ve basically got a year to earn our keep. And typically within that year, we’ll catch pentesters, or we’ll catch real attackers, or we’ll catch some network misconfig that you never saw coming… So you’ve got canaries in this zone where nothing should happen because it’s sealed off, except someone made a firewall change and now traffic is hitting it… And we have almost a constant refrain; and it’s a little bit unfair, but I’ll totally take it… Where customers will say “We spent so many million on our security products, but when we had that pentest, canaries were the only thing that caught them.” And it’s like “Yes, because that’s what we optimize to do.” But the look is just great for us, because the customer’s like “Well, we’ve just paid you 30k and you’re the only thing that caught those attackers.” And it’s why we’ve been able to consistently grow and keep our customers.

Break: [28:47]

I think the other thing that could kill you, which it sounds like you’re hyperoptimized around, is false positives. That would destroy your value as well, because we’ve all deployed, you know, Nagios, for instance, which is not a security product, but a network monitoring product… We’ve all been nagged to death by false positives, and like throw it out the window, you know?

Yeah… And so in the company, we take it so seriously. We’ve got blog posts going way back, where we’d blog about features that we’ve removed. So for example, when a Canary acts like a Cisco, and you can say “Enable fake SSH, enable fake Telnet, enable fake finger”, we used to say enable SNMP, because everyone’s got SNMP. And the number of things on your network that just randomly talk SNMP would set those things off consistently. And we could explain to people “Hey, please don’t enable SNMP”, but most people would, and then get that false positive. And so we disable it and remove it, and say “No, from now on you can’t do this thing incorrectly.” And yes, the company takes it super-seriously if – like, we promise you we’re not going to be the noisy thing stealing your staff’s time. And if we are, then we’re breaking our promises, and you should not renew. And so yeah, we all react pretty quickly.

It might be a naive question to ask this, but how are these attackers getting into the network? Like, where’s the holes at? Is it social engineering? Is it bad hardware? What is it?

It’s, again, a super-neat question. The short version of the answer is we don’t have to care. And again, that’s one of the benefits of Canary, is the assumption that they’ll get in with whatever the attack of today is. So they social-engineered Bob and they’re using his machine, versus you’re up against Mossad and they’re actually in the firmware of your Yeti microphone. They popped out on your network.

No way…!

[laughs]

The point is, it doesn’t matter to us, because now they’re there and they want to do stuff. And so traditionally, security tools have tried to preempt all of these attacks, and there’s always the next attack, right? They’re coming in via this, they’re coming in via phishing, they’re coming in via a new thing… But once they did, there’s a core set of things they have to do. They have to look around for stuff, they have to grab stuff. So we often say, we’re the stupidest product on the floor at RSA. We do what we say on the tin and we work. But it’s that simplicity that people can then rely on.

It’s uncanny how genius this is really, because what you’ve said essentially, to repeat your words, is you don’t care how they get it in the network, but there’s a particular set of things that every attacker does, and you bank on that happening. And you watch for it, and you masquerade as necessary in the network to attract, essentially. And then I’m sure you log, right? Once that happens, you get that authentication, who are they trying to be etc.

[36:00] Exactly. At that point we’ll push out – like, your console will get an alert, we’ll push an alert to Slack, to Teams, to your SIM, to your SMS, to your email, however you want to do that. But our pitch is one alert, when it matters. You should know that stuff’s happening, and you should get that clean message.

But yeah, we were lucky with lots of that stuff… I think we started off thinking this is a good idea, and as we worked on it more and more, some of those things kind of fell in our lap. But so far, it’s worked really well for us. And at this point, empirically, it just works; like, other than canary.love, we get emails at least a few times a week from customers going, “Yup, just caught our pentesters.” Or we get pings from pentesters saying “This stuff makes me sad.”

“This stuff makes me sad…” [laughs]

So just timing-wise… Because literally last week - there’s an Australian podcast called Risky Business. And the co-host is a pentester for years, and he not in a sponsored slot gave this whole talk about “Yeah, this stuff would catch him, because this is what he does, and that’s just how it works.” So yeah, we think it’s good. We see a future where everyone should be running at least some canaries on their network.

Gotcha. And since it’s so set it and forget it, where it can be almost forgotten until it’s necessary to be remembered… You mentioned all these different ways you can alert out; is there ever a time whenever those credentials have changed over time because of the set it and forget it that your alert actually goes unheard?

So when we started, like most engineers, I ended up being the non-engineer, and we were just a team full of engineering. And one of the things we realized is exactly that - what if somebody buys you and never installs you? Or what if somebody buys you, and got the message saying “Hey, someone just logged into SSH, and logged into RDP, and you never checked that message”? And so now we’ve got a tiny customer success team who literally do not try to upsell you, but exist almost as a mini sock and they’ll pick up an alert like that, they’ll reach out to you and say “Hey, this looks serious. Do you know that this thing happened? Is someone aware of it? Are you picking up on it?” and then we’ll build tools on the backend as we grow to make sure that even though that team is tiny, with three people, they can manage thousands of customers.

But all the time – like, we’ve spoken to customers who’ve told us… So when you buy canaries, by default it’ll email you and send you a text message. And we’ve got CTOs who tell us “I still get text messages from Canary, because you don’t spam us, ever.” So I think the trick there is to keep that promise that says “When we send you an alert, it probably matters.” And if we can show that that’s true, then people don’t farm us off six levels down.

What’s the footprint that you can masquerade as then? Is it a pretty large footprint? And how do you keep up with masquerading well?

Yeah. So when we started, when we shipped version one - we just did three, we call them personalities. A Cisco personality – or actually it was a switch, a Linux box and a Windows box. And version one just had that, and it was already useful. And today we’ve got dozens and dozens of them. So you can say JBoss server, I’m a Windows 2016 box, I’m a Windows XP box, I’m a macOS machine… Way down to saying like “I’m SCADA equipment. I’m a Siemens PLC.” And if you say you are a Siemens PLC, you can talk good Modbus. Like, if somebody thought they were talking Modbus to you, you’d respond in Modbus. And part of our team, that’s what we’ll do. We’ll say “Okay, we should build a software server. We should build a SolarWinds admin panel.” And we build those. And we have some customers, like a large retailer who will buy us and say “Listen, we’d like you to look like our point of sale system, so that we could do this.” And we’ll build those personalities for them.

[40:26] At this point, it’s pretty easy for us, because we’ve got this archive almost of machine parts. And when you deploy your personality, we really stress - I think I’ve said it dozens of times on this call already - we really stress that you should be able to say “Make this a DiskStation NAS and step away”, and it does everything. It creates the file share, it creates good names for you… We used a little bit of ChatGPT with our last install, where you can say “I’m in aerospace”, and it’ll create aerospacy files in aerospacy folders for you. You can say you’re in finance, and it’ll create that.

And so the default should just work, but if you want to mess with it, you can say “I want to run an NGINX web server, but change the header to this and upload my own certs, and actually on Port 1234 I want to run my own TCP service. When someone connects, say hello. And if they say hello back, log it.” So our watch words there have been that the default should be trivial, and anything else should be possible. So people can even customize their own personalities if they wanted to.

It sounds like you guys have thought it all… Certainly you didn’t think of it all at the start, but I’m curious about the start, because you have this perspective of the world with bootstrapping, VC funding, how to actually do this… And it sounds like you had a consultancy kind of help you bootstrap the product. Can you tell us kind of the story of how the product came together, and how much effort was there upfront before you started making these amazing inbound sales?

So part of it, or a big part of it was informed by our previous gig, where we were pentesters. And so we had a really good pentesting business from 2002 to 2010. I think we spoke at almost every Black Hat there was. And so again, small South African company, but we got to spread our wings internationally by doing research that could get shared like at Black Hat and DEVCON. And that also gave us a good amount of exposure.

So when I left that – so we sold that company in 2007. And more than anything, I wanted to build a company that was not tied to headcount again. Because pentesting is great, but just based on how many hours of pentesting you can sell. And so I wanted a product company, but didn’t know what the product would be. And so the plan was that I’d speak to a few customers and build a product for them that I could then resell to other people. And we tried a few products before Canary tried a “use this to phish you company” type product, which now has become a cottage industry; there’s tons of people doing that type of business. And then we tried out another product that didn’t particularly take off… And then Canary happened almost by accident, because I was trying to help a company, a really big media organization that was being hacked left, right and center. And when I visited them, we told them “Hey, you should take all the old machines that are lying around, get your intern to just put honeypot software on it, and drop these widely. It’ll be good experience for the intern, and you will get insight into where your real fires are.”

[44:07] And the next time I visited them, I said “Hey, how’s that thing going? Are we getting insights?” and they hadn’t gotten around to doing it. And the next time I visited them, they hadn’t gotten around to doing it… And so we said “There’s something here. We should make this so that it’s easy enough that even those people would actually do it.

There’s actually an interesting story with that, because we drew up the specs and we started building it, and I pinged – I think it was 12 of our previous customers; so people who used us for pentesting and trusted us. We pinged 12 of them and said “Listen, if we built this honeypot and made it quick to deploy, would you buy it?” And from the 12 we pinged, 10 of them said “No, we can do our own honeypots. We won’t buy this.” And it’s one of those interesting things that in retrospect sound heroic, but I thought most of them were wrong… Because from experience, almost everyone intellectually knows honey pots are a good idea, but almost nobody uses them… Because life just happens, and you don’t do it.

So when someone says “Would you pay for this?”, you go “We can do that. Why do we need to?” And so we bought version one anyway, and… There’s pictures of it, but the hardware that we wrapped it in was super-janky, because we 3D printed the boxes. And we made 12 of them, and we sent them out to these customers. Some really good names, like unicorns currently in the Valley. And then all of them came back and said “For 5k, we’d buy that.” And from those 12, eight bought, most of them are still customers…

And then what we were really lucky about is we got to grow the company and the product as sales grew. And I fully admit that that stuff needs super-fortuitous timing. But the early customers who bought version one - it had a lot of rough edges; and it was still useful, and they tolerated those rough edges while we got better. And it allowed us to hire more people, get better.

Today we’ve got people working for us who are way smarter than us, and so it allows us to start tackling hairier problems that we didn’t have the bandwidth to tackle initially. But I think there’s an important lesson that lots of founders get wrong… And that’s that you almost need to earn the right to work on the nicer problems. Initially, you’ve got to work on some problems that seem pretty mundane, but you’ve got to get it across the line for the customer. And if you solve those, and if they buy you, you get to solve other more interesting problems. And so far, we’ve managed to keep that balance right, and it’s worked well.

It’s interesting that you had that experience. In most startup or indie hacker threads that I read about people trying to do lean startup kind of things, where you’re asking people “Would you buy this?”, or you’re setting up a fake page that they would sign up for… The signal is usually the opposite. Like, they would say “We would buy it”, and especially if they know you they’ll say they’ll buy it, because they want to support you. But then you go build the thing, and then it finds out “Actually, no.” When it comes time to swipe the credit card, they won’t buy it. And yours is like the opposite. They said no, but then they bought. It’s interesting.

[47:37] Yeah. So I’ll tell you, that still becomes a problem, because after a little while – like, in year one we did a few thousands of sales… But I was horribly terrified that people were only buying because they liked me. We were pen testers for a long time, and researchers for a long time, and we had a good reputation… And initially our price tag was 5k. And almost anyone - they can find 5k. And so I was really worried in year one that people were just buying because they liked me, or liked us, and that the product wouldn’t stand the test; like, would they renew?

And I think one of the things that served us really well and continues to serve us is an almost neverending paranoia about like “Are we doing enough to justify that people are actually paying us?” And it might just be because we were so surprised that anyone would pay us… But to a person in the company, we still react with our hair on fire when we drop a ball. It’s like, “They’re paying us all this money, and we did that? Like, that just can’t be right.” And I think it creates the right type of panic.

I know lots of people hate it, because they remember with fondness a time when they used to buy their software outright… But we charge every year, right? So every year, people pay us the same amount. And I think in some ways that creates a really strong positive incentive for the vendor to keep doing their job. Because if we don’t show value, then people don’t renew. And so we don’t get to sit on our laurels, because next year we just won’t make that money. And so it kind of forces us to make sure we’re still keeping up promises and still adding value.

Yeah. It must be really hard to resist the urge to add big new features when you have that annual contract, right? The adding value - usually, you want to add some value, not just continue to produce the same amount of value.

It’s interesting, early on it was harder… So we’re adding stuff all the time. I think like most software or startup tales, there’s I think what’s called the genius of the ‘and’ as opposed to the tyranny of the ‘or’. Like, you’ve still got to be adding stuff, because there’s more value to be gotten.

But early on, you certainly are. In your early days, if you have a strong opinion on not throwing the kitchen sink at the product, you could be confused as just being lazy. You’re not feature-complete. Like, don’t tell me you’re a minimalist; you just don’t got stuff. And so early on, there were like five or six funded companies that started in the same space as we were. And I was worried about them, right? All of them raised $30 million… And typically, what most people like that do is they just pile on named features, like “We support this, and we support that< and we support XML, and we support…” Like, pick a standard and they make sure that they’ve got that logo on their site. And we’d get people then saying, “Hey, do you cover this? Do you follow this taxonomy?” And we were like “No, we don’t think that’s useful, and here’s why.” And interestingly, today almost all those folks have exited. They managed to raise another round, they went to 60 million, and then they either pivoted out or they folded. And as time went on, and as more people start taking us seriously, you get a little more credibility to be able to say “I know, this thing’s popular, but we don’t think that’s the right way to do it. If you want to do it, there’s other things you can do, but here’s why we do what we do.” And I think you try to do the right thing, and you sweat all those details… And sometimes you’ll get it wrong, and then you’ve got to figure it out and put it into the product. But mostly, we’ve been pretty good with those calls.

[51:56] And I’ll tell you a stupid thing that we blogged about a little while back… But at some point, like most companies, we went for a visual refresh. So we wanted to update our frontend JS, and we went for ok. When we built version one, our graphics skills were terrible. Now we’re better let’s make v2 pretty. And we worked on it forever, and we trialed it with our first customers, and they liked it… And literally, the week before we released – so we use FreshBooks, or we were using FreshBooks internally. FreshBooks mailed us to say they have gone through a front end change. And our reaction was “Damnit!” Because I don’t want to learn FreshBooks’ new front end. I want it to be the same as it was, because FreshBooks is not my life. I just want them to do stuff. And we had this discussion internally that said “Are we doing that to everyone who’s just been using us for three years?” Because almost a type of vanity, like “We want this new thing”, they just want to forget about us… And so we scrapped that whole thing. We still did a look change, but we made sure it was super-close to the last thing. We didn’t break away from usage patterns that people had. We gave people a way to slowly go through it. So we try to be thoughtful about that sort of stuff. Like, to add new things, but not gratuitously.

Yeah. Adam, he’s speaking directly to you here, isn’t he?

I’m over there nodding because - yeah, we use FreshBooks, and similar… Like, I’ve been using FreshBooks for pretty much ever. Just forever. So long that like it’s like version one interface for me. And when they told us that, I punted so long, to the point where they were like “You have to move to this new thing, because we’re just done maintaining the old thing.” And I fought with them on the phone, basically.

It’s so interesting, because in that story you see exactly that thing; everyone knows this experience of using something, and really not wanting it to upgrade. But everyone thinks that products should keep upgrading to stay fresh… When realistically, most users are like “Listen, I don’t want that. I want you to just work.” And there is a sweet spot where you can add functionality, and add stuff, if you’re mindful of “I’m giving you new potential tools without changing the way you do stuff.”

And they keep making it more expensive too, by doing these things. Like, there’s just so many things in FreshBooks that we don’t even use… And I’m like “You’re the best at this one thing we really need, and everybody else pretty much sucks in comparison”, which is the good thing about FreshBooks… But everything else they offer, I’m like “I don’t want it, nor do we need it.” And so we have to pay way more than I think we should ever should have to for what FreshBooks gives us. And I love their software. I’m happy to pay for good software. I’m not being cheap by any means. But we’re like, it’s more expensive than necessary because they keep layering on these features.

I am so super with you. And I’ll tell you, again, just because this plays perfectly into our thing… So we’ve been running Canary now for eight years, and we’ve never increased oil prices, ever. A big part of that comes from the same thing - we picked a fair price when we started, and the company is profitable, and we’re doing well, and we don’t have to send a price increase all the time. And we’ll often have people say – like, those Canary tokens that we give out, literally millions of people use them. And you almost never talk to a VC, or a finance person, but there’s so much value there. Why are you leaving that money on the table? As if leaving money on the table is a horrible thing to do. For us, it’s like “Well, we’re doing really well with our other product. And this stuff gets to help people who don’t pay us… And we get goodwill, and we get people become aware of us.” If we’re recruiting, I get to say to a young student, “You can go write bank interfaces for First National Bank, or you can work with us on tokens, which just got 3 million users in December.” It’s immediately attractive to them.

[56:16] And so we get all these benefits, but we don’t have to extract every dollar from every customer. And there’s an amount of user hostility that we’ve come to tolerate from lots of our products. And we just don’t think there has to be – and again, we’re not complete hippies. I want canaries everywhere. I think they’re useful. I want to beat all the other VC-backed companies, because I think our products are better. I just don’t think it has to be done at a user’s expense.

That’s good thoughts. That’s wise. I mean, most people would think that you should, as you’ve said, extract every dollar from the customer; not because you’re greedy, but because that’s what capitalism does, it’s what a business does. Businesses are meant to make money, so why would you leave money on the table when it could be made and used and invested to build out your business and do more things? But that’s kind of like – that’s freshbooksing it.

I mean, you have that luxury, right? Because you don’t have anybody to answer to, do you?

So I think that’s certainly a part of it. I think when you’ve taken investments - and particularly VC investments - there’s a growth rate… And what’s interesting is, we’ve shown good growth. We get VCs pinging us all the time, because we track really well as a VC-backed company. We just haven’t done it with VC money. And again, I think what’s interesting is when we started, we made lots of these choices because that’s the sort of company we’d like to interact with. But today, they also just make good business sense.

When COVID hit, I was terrified, because we’d see headlines of our customers laying off huge numbers of their security staff. And I was worried, like, sooner or later, that’s got to cut into our sales. Like, they just laid off 60% of their stuff. And they’d renew us at full, at all the their canaries again. And many of them told us, they’re like “Hey listen, you folks are so affordable. We’re not throwing this out. Like, if we’ve got one security person, he’s managing the canaries that are there.” And so in part, us not being crazy expensive and making sure we always add value meant that when people were doing cuts, we just weren’t the first thing that got cut. It just worked for us.

And then, to crazy extremes, again during COVID - we had a handful of customers ping us and say “Listen, we are on the verge of going out of business. We love you guys, but we can’t keep this.” And for lots of them, we said “Okay, we’ll stick around with you. Let’s chat again in a year.” And most of them who survived came back a year later and said “Hey, we’re back. We’ll pick the subscription up. It’s all good.” And for the most part, they’re customers for life now. They’re like “That was great.” And again, for us it’s not crazy altruism. It just makes good business sense. Like, those people really love you. They’re just going through a really bad time. And post-COVID, they were back.

Well, so many people get into business, and I don’t know what really makes this happen, but they kind of get unkind. They don’t make kind decisions. They don’t have grace and forgiveness in scenarios like that, where – just treat people with kindness. Sometimes that doesn’t go very far though, because you might give somebody kindness and then you get abused. And I get that. I totally get both sides of the equation. But Jerod and I are the same way, the way we operate this business, Changelog Media. We’re so kind with folks, and we’re so forgiving… And we love the relational aspect of every brand we get to work with… And the ones that aren’t in that relational aspect just don’t stick around long, because it’s just not how our DNA is operated…

[01:00:20.08] It’s too transactional.

Yeah, it’s just too transactional for how we operate as a business. And we’re here for the long-term, in the trenches, to help not just our brands we work with, but the people listening to this show right now; we vet everything like that, and we care.

Right.

And sometimes we get the short end of the stick because of that… But more often than not, it works out.

I think so. I think in the fullness of time, that’s how – and look, again, one of the things I often say is the thing that we’ve been most lucky with… And we’ve been lucky with lots and lots of things. But probably the biggest is, from my previous company to this one, we did things our way, and the market rewarded us. And I know lots of really good people who’ve done the right things, and the market kicked them in the teeth. And so they end up learning “It doesn’t matter what you do, the market is gonna kick you in the teeth.”

We’ve just been lucky because we were – at my previous company, it was like “If we work really hard on this research, we’ll get to talk at Black Hat.” And we did. “And if we talk at Black Hat consistently, we will become international trainers.” And we did. And with things, it was like “If we truly add value, people will appreciate it.” We’ve dropped balls, right? Like, early on, I remember with our early deployments, we’d have canaries deployed in the wild, and canaries were dying. This was in year two. And we were using SD cards for disk storage… And it turned out that our SD cards had a fault in them. So like 200 canaries in the wild died. At that point, it’s like the worst thing ever. People trusted us, they bought the stuff, and suddenly there’s no disk on them. And at that point, we just worked like hell, we got new units out, we made sure that would never happen again… And we said to those customers “This is what happened. This is how we’re making sure it’ll never happen again. Thanks for trusting us.” And they did. And we got past that.

And yeah, I think there’s room for kindness, and if you’re lucky - and it sounds like you folks have - you get to build an org where the org then holds you to that. So that’s what people who join us now sign up for. They want to work in an environment like that. For engineers, what’s really important to us is the craft of what we build. We want to build stuff we’re proud of. We want to build stuff that customers really like. And so instead of building a company that’s trying to grab every dollar, and optimize for everything we can grab, we optimize for “Can we really nail this problem? Can we do this thing so nicely that everyone goes “That’s smart”? And so then we start to attract those sorts of people. And hopefully, that becomes your flywheel, and you just get more and more of those people. And so far, it’s working well for us.

On the dead canary front, a couple of thoughts… The first one is it seems like your move away from hardware and towards software makes that less of an issue…

So interestingly, hardware canaries still sell really well.

People like a device.

Yeah. And there are lots of places where the device still just makes sense. So the one example that I mentioned earlier - people doing an acquisition, and they just say “Look, we’re not going to get to taking in that network for another six months. But today, we can just have you ship five canaries there. Just ship the hardware, and someone there will plug them in and they’ll work.”

[01:03:59.23] So to the other part of that question, we certainly had to learn lots of stuff along the way. Supply chain stuff that we hadn’t gotten a hold off, shipping hardware… Version one of the hardware was truly ugly, even past the 3D printed one. And there’s this really good [unintelligible 01:04:15.14] who’s this crazy hardware genius… He had a blog post at some point that said “A message to all startups: you are not Apple.” And the thing is, when you’re making hardware, everyone knows how pretty Apple devices are. I know how pretty my iPhone is; I don’t want to ship something that looks junk. But you haven’t earned the right to make those beautiful devices yet. We had to sell our first few hundreds of these ugly things. And we had to make sure that it was functional enough that it was still useful enough to add value.

And today I love all canaries. They are beautiful, they’re well designed… We’ve just changed the boxes that they ship in. And again, we’ve spent crazy amounts of time making sure that they are a lovely experience for people opening them. But again, I think it’s a tough line of like having to earn that right as you go, if you bootstrap. I think if you raise a bunch of money, then you can aim at lots of that stuff on day one. But I think that brings a whole class of problems for people, too.

When you go to the generated quote though, it says “Five beautiful thinkethsts –” Gosh…

Thinksts.

Thinksts. I’m so sorry…

No worries.

It’s just stuck in my brain. I’m the fool here. Okay? I’m gonna admit that. These five beautiful devices; so you’re saying, basically, in your question, Jerod, they’re not a hardware company? Or you’re just hardware because you have to be.

Well, he was just saying that they have software canaries now that they can deploy, which I think would be a lot easier to deploy in terms of just logistics.

Yeah.

Well, the simplicity really is like you ship them the device, they – let me assume how you would deploy this thing. You plug it in, literally, into the wall to power it up, and then you put an Ethernet cable into it, hopefully that goes back to the switch somewhere; it DHCPs back to the primary, it gets an IP address, and you have a console that manages it. That’s it.

Boom.

Yup. In fact, just for the geeky listeners, I’ll tell you a little more… So when you plug it in – so it’s cryptographically paired with your console up in AWS. So your Canary is tied to that. And when it boots, all communication actually happens over DNS. And so if you take it and plug it into some network, as long as it can resolve DNS on that network - so not even port 53 going outbound - like, it can talk to your internal DNS server - it will get a message out to the console via DNS saying “I’m now awake. Do you want to give me a new profile?” And then on the console, you can say “Yes, I want you to be a Cisco router.” It’ll get that message when an alert happens. And again, we’ve built this whole communication channel on top of encrypted DNS, which is something that most users never think about. But the reason we’ve done that is if someone’s plugging these Canaries in on a complex network, we don’t now want them to have to open holes in networks and firewalls so that these things can communicate. You plug it in on your network, and if it can talk DNS, it just works.

How do you get all that done via DNS?

[01:07:30.00] It’s pretty cool. And here’s a funny story to it. In about 2007, one of the talks that we did at Black Hat was on a tool we built that allowed you to steal information via SQL injection. So SQL injection attacks, I’m sure most of your listeners have heard of. And so we built this tool where as long as you could get SQL injection going, this tool would allow you to pull data through easily, and it could do it just via a SQL injection attack that just had timing attacks, or DNS attacks, or all of that stuff. And so when we built version one of Canary, the first network that we took it to do a test on, in fact the network we had asked the intern to build honey pots and they never did - I went there to tell them “Hey, try this”, and you see the problem. It’s “Hey, this is not gonna get out. They’re gonna have to talk to networking to allow this to get out.” And so I went back and said “No, what we’re going to do is take our DNS channel from that research talk that we did, and we’re going to make canaries communicate with the console via that DNS channel.”

And so we’ve hyperoptimized that, to the point where – remember, our promise is you buy these and forget about them. And we put out new hardware versions almost four times a year. And so if you buy Canary in year one, and you’ve just got it sitting in some basement somewhere, today it’s running Canary current, and it’s pulling those updates just via DNS. And you never have to think about it. You just get an email saying “Your Canary can now do these things also.” It’s fine if you leave it, it’s still running what you had it running, but you’ve now got the capability to do these other things. And all of that is just via DNS, and customers never have to think about it.

I don’t know the DNS protocol very well. I know it’s UDP, so it’s stateless. But you can open up a DNS, and you can just like send stuff over port 53, or whatever it is?

No, so the easiest or shortest way – and obviously, it needs a whole bunch of optimization, but in the easiest version you think about it as you are our Canary and we’d now tell you “Okay, you should go get your update”, and you’d send a request going “Hey update.myhash.canary.tools”, and I then respond to you with “Okay, the answer is hash.hash.hash ask me again.” And you’d go “Okay, ask me again” and I’d go “hash.hash.hash” and on your end you’d assemble all of that and say “Okay, it’s now a thing.”

So it sounds really slow… [laughs]

So interestingly, for a Canary to give an alert, it’s really tiny. You get to push that all out, you get that. But if you were doing an update of the sort that we would do four times a year, you’d basically get a message saying “Hey, your Canary is updating” if you looked at it. It would run for about a day, and then it would be updated. And even there, I think the benefit of being practitioners, and again, I think of really caring comes in. So with version one, or for the first few versions, we’d have, like everyone else, “Your Canary is now 3.2.x”, or 2.9. And at some point we’re like “Listen, users don’t care.” Like, if you were user, you’re either up to date, or you not. And so that’s what our version numbers according to customers now is. Your Canary either says it’s up to date, or it’s not. And if it’s not, you hit the little button, and it’ll request an update, and it’ll come down. But other than that, customers shouldn’t have to care. Like, I don’t care what version of Chrome I’m running, I just want to know that I’m not running something old.

That’s really cool. So all communications from the Canary go over DNS.

Over encrypted DNS. They go to console that way. Yup. And at this point – we’ve been doing it eight years, so it absolutely works. We’ve hit every edge case, we’ve fixed it, we’ve pushed binary updates multiple times to thousands of devices. Empirically, it just works.

Break: [01:11:57.19]

Tell me about your hardware then. So like you’ve gotta care about hardware at some point, because it does look good, the one I see; it looks nice now. So version one 3D-printed, and later version one not 3D-printed… What’s it like now? Do you care, I suppose, deeply about the hardware?

Yeah, we do. So over time – initially, with version one, almost all our design stuff was done by me. And I’m a technical hacker. I’m not the best design person you’d get. And so I used to do our early stuff in OmniGraffle, or work with an external designer. And now we’ve got a great designer on the team from Canada, and he lives and breathes this stuff, and so he’s doing pretty stuff all the time. We’ve almost got to hold him back just with “No, we’re not going to spend time on this, we’re going to spend time on that.” And again, for me that falls into one of those categories of earning the right to do cooler stuff as time goes on.

What is the hardware? I mean, it seems Raspberry Pi-esque, at least from a footprint… What’s the actual hardware built on?

Yeah. So in there we’ve got a tiny, little daughterboard that we manufacture here in South Africa. You can swap out with any number of small-factor machines inside… But again, it’s something Jerod said that was interesting… If we do a Hyper-V version, a VMware version, or a hardware version, we charge exactly the same for all of them. And so our pitch is that’s not something customers should ever have to care about. So with most of these, they’d be running the equivalent of Pi 4s, with a small daughterboard in there that we have, that drives that little button that you see, that drives some of our other stuff. But again, fundamentally, it’s pretty simple.

So is it built on the Raspberry Pi for that, or is it something –

Yeah, the current versions are.

Okay, so daughterboard on the Raspberry Pi 4, powered via plug into the wall, not POE, right?

Yup. Yup. Not POE.

And then all you’ve got is a barrel port plug, it seems, based on pictures, and then a single LAN port.

Yup, exactly right.

Okay. A reset button…

Yeah. So the little LED that you see is actually also a button. So you can boot and hold on that button, which would put it into configuration mode. And fundamentally, we want that to be dead simple, so there’s nothing you can do on it, that’s wrong. You can hold on that button to reboot it… And the way we run the service is if your hardware device, if you’ve run over one with a truck, you mail support and we’ll just send you another one. Like, the point is that you should never have to think about it. And so you just get another device, and it just magically shows up.

So when you were alluding to the hardware supply chain challenges from before, obviously the rest of the world Raspberry Pi’s in the last couple years have been in high demand.

So it’s interesting… The SD card issue was one that took us by surprise, more even then Pi’s in demand. Like, we didn’t realize the different– so essentially, what happened is SanDisk had a speed wobble at some point, which I guess normal people don’t have to care about. And so SD cards were in short supply. And we went out and bought a whole bunch of SD cards from wherever we could to shore up our supply. And it turns out there’s just tons and tons of fake SD cards on the market.

No way…

[01:17:40.22] So in SanDisk packages, but just poor-quality SD cards. And when we realized that we did a bunch of testing – because you can get SD cards for $20, ranging up to $200. And we were like “Well, okay, if the $200 one is going to stop us having this problem, let’s find out and we can plan around it.” It turns out you just need legit, good-quality SD cards, but you can track them, you can put in quality control to make sure the batches you buy aren’t going to fail after 300 reads, or 300 writes. But again, that’s the sort of stuff we had to figure out as we went.

So all of your hardware is powered by then an SD card, not the optional…

NVRAM.

Yeah, exactly.

And so what you end up doing a lot - and again, it’s something that you don’t know early on - is you start building failsafes that you can in software. So wherever you can for that stuff, you’ll start adding watchdogs, you’ll start adding more robustness. And because we’ve got a communication channel between the client and the server all the time, we can start having the client say “Hey, send me my config again. I’m in trouble”, that sort of stuff. So yeah, you ended up building robustness in in software.

So are you building your stack on top of the Raspberry Pi OS, or is it a different image? Give me from the hardware up. What do you do?

Yeah, so we’ve got to have our own custom kernel, because we’re doing packet mangling… So we need to be able to fake out that our operating system is actually Cisco IOS. And so we have our own hardened image that goes on there, that we will customize, that we will maintain, and we will maintain that internally. And fundamentally, we then run a master service that runs all of the fake services that the service claims to run. So we have a hardened base to make sure that we don’t get caught out that way, and then we have a system that fakes out the rest of the services, fundamentally. And then you’ve got to have a component that’s communicating with the console; we piece those together. And then the console becomes its own software, because that’s gonna handle alerts, and integrations, and all of that stuff. But those become the two big pieces of it.

So now that I know more about your hardware and your software, let me suggest an attack.

Sure.

Let me hypothetical with you, and you tell me how your system would react. The attack is an inside job. I work within, I know that we run Canary, I know where they’re all at… And I either go unplug them, or I decide to pull out your disks. And then I submit my attack, because now I know that the guards are not there, and I can go… And I’m part of the security team. Or maybe I know the security team. My friend – I’m in finance, and my friend is in security, and he has a loud mouth, and we drink a lot together; whatever it might be.

Right…

I now know how to locate the Canaries, either dismantle them by either pulling out the SD card - because maybe it’s accessible, maybe it isn’t, via the hardware… Maybe it’s inside the actual shell and I’ve gotta unscrew it, or unplug it. So I take down all the Canaries. What happens?

So if a Canary is down, by default for eight minutes, but it’s configurable, it’ll reach out to you and tell you “Hey, listen, I’ve just been turned off, and that wasn’t part of your plan.” And so in some instances, you’ll be like “Okay, that’s because that section has just powered down. We know it.” In fact, someone tweeted yesterday that it’s the best-quality indicator of when your network is down, because you will get an SMS saying “Canary 52 is now down.” But the simple thing there is a Canary going offline is a surprise. And so you will get an alert telling you this Canary that should have been up isn’t reporting in any more, you should go figure out why.

But if you know are all the Canaries are, you’re just not going to touch them. Like, they’re Canaries; you’re gonna – inside jobs… I mean, it’s like having physical access.

[01:21:52.13] True… But I’m in finance in this scenario. I’m in finance.

Okay… [laughter]

I don’t know what they’re configured as… I’m just saying, I’m not in – I was trying to throw a…

Yeah, I’ll tell you two interesting versions of that. The one is - and I feel strongly on this - is one of the original sins of the security industry is them promising too much, and trying to be too much. And sometimes people need to be able to say “Yeah, we don’t do that.” Like “Yeah, we wouldn’t catch that.” If you know where all the Canaries are, and you don’t touch them – like, you know where all the tripwires are; that stuff’s not gonna catch you. And I think people should be okay with saying that. And our pitch to try to mitigate against that is that we want to make things that are easy enough to deploy, that a person can deploy it without letting the whole company know “Hey, here’s what I’m doing. We’re doing this Canary rollout.” Like, literally, go plug it in, forget about it. It’s in that corner. It doesn’t need huge shenanigans.

And Canary tokens add trickiness, just because they could be anywhere, and they could be – we’ve given a few talks on Canary tokens, because some of them are really dependent on how tricky the security team wants to be. So some of them are obvious, like that AWS API key that I mentioned. But we’ve got another one, for example, that’s a legit WireGuard endpoint. And so you take your CEOs phone, and you add a WireGuard tunnel on his phone that says “Secret exec network 123.” And you forget about it. And what you’re waiting for is when he gets his phone compromised, when he’s going through customs into China, when that phone gets grabbed, it’s the sort of thing that an attacker who you’re interested in looks at it and goes “I see. I’ll use this endpoint. I’ll check what this is.” And our pitch is, if we can make those things easy enough to do, then security teams can do them.

And so if you take – I know lots of vendors use it, but if you take people having their SolarWinds moment, where attackers have compromised the build server deep in a network, and the only time they find out about it is after the attackers used the build server to build new software that’s been deployed to all of their customers, that sort of attacker who finds AWS credentials on that machine has to try to use them. Because maybe that’s SolarWind’s cloud environment. Or if they find a VPN endpoint on that machine, they’ve got to see what’s at the other end of it. Which means in week one you get notified that a machine nobody should be touching is doing strange things, instead of waiting till you read about it on CNN. And mostly that’s our pitch, is “Do this now, forget about it. It will be good for you.”

So as you look at building, maybe not more features on Canary, but new products or services, one that makes sense, I think, as a follow-up is like mitigation, right? So now we know there’s a problem… Well, our good friends at Thinkst let us know; maybe they can help us fix it.

“Now what?”

Yeah, exactly. “Now what?” I’m sure that’s crossed your mind… You’re nodding your head, so you’ve thought of this… [laughs]

As a service, it’s something that we’ve stayed away from, largely – and there are people who roll incident response, right? Anyone gets in trouble in the world, and they call in Mandiant, or they call in bunches of folks like that. And part of our pitch has been, again, we want to do one thing really well, and we’ll partner with those folks.

[01:25:56.10] So we have bunches of MSSPs, managed service providers, who will take Canary, deploy them at their customers, and have those alerts go to them. And so what they’re getting is they’re already trying to manage all of these customers; they deploy those Canaries at those customers. If something happens on those networks, they get the alerts, they then react to those customers. And for us, it’s a good deal. There’s at least a few MSSPs in the US who have Canaries deployed at every one of their customers. And for us, that’s perfect. We’ll keep making this thing that works well for you; you keep offering that service, and everyone’s better off for it.

Keep it simple, keep it focused. Have you ever had the bug? I’m sure you’ve had lots of people walk up to you with large checks. Have you ever thought “Maybe we should do this one? Maybe we should take some funding and do something bigger?”

[laughs] That’s a super-good question, and super insightful. We have. So we have conversations with lots of VCs who ping us periodically. And in 2019, one of those VCs - probably one of the best-named VCs in the world - pinged to say “Hey, would you do breakfast with this named partner?” And I was like “Of course I’ll do breakfast with that named partner. Are you kidding?” And it was great. Like, they say “Don’t meet your heroes”, but he is every bit as amazing as every talk of his that I heard. And I came back to South Africa, and they phoned me and said “Hey, would I come up and meet the other named partner?” And I did. And they did this thing that said “Hey, here’s why you should take money from us.” And we flirted a little, and my take was “Listen, I’ve got money in the bank. It’s our own money, and I’m really worried that this is how focused product companies lose their focus with this stuff.” And they said “Look, we won’t take a board seat. We’ll give you all this money, we’ll tell people why you’re great.” And we were super-tempted. And again, because it’s super-flattering, right? Like, I’d read about those dudes forever, and they think I’m cool, and they think our company is cool, and they throw barbecues that Obama attends… That stuff is flattering as hell. And we flirted with them for about a year, and decided not to, and I’m still good friends with them. I am officially a scout for them, which means I can invest some of their money in small startups… But we figured we didn’t need it. And yeah, I think we’re better off for not having taken them.

I’ll still listen to everything they say, and read everything they write… I just don’t think that the business needed them. And yeah, at this point – I think VC, almost as a segue, I think that the VC model isn’t super-well suited to building good security companies. I think there are some companies that… I think if you’re trying to build the next social media powerhouse, you should raise VC… And it works well for VCs, because they’ll give money to a bunch of people, and as they see which one makes it through, they can give more money to that one, and then the winner will take it all. But I think in security there’s a side problem that makes that harder. And I think that the VC model kind of muddies the water. And I wish more founders knew that it wasn’t a law of physics that you absolutely had to do it the VC way.

Well, it makes me think back to years and years ago, when 37signals was just starting to take off, and famously, Jason Fried and David Heinemeier Hansson took investment from Jeff Bezos… And it was more like it sounds like what you were being offered; it wasn’t like board seats, control, blah, blah. It was more like “Here’s some money. We would like to be a part of this.” And their stance back then was “We took some money off the table”, or something like this. “We didn’t need to.” They didn’t need to either, according to them.

[01:30:16.11] Exactly.

And that seemed like – I don’t know the history of that, did they buy that back from him, or it’s just the case, but it seems like you could have done that, had your FU money and then just continued along your way.

So it’s interesting, for multiple reasons. One is we based a lot of our stuff, including lots of our company thinking, on the 37signals box early on. Like, opinionated software, all of that stuff; lots of it was informed by early 37signal thinking. And look, for us - it sounds like a terrible thing to say, but we make good money now. For the first few years – but we’ve got a few million in the bank, and it’s not buy an island money, but we keep growing, we’re doing well. We can pay dividends at this point; we pay the company good bonuses based on that sort of stuff.

So again, I think that lots of people have a pretty static view on that path to generating wealth, and it’s largely because VCs were the ones talking about how to build companies. And so I think lots of the literature out there was on doing things like the raise a seed round, raise your next round, keep doing that way. And again, I wouldn’t begrudge it, because I think that’s perfectly fine. But the biggest problem I have with that stuff is that it’s super-distracting, and almost runs in a completely different direction from founders focusing on products.

Ages ago, Paul Graham had this essay where he spoke about the top idea in your mind. And you’ll see how often these days founders who are on that raise money VC route hamster wheel - that’s the top idea in their mind. It’s “How do I raise the next round? How do I talk to analysts so I look good, so I raise the next round? How do I talk to VCs?” Which means almost by definition, the top idea in their mind is not their product. And yeah, I think we are all poorer for that. And I’m surprised that it’s so acceptable. And I know it’s terrible, because every founder in the world secretly thinks he is Steve Jobs… But one of the things that I super-appreciate about Apple today is that we get to see a multi-trillion-dollar company where they care deeply about the product.

One of the jokes [unintelligible 01:32:53.01] find CEOs of companies who can demo their product. In the security world, it’s shockingly rare. When you had Symantec and McAfee as giants, you think the CEO of Symantec is going to sit down and explain how they fight? No. At that point, he allocates capital. And what happens is no matter what you say to the company, the company knows what matters. And so the people in the company are not then optimizing around building the best product they can. They are optimizing around acquisitions, mergers, capital allocation, sales stuff. And I think there just needs to be more focus on the product.

What you’re describing is being grounded, right? Like, if you can demo the product, you’re kind of grounded in what you’re producing. You’re grounded in the value that your employees create, that the things you do are delivering to the market to create that value, and to receive cash value back from that value being executed and delivered. It’s a grounding in your company’s purpose; not chasing the money to a degree, or schmoozing with networks and whatnot to get more capital just for capital’s sake.

[01:34:17.27] Yeah. And again, I’m pretty convinced it’s also the path to winning. I’m pretty convinced… And again, we’ve been lucky so far, but I think the market rewards that stuff. You end up making a good product, and the market hopefully rewards it. [unintelligible 01:34:35.01] we just did the Black Hat conference, where we had a booth… And one of the things I was talking to someone about which was super-interesting is when we do a booth - like, we’ve got this really long blog post out on doing booths and why we think it’s actually good for people. Young me hates it, but booths are super-good for us. We do a booth at RSA, we do a booth at Black Hat, we get to meet all our customers, they come by and chat to us… People we’ve never seen before come and say nice things about our products, other people hear them… But at Black Hat this year something that occurred to me is we have this booth, and I’m there for the full two days… And Marco who’s our CTO is there, and Bradley, who’s one of our other founders… So literally, all of our original founders are there, plus some of our engineers. And so for two days, people are rolling up to us, talking to us, “Hey, I’ve been using you for six years. Hey, I did this”, but that’s surprisingly rare on the showroom floor. Because on the showroom floor, what lots of people have done is they’ve paid a whole bunch of young interns, a whole bunch of college students to say “Scan as many badges as you can. You scan the badge, you then get to spam all these people, trying to sell stuff”, and again, it’s horribly mixed incentives.

For us, the thing is we get to meet our customers, and we get to do demos with new people who might be interested in the product. And it’s so counterintuitive, because if you talk to – with any VC, one of the playbooks that they will tell you is the truth is not in your building. Go out and meet customers. If I told you “You’re on the showroom floor, and you’re going to meet 20,000 of your customers in two days”, why wouldn’t that advice mean that every CEO, CTO and chief product officer is the person on that floor? You’re gonna meet 20,000 of your customers or potential customers; you can talk to them about the product. But it’s just not done. Because fundamentally, what the execs are doing is they’re sitting in a suite somewhere, trying to arrange their next raise, or trying to talk to analysts, or trying to talk to the media. And again, the state of the products in our industry are a reflection of that; mostly, we build terrible products, because people just don’t care enough about them.

Do you think that that’s unique to InfoSec? You mentioned that you think it’s particularly a problem in InfoSec, but it seems like that would be more broad-sweeping, perhaps.

It’s a great question. I think in other verticals - and I clearly can’t speak for all of them, but I think in some verticals, the vertical itself keeps you honest. I think if you have five competing social media companies, the ones that suck are gonna fall away, and the ones that people use get traction. And the thing that InfoSec has that’s unique there is it’s really hard for most customers to tell the difference between good products and bad products. And instead, what they then use as a proxy for judgment is funding.

[01:38:00.25] So companies say “We’re funded by big name X”, and customers then say “Well, you must be okay, because you just got funded.” And you’ll see it; if you check the industry, you will see how many of the press releases are just new funding rounds. “Here’s what we did, we just secured a new funding round.” It’s like, tell us you’ve got new customers; tell us you’ve solved a problem. Don’t tell us that the people who gave you money before gave you money again. And mainly, what happens with that is that becomes a proxy for quality. Customers then buy it. Investors then say, “Well, you’ve got all these customers. I should invest in you for another round.” And what it does is it means that bad products last longer than they should… Which is also not great for VCs, because it now takes them longer to figure out that they’ve backed a product that isn’t sustainable. And that’s why I think that stuff is bad. I think focusing on the product is a quicker route to honesty. Because unless you make something people want, you don’t get to fight another day.

That resonates with me. As I said, I did some penetration testing right out of college, and I was kind of – I went to a few conferences, ShmooCon, Black Hat etc. and I talked to people more of maybe I could work at one of these places, or whatever. And the vibe I got, in general, was like lots of snake oil here; lots of just like sales going on, but not much substance. And I never really liked that field, and so I kind of left the community, so to speak, and went into web development.

It’s largely still that. And there are a few companies now… You’re starting to get more practitioner-led companies. And I think one of the big things that certainly we’re a beneficiary of is that - I’m guessing 15 years ago, even if you made a great product, you couldn’t sell it. You still needed the traditional coin-operated sales team that went out, and strippers and steaks, and all of that stuff. But today, what Slack and GitHub and Box and the empowerment that engineers have - you don’t need that stuff. Literally, we cleared 16 million in ARR without an external sales team, because people will try you, and engineers will try you again, and then they’ll pull you into the org.

And so I think there’s never been a better time for developers, for engineers who’ve been through the idea maze to build their products and give it a shot; it’s possible now. It’s as good a time as any to throw your hat in the ring.

Well, I think that’s a great point to end on. Adam, do you have anything else you want to ask Haroon before we let him go?

One more question, just waiting for the Plus Plus.

Oh, we’re saving it for our Plus Plus people; these are our insiders. Changelog Plus Plus, our paid supporters. So we’ll save that for the post show. For now, we’ll just say - man, thanks for sitting down with us. Thanks for sharing what y’all are up to, your design decisions, your extreme focus, and your willingness to turn down large bags of money, because you already have enough bags of money, and you’re doing just fine, and you’re staying product-focused. That requires discipline, and that’s pretty cool, pretty unique out there, so…

Thanks for having me.

…happy to hear about it.

It was awesome. Thank you for coming.

Thanks, folks.