Local cert management for mere mortals

Hello, hello, hello, and welcome to this episode of Go Time where we’re going to be talking about something that is important, but that is hard to do locally… And we’re talking about sort of local security and certificate management. If you’ve ever tried to do this locally, yourself, and you run into some of the browser warnings around your local certificate not being – something wrong with it, and kind of scary messaging and things of that nature whenever you’re doing local development, then this episode is for you, because we do want secure local development, don’t we…? Joining me today as my guest to talk about this topic, because they are actively working on making this easier for everybody, are Christopher Stolt. Hey, Chris. How are you?

I’m good. Thanks, Johnny.

And also joining is Ben Burkett. I just know I’ve ruined your name just now. It’s Burkert?

That’s it, yup.

Oh, second time. Second time’s a charm. Glad to have you. I’m your host, Johnny Boursiquot. And yeah, it’s been a minute since – I think at least three or four weeks since I’ve done one of these shows, and having Chris and Ben on the show is kind of a return back to more regular hosting for me as well, so I’m also glad to be here.

So the topic of the day - obviously, having a secure local environment. Let’s sort of level-set a little bit and talk about why do you want that. Is it just a matter of production parity? Why would you want to go through the trouble of setting up local certificates and all these things?

Yeah, dev prod parity is really the key reason. The more you can make your development environment look like your production environment, the more likely you are to catch bugs and issues that kind of only crop up in production environments. And those are typically very expensive, costly bugs to deal with, problems to fix… So dev prod parity is one of the things where I think a lot of folks - it was maybe a new idea ten years ago, but I think it’s really caught on to developers, for the most part. But there seems to be this lack of dev prod parity when it comes to encryption, and HTTPS… And so you can certainly run into these bugs when you’re working with encryption, and having a way to surface those bugs and identify those problems earlier in your development cycle leads to just better software quality in general.

[00:05:36.21] Yeah. And I’ll add in - Johnny, as you know, I have a long history in my career doing support, building on support orgs, and we hear time and time again in support “Oh, well, it works in development.” And that’s like the classic thing. But the reality is your development often doesn’t really look like or mimic production on a number of reasons. So really, to that end, and to everything that Ben was saying about that dev prod parity, it’s not about security because you have sensitive data in development; if you do, you’re probably doing something wrong, and you want to stop doing that… Keep that user data out of your dev environment. But really, it’s about that dev prod parity, so that you can catch those issues before you go into production. And then there’s a number of things that you might develop against, that even in development require a properly set up and secure local environment.

That’s actually a good segue, because this kind of problem, although we all know that it’s a hard problem, it’s not trivial, and you must love security to go into this kind of work… Chris, why don’t you start off and kind of give us a little bit of background? What led you here? What led you to solve this kind of problem?

Yeah. Awesome, thanks for asking. So where do I even start? So early, early in my career, I was doing like DevOps type stuff at GE. Huge company, lots of different infrastructure… And back then there weren’t tools around security, so we were hand-rolling everything using OpenSSL at the command line to provision certs. Time would go on, we’d forget about like what certs we’re provisioning, and were provisioned where… Something would expire and we’d have an outage…

Rolling to a couple of jobs later and I’m at Heroku, and this is still happening. And before Letsencrypt came on the scene - and I think we’ll dive into like Letsencrypt and kind of what they’ve done here, which is pretty impressive to help with this… But before they were on the scene, we would be helping customers set up HTTPS on their application around this. There’s a big push from people trying to secure some pages that had sensitive information on them, login pages, pages where you’re viewing PII, and stuff of that nature, to just say “Hey, let’s secure the entire app and split the entire app behind HTTPS.” So there was a big shift to that. And customers would show up to Heroku and they’d upload their certs and it wouldn’t work. And they’d reach out to support, and we would look at it, and there’s a thing called a certificate chain, which chains back up to telling the browser who to trust, and why they can trust this cert… And it didn’t work right. As a support org, this is not really fundamental to Heroku, or the service that we were offering, but we were kind of forced to go in and help the customers. So we wrote a bunch of things - the support team did - that eventually got baked into the Heroku CLI that fixes that cert chain automatically when you upload a new certificate.

So this is just a thing on a bunch of different levels that’s plagued me in my career, time and time again. You know, we’d automate a whole bunch of stuff with Letsencrypt and Acme at Heroku back then, but it was just hard and painful, and I wanted to fix it for everybody. And when Ben – and Ben and I have known each other for a long time. And when he came up and was like “Hey, let’s work on this problem”, I was like “Yeah, let’s solve this for other people. We already know the pain, we feel the pain, we’re there… So let’s just endure that, and then other people don’t have to.”

Yeah, we were talking about sort of – obviously, we’ve both spent some time at Heroku… And certificates was always – you had like a heart palpitation whenever certificates came. [laugh] Things would expire, and you’d be like “Crap! Why don’t we have better tooling and systems around this?” Even some sort of reminder, like “Hey, this thing’s about to expire in 30 days, 90 days”, whatever it is. There was so much tooling built around this stuff, and I came across a lot of that stuff during my time at Heroku… But Ben, I want to hear about what led you to this kind of work.

Yeah. As Chris was saying, I’ve been an IC my entire career, software developer… And certificate issues, particularly internal CAs, and kind of the problems with those, is something I’ve just seen kind of every place I’ve been; the same problem keeps popping up and up. Issues with internal TLS, internal CAs kind of mismanaged, or just causing problems, like you’re saying… You know, certificates expiring, and then everybody having to wake up because one of the various backend services is now down due to an expired certificate.

[00:10:05.28] I remember being at CloudFlare, where they have just an amazing engineering security team, doing some of the most impressive security work. And this is a company with some of the best security engineers in the world, and I still saw issues with internal certificates and outages, and it just made me think “If we can’t do it, and we’ve got these great security engineers, this is certainly a problem that everyone’s experiencing. Why does every software engineer that – why does everyone need to be woken up in the middle of the night with these problems? Why can’t we solve these issues that are interrupting our sleep?”

Yeah, so I was just gonna chime in… I think Ben kind of hit on a thing that’s maybe a question that perhaps some people in the audience might be just thinking right now… It’s like - yeah, so you have the best security engineers at Cloudflare, and we’re still having this problem… So why do these two guys just think that they can go off and solve this problem that no one else has? And I’ll just kind of hit that nail on the head real quick. I think for us, it’s really not that it’s the hardest problem to solve, it’s that it’s a really unsexy problem that people just never cared to spend time on. Certificates tend to be a thing that - they plague you when you have the outage, that sucks, and then it goes away, and you don’t think about it until that cert expires again a year later. And a year later that’s painful again.

So it’s not that [unintelligible 00:11:27.21] everyday thing, so people just kind of put it in the back of their mind, and forget about it. We’ll get into really what Letsencrypt has done to revolutionize this stuff, and it’s awesome… But other than that, there’s not been like big advancements and really a many people putting a lot of effort into really, truly trying to solve this… And we just think that, given the right amount of attention and care, that we can do it. It’s not that it’s the most complicated thing, so we’re not gonna go reinvent the world or something here. We’re just trying to make things take away people’s pain, really.

Bringing some convenience to it. It’s one of those things, like “Why can’t I just set this and forget it? Why do I have to be involved? Why does it have to cause a disruption at a same interval every year? Why is this a problem ongoing?” So that convenience factor.

Absolutely. And the other aspect here too is really that - and this gets back into Ben and I’s - and yours, Johnny - the history at Heroku, right? No one’s taken really a dev tools, like a developer-first, developer-centric approach to this. We haven’t really seen that anywhere where they really put like the developer experience first… And we think really just starting there. Because at the end of the day, it’s developers building these systems and these applications and these services, and they’re the ones that are serving up the certificates… So if we make it really easy for developers to just incorporate this in a way that feels natural to them, we think that that’s going to be a win. So that’s really where we started.

Break: [00:13:00.07]

Let’s walk through the process right of dealing with certificate management. And along the way we’ll sort of highlight what’s different if it’s an internal certificate, versus something that’s external… What is the process? If I’m a developer, I sit down, I’m like “Okay, I need to secure this application. Along the way, I also need to secure my local environment, so I can get dev prod parity”, right? But I just have a business, a SaaS solution that I want to secure. What’s the process like of obtaining a cert? What am I doing?

Sure. I think you can start kind of by drawing a boundary around where internal certificates and internal CAs are different than public certificates and using a public CA, and then where they’re the same. And where that kind of line in the sand is is once you have a certificate on a server, everything’s exactly the same after that point. Your server is going to set up TLS, and accept TLS connections in the exact same way, regardless of whether the certificate is signed by an internal CA, or a publicly-trusted web PKI public CA. And then likewise on the client side, once your client is configured with its set of trusted certificates, kind of referred to as its trust store - once it has those, when it connects to a server or dials out to a server, that connection establishment is exactly the same, regardless of internal certificate, public certificate. So really, it’s about kind of what goes into getting those certificates on servers, and then updating those clients to trust the certificates being presented by servers.

In terms of getting certificates on the servers, these days, kind of post Acme 2015-2016, pretty much all servers public certificates are provisioned via an Acme API interaction with a CA that supports Acme. Acme is an RFC standard for acquiring a certificate from a CA. So there’s Acme now to actually provision a certificate on a server.

In terms of internal TLS, you don’t see as much Acme adoption. You still see a lot of kind of manual or automated certificate provisioning… For example, Kubernetes has a component built in called Cert Manager. Cert Manager is in charge of distributing keys and certificates into containers, so that they can be TLS serviced, so they can accept TLS connections.

And Certificate Manager has a few modes. It can run its own CA, where it’s kind of signing and issuing certificates. It can delegate to a public CA, like Letsencrypt, or one of the other ones… And then you have kind of maybe 30 years of just various tools to solve this problem for internal CAs. At companies there’s things like Dogtag for Java… OpenSSL is like the big one. It’s –

The granddaddy. [laughs]

The granddaddy, exactly. It’s so hard to use… It’s one of the most confusing CLI flag tools. It’s impossible to learn or memorize –

I wish we could put up the GIF of – there’s like a Visual Basic type UI that’s just like endless options… And that’s like the interface of OpenSSL.

[00:18:04.00] Yeah. There’s so much surface area… It’s bananas. And so that’s kind of the – that’s the provisioning side, that’s getting certificates on servers. On the client side… This is where things are kind of nicely handled for everybody on the public side. We have what’s called the Root Trust Programs, and that is the groups of browser creators and operating system creators, companies like Microsoft, Google with Chrome, MacOS… They are kind of curating a set of trusted CAs, and then they’re shipping those into the software that they’re publishing. So your operating system comes with a set of trusted CAs. Your browser does as well. Even your language - some languages will ship their own kind of trusted store set of CAs.

And so when it comes to using a public CA, your HTTP or TLS client probably already comes preconfigured with the set of public certificates, and that kind of whole management of those trust stores is just handled for you by these organizations.

On the internal TLS side, things are a lot hairier. There’s really no – there’s no equivalent organization, or even a nice set of tooling that kind of handles all of this for you. So this is really where a lot of the challenges of internal TLS start showing up, is getting all of your clients to trust the certificates that are being presented by your servers.

Yeah. And if you want to kind of break some of that down real quick… So the trust store - it’s the same trust store when you’re doing internal certs, self-provision, and self-signed certs, or the publicly trusted certs. It’s all the same kind of database, if you will, for lack of a better word right there on what that trust store is. And the only difference is the globally-trusted certs - your systems and your browsers come preconfigured with those globally trusted certs in them. For anything else, you have to kind of manually add that, and that’s where that toil comes in, is manually adding them in, remembering what you added in, being able to go in and audit and/or clean that stuff out if there’s a key compromise somewhere, or anything like that. That’s the hard part.

And on the provisioning piece with Acme, I’ll just come back to Letsencrypt. I mentioned them a couple of times. They’re the entity that got spun out of Mozilla a while back, that decided to solve encryption on the public web… So for all the globally-trusted stuff - browser, direct to server - they created Acme, which is that protocol that Ben was mentioning, to automate a lot of this. But that doesn’t work right for local encryption, or even for backend encryption, because you have to do this whole DNS dance… And that’s some of the complicated things that don’t work well… Trying to mess with the DNS when all you want is a cert locally, in your dev environment, to get that dev prod parity, or to program against HTTP/2, or something like that; it’s going to be a bit of a showstopper.

Is that what’s sort of primarily driving all the bespoke things that people are trying to do internally to try and solve this problem? There’s no one way to approach it; there’s no standard that meets everybody’s needs, right? Is that the driver?

Yeah. That, and kind of like a handful of reasons. A lot of people do use public CAs for kind of internal traffic and internal TLS traffic…

It’s easier. [laughs]

It’s easier… In some respects, it’s absolutely easier. In other respects it can be more challenging. For example, the API limits on these public CAs is pretty low. If you’re not careful, it’s easy to kind of blow past your budget on certificates that you can provision for one domain… And the Acme workflow process - to actually perform an Acme challenge, you have to prove to this public CA that you own a domain. You do that typically by publishing a DNS record… And so that can be a slow feedback loop.

It takes time to propagate all that, right?

[00:22:01.05] Exactly. And that’s too slow for, say, a container boot process. And so you can mostly automate the process, but then you have to kind of hand-tweak some things here and there. So using a public CA for internal traffic - it’s great, because it handles your clients for you. But the downside – there are downsides, kind of issues you can run into with the DNS stuff and with other stuff.

Okay, so it sounds like depending on what you need internally, there is no one-size-fits-all. The public side’s a little easier to deal with, but for the internal stuff, especially for the example you gave that was a container orchestration use case, where you’re spinning up containers and everything that you need; services, service communication, you need TLS… If we bring this back to sort of my – I’m sitting down, I need to do development, right? My developer experience, if I want to sort of have a secure, prod-like environment to do my development, then it sounds like, depending on where I work, there could be different solutions for the internal certificate management. And I don’t know about you, but this sounds like a handful to me. Like, I don’t want to do this. I don’t want to do this. There’s just too many things that could go wrong. And we’re talking security here… I’ve been doing this for a while, but I’m not a security expert. I just want to hand this off to something, right? So y’all have been working on trying to solve this problem. Tell me a little bit more about what you’ve come up with so far.

Yeah. Well, first off I’d say I don’t think you’re unique, or – your experiences, I would say, lots of developers out there are in the same situation. And I certainly was annoyed, and this was a problem in my places I’ve worked… And my typical solution was to just turn off TLS locally; just figure out a way to disable it, and really not get any of the dev prod parity. And I think that’s kind of typically what developers do, is they figure out a way to bypass TLS in their local development. And that’s not so great for dev prod parity. It also can be a challenge - you know, if I’m a Go developer, and I’m writing gRPC services, that can actually impact how I develop locally.

For example, gRPC has features like bidirectional streaming. The client can create a connection to a server, and then while the server is responding, the client can keep kind of pushing parts of the request over to it. So that bidirectional streaming depends on HTTP/2. And HTTP/2 doesn’t require TLS, but it kind of uses TLS by default. And so there’s ways of working around the lack – if you lack TLS, you can still do HTTP/2, but you have to kind of do this funky, complicated upgrade procedure.

And then your development, if you’re doing all these things in development, you’re further drifting from what your production environment looks like. So it’d just be better if you as a Go developer building backend services with gRPC - you want to use gRPC streaming - it would just be better if it was easy to use TLS everywhere: development, staging production. And yeah, that’s one of the – besides the gains in security, there’s actually gains as an application developer to using TLS everywhere.

Yeah. I can just chime in real quick with a… So what we’ve done at Anchor - we have two things right now, which is Anchor, the core product that we’ve been building, and then this new thing that we launched recently, lcl.host. And so the core Anchor product is really meant to address the backend encryption and internal TLS setup, maintenance, distribution, all of that.

[00:26:01.28] So really to address all of the issues with using the public, globally-trusted certs right out of the gate, so you don’t have to do a DNS dance, there’s ways that we can work around that. And then we don’t have to worry about rate limiting and other things of that nature, because we know that hey – we know who these people are, and some of the global certificate providers, they’re just getting any person off the internet requesting the certificate [unintelligible 00:26:25.11] so we actually know who you are, and we can say “Yup, you have a big infrastructure. We’re cool with this. We can change rate limits”, work around all of that. And then also address in a very elegant way all of the things that crop up with internal CAs, all the things that Ben’s experienced at CloudFlare, and all of these other places… And make those all smooth, and all of that.

And along the way of building that, what we realized and what we kinda came to learn was “Hey, this [unintelligible 00:26:55.00] is important.” And what we really want is a tool for developers to get that as quickly and easily as possible. And really address not just parts of it, like all the tools that are out there, or some of it, or most of it, but address the whole thing. So really, what we want is a single command - this is going back to the things that we learned at Heroku, was “Hey, let’s boil stuff down to five commands, or three commands. Can we get it down to a single command?” So one command that configures your cert store, gives you a domain name that is not localhost, that you can run things against, because your browser on localhost gives you not necessarily all the time a true secure context inside of your browser. It can do some weird manipulative things there to present like it’s a secure context to help you work around some of these issues… But we want to give you a real, full domain name–

A true experience, right.

Yeah, to work around that. You don’t want to mess with that C host. If you’ve ever – for everyone listening, if you’ve ever put anything in that C host, you forget about it later on, and then you’re like “Why is this not working?” and it takes you forever to remember to go look in that C host, and you’re just like “Oh, yeah.” And you’re like changing – you’re like “Oh, let me point this at Google. Let me point this at my ISPs, DNS server.” It’s like, “None of this is working. What’s going on?” So none of that. And then it needs to be able to manage your entire trust store for you, in a way that isn’t just opening yourself up, like “I’m going to turn off security in my browser.” That’s not good. “I’m just going to trust anything, shove anything into my local trust stores.” That’s not good either. So what we came up with was lcl.host, that second thing that I mentioned. And that is a command that you can run at the command line. You run anchor lcl once you install the command line, and it does all of that for you off of that single command. It sets up your local trust store, it gives you a subdomain on a domain name that we own called lcl.host, that those subdomains map to your local environment, and it gives you your certificates. And then we show you usually maybe up to like two to three things that you might have to do in your app, like add a plugin, add a package into your application, and then you reboot it and your application’s now automatically configured with Acme, to fetch certs off of Anchor. And the only thing that is put into your trust store, or certs.com from Anchor, from your account. So not even from – Johnny, if you run this and I run this, my system would not trust certs that you’re provisioning, and vice-versa. So it’s giving you the narrowest thing possible, but doing it in a very automated way. Again, off that kind of single command, anchor lcl.

So at lcl.host – actually, when I saw that URL, I was like “Crap! That’s a domain name you can actually have.” You can have a .host domain. I thought that was pretty cool. I didn’t even realize you could do that. But yeah, lcl.host. I’m basically looking at the install; it’s just a brew install, if you’re on brew. Is this supported across different operating systems as well?

[00:29:57.21] It is, yup. Right now Linux and macOS, and then Windows in the future.

Okay, cool. Yeah, just a brew install and anchor lcl, and that starts the process. This certainly looks easier than the hoops I’ve had to jump through to try and get something like that running. So the main value here for me, obviously - and we went through a demo a few days ago, and I can definitely appreciate how easy it made the process. I know for some who are not familiar with that whole notion, the moment they hear “Oh, some third-party? Who are these Anchor folks? Some third party is gonna be providing my CA? Why should I trust a third party to do this for me?” Obviously, y’all are developers, and you’re trying to solve a problem for other developers. But from a business standpoint, what is it that you’re ultimately trying to do? Is it just solving this particular problem, or where are you trying to take this?

Yeah, so first for lcl.host, there shouldn’t be any big concern with the certs that are being provisioned. We’re providing them for you; this for your dev environment. You’re not really - again, going back - you’re really not trying to protect some sort of information in development. You’re trying to get that dev prod parity. So if you think of it in terms of those things, you don’t care where that key is coming from unnecessarily, like who owns it, because you’re not trying to protect the data; you’re trying to get that experience, so you can get all the gotchas that you get in the browser etc, etc.

That’s absolutely true. We also build some security features into the CAs that we install onto your system. So as Chris mentioned earlier, we build CA for your exact development environment, for you as a user. So that CA is not going to be shared by anyone else. Like Chris was saying, a certificate, Johnny, from your machine, is not going to be trusted by my machine. But that CA itself has some properties built into it. It has what’s called a name constraint. That name constraint enforces a namespace that valid certificates have to belong to. So for that local CA that we build and install in your system via Anchor LCL, that CA is only allowed to issue certificates for lcl.host subdomains and local host subdomains. If Anchor were to issue a, say, google.com certificate that’s signed by that CA’s key, your client would not validate that certificate. That’s baked in at like the X.509 layer. All X.509 clients kind of perform the same operation to validate a certificate, and that operation will fail, because it will see that name constraint.

So let’s say Anchor - something bad happens, we lose root keys, those sign in keys for some reason… You don’t have to be concerned that those keys could be used to issue man-in-the-middleable certificates like a gmail.com certificate, because that constraint is always going to be there.

Yeah, thanks, Ben. Totally. So it’s not that the stuff’s insecure, it’s we are always thinking about, every step of the way, blast radius, stuff of that nature, how do we limit this and make things as tight and secure as possible, with very little to no input from the end user, so that you don’t have to think about these things. And then what I was gonna say is - we’ve been over to the Anchor aside. We’re going to be maintaining the certs, but we don’t necessarily have to provide all of the keys, and own all of the keys, and hold them. We can do things that allow the user that’s showing up when you’re trying to set up and run internal TLS for your backend infrastructure for the company or the users of our product that are employees of that company to bring their own key, and things of that nature. So we can really put all the control and the ownership back into our end users’ hands without them having to do the hard parts, which are setting up and running an internal CLA; making sure you have all of the extensions right. Have these things been run through the [unintelligible 00:34:11.21] linters and stuff of that nature, so that they’re of the type of quality, that high web PKI quality that you’d expect from a publicly-issued CA? And we’re providing all of that out of the box.

Break: [00:34:28.29]

Is any of the tool chain open source, or how are you approaching open source in general?

Yeah, the CLI tool itself is open source. It’s on our GitHub org page anchor.dev/cli. We use a dry version of mkcert, an awesome dev tool built by Filippo, who is on the Go security team. It’s kind of the standard de facto way to interact with trust stores, and we use that – we hope to contribute to it in the future, and… Yeah, it’s awesome software.

The SaaS version of Anchor, the certificate issuing parts of Anchor - that’s close to our SaaS product. But we do publish a lot of usable Acme clients that you can use in your code. For example, we helped write Puma. Puma is a Ruby web server for your Rails app. We wrote an Acme client so that Puma can use Acme to provision a certificate on boot. And a couple other languages… JavaScript… Go actually already has great Acme support. There’s the built-in, kind of blessed x/crypto package. It has an Acme package and an Autocert package. Autocert’s a great little package. There’s also the Caddy team, and Matt has written another great Acme client - I forget the name of that one, but that one’s even better. It’s got better features, and stuff. So Go has great Acme support out of the box, but we’ve been trying to bring Go’s level of quality Acme to other languages, and those are open source and on our GitHub repo as well.

[00:38:00.23] Yeah. And also Python - we provide the Python integration in those libraries. And yeah, to Ben’s point of bringing that good experience is really like we’re looking at Autocert as like “Hey, this is a model that does certain things”, and what we want to do is kinda help establish a set of patterns across all these different open source languages for how this stuff is done, which means that at boot time you can fetch your cert, and then cache it so that – it could be Anchor, or really any Acme provider. It does not have to be always in the boot process, in that boot chain. And then to auto-detecting when certs expire, a future step would be to detect “Hey, has a certain revoked?” So if those things happen, then the Autocert features of all these different languages will just say “Hey, I need a new cert”, and it’ll hit the Acme endpoint up to fetch a new cert.

So that whole thing where like certs are expiring, or they become invalid, you don’t have to think about it. This server just handles that at boot time, or even at runtime some of this stuff can happen, depending on the language and where the hooks are in those languages.

Yeah, the Caddy web service has this really cool property where when it’s properly configured with Acme, it will use OCSP requests as a heartbeat for a certificate. So it’s kind of like periodically pulling the CA that it provisioned the certificate from, and asking “Hey, has this certificate been revoked?” And so if you revoke a certificate, usually there’s a long process before that revoked certificate kind of gets rolled out and things detect that it’s been revoked. But Caddy doing this heartbeating through OCSP - which is not really the intention of OCSP - it’s able to do this really cool heartbeating trick and quickly rotate stuff when it does see revoked stuff.

Yeah. And when Ben says that can be a long time, we’re talking about measuring that in terms of like days, weeks or months. Again, it can be a very long time that you just have certs out there being served that have been revoked, that you really want taken down.

Very cool. Very cool. Yeah, we’ll definitely have links in the show notes for certainly lcl.host, and also github.com/anchor.dev. There are a few Anchors on GitHub, but it’s anchor.dev. That’s the org on GitHub.

That’s it.

Very cool. Very cool. Well, I think it’s time we transition to some unpopular opinions. What do you guys think?

Let’s do it.

Jingle: [00:40:34.10]

Alright, alright… Loaded up. Who wants to go first?

I’ll take it away. So my unpopular opinion is that everybody in your company - and this is doubly so for tech companies, and especially dev tools companies - works in support. It may not be your full-time job, but especially as developers building products and shipping code and features out to users, the best way to really understand the user pain and the user experience is by engaging with your customers. Go talk to your users, engage with them, see how they use it… So many times we see people – somebody hitting a problem, they’re writing in to support; depending on the support org, they can be highly technical folks, engineers themselves, to maybe less technical folks, but you are trying to assist them, talking to engineers and they’re like “Oh, this is an easy solution. You just have to do X.” But when you jump in as an engineer and you’re helping that customer, you start to see “Oh, this thing that I built - it has weird edge cases, and some rough edges and areas where I never thought of, and they’re using this product or feature in a way that I hadn’t really thought of”, and the best way to learn about that and get that true experience is to do some support, jump in and answer some questions, work with your customers and see what they’re all about.

[00:42:08.13] Yeah, I don’t think you’re gonna get a ton of pushback on that…

Maybe not. I think engineers tend to not really want to engage with customers, and I understand why, but you know… It’s not as scary, that’s what I’m saying. It’s not as scary as you might think. And if things start to go sideways, have the support team jump in and smooth things out.

Show you the ropes, right. Right. Yeah, it’s one of those things… It sounds scarier than it is. At Heroku I think everybody – part of the culture was at some point you’d do some time, you’d spent some time with the support team, and that kind of thing. And it wasn’t as scary as it sounded. Obviously, there’s this idea that if a customer has a problem, you have to know how to provide the solution, lickety-split. Right now. Like you’re just an answer machine. But sometimes there’s documentation for common use cases, but sometimes a customer is trying to figure out how to solve a problem that is not immediately sort of readily addressable, maybe because your documentation could be a little better to help them find something, or maybe it’s a use case you haven’t encountered, and their experience could help improve the experience for other customers that come after… So I think it’s a net positive for having – basically, contributing to the support experience is a net positive for everybody.

Yeah, yeah, absolutely. And the one thing to remember, if you’re getting in a conversation and it gets heated for any reason, and the customer is like really just unhappy, the support team is right there. They can jump in, they can chime in. You’re not an island on your own… So dive in, try it out sometime.

No insulting customers. No insulting the customer. [laughter] Alright, Ben, what have you got?

My unpopular opinion - I think that Go’s great cryptography libraries are a significant reason for Go’s success. A lot of people reference the parallelism, channels, static builds, a whole bunch of other stuff… But for me, what kind of first got me really using Go on a daily basis was the awesome cryptography packages. And when you think about it, static builds, they’re amazing. They’re like the best features of Go. But static builds wouldn’t really work if Go didn’t have great cryptographic libraries. If Go had to link to Open SSL in order to do an HTTPS request… If every time you wanted to use the network with a Go application you had to be running inside of a container with Ppen SSL properly installed… That kind of shoots the whole static builds right in the foot.

Personally, I got into Go because I wanted to do things like build little reverse proxies that could serve TLS traffic with certificates that came up from a database. If you’ve ever tried to use OpenSSL’s libraries, besides being a C project with years of cruft and problems, the interfaces are just really hard to use properly. Part of it’s because it’s got both like evented and synchronous mode, and then as they’ve added new APIs, they haven’t really cleaned up the old ones. So doing just something simple - maybe not simple, but something like “I want to be able to terminate a TLS connection, and I want to look up the certificate and key from a database”, that’s actually kind of hard to do with just Open SSL. And Go was like the first language that I saw that was willing to say “We’re going to implement our own TLS stack, and we’re not going to pick up this thing over here that has all these problems.” So yeah, I think like Go’s security team is amazing, [unintelligible 00:45:56.23] everyone’s working on the team is awesome, and I think it’s kind of like the unsung hero of Go’s success.

So Ben’s saying don’t use Open SSL. Got it. [laughter] Cool, cool. Awesome. Well, it was great having you, Chris and Ben, talk about admittedly a thorny issue on developer sides… But hopefully it sounds like you all are doing some interesting work, and you’re doing work that is going to be appreciated by a lot of developers, myself included… So I can’t wait to try lcl.host some more. Links in the show notes… But yeah, thank you all for being here.

Yeah, thanks for having us.

Thanks. This was fun.