ngrok and Go

Welcome back, everyone. This is the Changelog and I am your host, Adam Stacoviak. This is episode 210, and today Jerod and I talk to Alan Shreve, the creator of the beloved ngrok. Everyone I know uses ngrok. We talked about what it is, why it exists, why he wrote it in Go, and ultimately why 1.0 is open source, but 2.0 is not.

Our sponsors today are Rollbar and Toptal.

Alright, we’re here today talking to Alan Shreve. Alan has made this thing called ngrok and a bunch of other stuff. Jerod, we use ngrok every week when we ship Changelog Weekly.

That’s right.

Where did this topic come from? Obviously we use it, but where else?

That’s basically where it came from. I’ve been a fan on ngrok for a long time. There are lots of tools that do similar types of things; this one seemed to be a cut above to certain degrees and in certain ways, and I just really appreciated it. Also, notice that it’s gone through multiple versions; it used to be open source and isn’t open source anymore, so I thought, “Hm, this could be an interesting story here.”

Yeah. Well, we almost didn’t have it on because of that. We considered it.

Yeah, well we had a little debate and asked our members what they thought about it in terms of is that an interesting thing or not. The consensus was…

Let’s find out!

Let’s get Alan and talk to him about it, find out what happened and see if it’s interesting. So Alan, thanks so much for joining us.

Thank you so much for having, and thanks for the really nice intro. It’s always a pleasure to hear about people who enjoy the software that I make.

We’ll obviously dive deep into ngrok. Like I said, fire up ngrok at least once a week - I do, personally - to ship our weekly email. But one thing we like to do is figure out where our guests come from. Obviously you’ve got a lot of interesting things happening around ngrok, you are a fan of Go - we talked about our Go podcast called Go Time - but take us back further. Help us get to know who you are and what got you into software development.

It was kind of serendipitous how I got into software actually. It started when I was going to high-school, actually. Our high-school had a requirement that you had to take one computer literacy class, because computers were the up and coming thing at the time, and we wanted to prepare our students for the future… [04:08] In addition to typing, of course. So you had your choice of a couple classes, and one of them was a class that was called something like, “How to use Microsoft Office.” I actually picked that class because I figured it would be easy, because I knew how to use Microsoft Office, so I figured I would have to do no work to actually do that class.

My best friend at the time, when I told him that after I came out of the first class, he was like, “What are you doing? You are wasting your time in that class. Come take programming with me.” I listened to him and joined the programming class, and instantly fell in love with it. So I really owe a lot of it to him.

It just seems like an unusual way to get into it. Obviously we all listen to our friends, but Microsoft Office? I mean, really?

[laughs] Yeah… I mean, that wasn’t the title of the class, “How to use Microsoft Office.” It was like, “Computer Business Applications” or something, but I quickly realized what it was the first class that I went into it, and they were like, “Create a PowerPoint presentation with three slides”, something simple like that.

I gotcha. I can recall feeling cool putting Microsoft Office on my resume at one point in my life.

Really?

Oh yeah, of course. Like, way back in the day when I had no understanding? Of course…

Yeah, ‘Proficient with Microsoft Office.’

Totally. I mean, I could type in that thing hardcore, good stuff.

I had a little bit of a similar path in college. I went into MIS (Management Information Systems), which is basically watered down computer science with some business stuff flurred in there. So I very much had the intro to Microsoft Office type of a course, and it was just the most boring, worst, terriblest thing in history. I’m curious to find out how you started from there and then actually continued towards writing software, and not just switched careers at that point.

Yeah, I’m not sure if I explained it correctly, but basically I was in that class and my friend told me, “You’re wasting your time in that class”, so I transferred out of it and into a dedicated programming class instead, to satisfy the computer literacy requirement that we had in high-school. So it was in that programming class that I actually started to learn how to program, on Turbo Pascal to start with, actually.

Wow.

Yeah…

Turbo Pascal…

What’s the difference between that and Pascal?

It’s faster.

[laughs] It’s turbo mode.

Yeah.

Have you guys ever used Turbo Pascal?

No.

No, Visual Basic I think was my intro, and then straight into Perl and open source from there. And also C and C++.

Turbo Pascal is an interesting environment. It’s really, really well suited to beginners because it basically… It had this really distinctive user interface too, just like this blue screen with yellow text for the code - by default, at least; I never bothered to learn how to change it. It felt very ’80s or ‘90… Before my time it felt, I guess. [08:00] So it was just this whole environment, it took up a full screen mode by default; it wasn’t like in a window, or anything. You were just kind of immersed in the code, and then when you ran the code there were two options. One was “Run the code”, and it would print like a console window, like what was coming out of STDOUT, and the other mode that it had was like a debug mode where it basically just fired immediately into a step through debugger with this big yellow line, or I don’t remember what color it was, that highlighted the line of code that was executing, and it would just step through the program for you. As a beginner, having to never learn any… Like, I didn’t have to learn about source control, or have to learn how to organize a project, or have to learn what a compiler was… It was just a full environment where there was a Run button and a Debug button. That was really great to start out with.

Take us from there to Go and ngrok.

Oh man, there’s a lot of stuff in between that. [laughter]

Well hit the highlights.

I finished doing the programming course in high school. After I learned Pascal I learned C, and then working the most basic knowledge of C++. Not how to use it well, of course, but how to define a class and stuff like that. I kind of actually lost track of programming for a couple years before I got an internship working at a local software firm. My first job actually was interning for this local software firm that no kidding made Fortran compilers, and they still make Fortran compilers to this day.

Wow…

Yeah, it’s one of those where you’re like, “Wow, someone still does that…” It’s like their only line of business, it’s crazy. So I started there. I actually wrote about it recently, when I was launching Equinox. I originally started there as an intern. I was actually packaging the software, which at that time meant physically assembling boxes and burning CDs and printing CD labels and stuff like that.

Well before an idea like package managers that we talk about today. Back then it was like some tape and some cardboard, right?

Yeah, I was the package manager. [laughter]

That’s a whole different version of it, for sure.

I know, right?

You said you got lost from programming… What happened to get you lost and why did this internship pull you back?

I got kind of lost because in between finishing the programming course and starting that job - in that job I eventually transitioned to writing code for them, instead of just packing boxes… In between that I’d kind of outgrown what my instructor could teach me, and resources on the internet for learning how to code and going beyond what I’d learned weren’t nearly as good as they are today (they are fantastic today). It was really a struggle to find… Not that there were no resources, but it was difficult to find the next step. I always ended up finding things that were below my level or way above it, and I couldn’t find materials that understood what I already knew and could push me gently into new things. [12:09] That was actually kind of frustrating, so I kind of lost track of it for a couple of years while I was sorting that out.

I remember I bought a book on Visual C++, which was a horribly misnamed book; it was actually a book on how to write Win32 GDI applications; I made my own Paint clone… But it explained nothing about the underlying model of it. It was like, “Now you have to send a WM_PAINT command”, and I was like, “Okay, I’ll type the code to send the WM_PAINT command. I do not know why I’m doing this.” [laughter] Yeah, so that was just a very frustrating time.

So you still haven’t gotten us back to Go… So keep going, we’re following the path. You’ve hit your internship, you were a manual package manager, you’re getting back into software… Keep going.

So I actually started writing code for that company. Then I went to college, I studied Computer Science at college, so I did a lot of programming during college, and it kind of took off from there. During college I did a number of internships trying to find the kind of company that I wanted to work for, so I went through a couple companies. I interned for VMware and Microsoft, and Fog Creek, and I ended up learning essentially that the ideal company that I wanted to work for was a company that was small. I didn’t really enjoy being part of a giant corporate environment, but also one that was tackling really difficult technical problems. I was looking for some combination of small but very hard technical problems, which is not always easy to find. It’s a lot easier to find now, but at the time it was not super easy to find.

So you interned at VMware and Fog Creek?

Yeah.

What were the kinds of things you were working on at VMware? What year was that roughly?

I interned at VMware for two summers; those were my first two years of college, and I was working on the QA team for what they called at the time Virtual Center, but has now been rebranded into something as part of vSphere - I honestly don’t know, because they’ve changed the branding so much.

Of course.

Yeah, so I was working doing QA for them, building essentially automated test harnesses, that kind of stuff. Then I did the next summer at Microsoft, where I got to work on the Windows 7 kernel; I actually worked on diagnosing performance issues in the file system driver stack, and then I kind of transitioned away from them entirely, interning at Fog Creek, where I actually helped them build the very first version of Kiln, which is their GitHub equivalent for Mercurial.

[15:51] I asked that because you said that you liked small companies with difficult problems, and VMware’s nothing like that, really. Maybe you were on a small team or something, but maybe you found out what you didn’t like about a big company tackling a hard problem, too.

Yeah, going through those internships was really the learning experience that brought me to that understanding. Before, I really didn’t know what I would enjoy, but doing all those things was really helpful to learn what it was that I was looking for.

Those are fairly well-known names, too… Household names, so to speak, at least for us.

Sure.

When it comes to an internship, I think… I’m camping on these questions, and I know we’re trying to get to your experience with Go and how you got there, but I’m thinking for the listeners out there who are thinking, “I wanna get established somewhere”, maybe a similar path that you’re taking, these are household names, so to speak, and you got an internship there. What was the process like, or how did you go about getting an internship? Was it just that easy from a college standpoint? Did they make it accessible, or did you have to audition, so to speak? What was that like? And also how important that was… I guess you just said so, but how important that was to where you’re at now?

It’s hard to say, but I think it certainly is important for me. I always felt having those things on my resume was really valuable. As far as how I got those internships, we had career fairs at college, and a bunch of companies would come and you would talk to them. VMware had a booth there that I went and talked to. Of course, nothing is always that cut and dried; part of it was that my friend’s brother worked at VMware at the time, so I have an in there…

Nice.

…and he was able to help take a chance on me, which I’m eternally grateful to him for.

It’s who you know sometimes… Lots of times.

Yeah. And the other internships… One of my friends interned at Microsoft one of the summers and was like, “You should try it there, I had a really good time”, so he introduced me to those recruiters.

That’s a good sales pitch… I might try that.

Yeah, right? [laughs] Fog Creek was the only one that I kind of went into blind, or I just applied blindly to them.

And they’re probably the smallest teams of those teams… Microsoft’s huge, VMware’s huge… Twilio was small, now they’re bigger. That wasn’t in your mention of internships, but it’s on our list of where you’ve been before. Fog Creek seems to be – maybe they’re getting larger now, they’re growing obviously, but they started out a fairly small team.

Yeah, I think they were maybe like 30ish people when I interned there. 30 or 40, something like that. Applying for the internship there is pretty easy though, because if you… I mean, I’m not saying that it was easy to get in, but as far as like doing all the right things… Joel wrote a blog post about, “If you want to intern at our company, you should do all of these things.” So I read that blog post and did all of those things. They definitely made it accessible, and I think a lot of companies are trying to follow that example. I’ve heard it’s pretty commonplace for companies that you interview at these days to provide you with, “This is what you expect when you come into the interview. These are the kind of questions that we’re gonna ask”, and I’m really in favor of that kind of attitude, of not just taking candidates and bringing them in blind, where they have no idea what to expect, but instead taking candidates and saying, “We want to set you up to succeed.”

[20:15] We’re coming up close to our first break, and we still haven’t gotten to Golang, we still haven’t gotten to really where open source fits into all this, so we obviously have to crack that nut open, but let’s take a break real quick, hear from a sponsor, and when we come back we’ll talk and get a little closer to your path to Golang and also how open source fits in. We’ll be right back.

Alright, we’re back from the break with Alan. We’re talking about… I guess we were trying to get to the open source path. Jerod and I mentioned ngrok and that’s an interesting project of yours, obviously. You’re a fan of Go, so where did your path to Go begin? At what point did you get to that? I know Go has been out for like six years now roughly, so at what point did you pick up Go?

I picked up Go towards the end of my time at Twilio, actually. Not that it actually had anything to do with Twilio at all, where I was working at the time. I guess it kind of had a little bit to do with the work that I was doing at Twilio. I was working with my friend Jeff Lindsay there…

We know Jeff.

Yeah, Jeff’s a great guy.

We’ve had Jeff on this show before.

Yeah, he’s awesome. At the time Jeff was really into asynchronous programming in Python, and we had a couple services at Twilio that were written in Twisted Python, and Jeff was a big proponent for another asynchronous Python technology called Gevent [00:23:18.19\], so I’d started building some stuff with that at Twilio. When Go came out I was kind of eyeing it, and I was like, “This looks like it’s the Gevent model, but not a hack on top of some other language. This is that programming model at the core of the design”, and that really appealed to me. So that’s kind of like what instantly got my interest about Go. Just to kind of expand on that, the way that Gevent works is basically it’s built around… [\00:24:01.26] Whereas Twisted Python or Node are built around an event loop with callbacks when events happen, Gevent and similar technologies are built around the idea that the runtime should handle that for you. So you can basically spin up these things called greenthreads, user-mode threads that you would pretend like they were… When you did IO or any any other kind of operation that would block the event loop normally, instead of having to manage that yourself, you would actually make those calls as if they were blocking calls. But the runtime would be smart enough to realize that they were blocking calls and manage the event loop for you. So they would basically take that user-mode thread that was about to block on a blocking call and put it on its own scheduler queue, and then stick an async IO event into the event loop that it managed. So it’s a very basic intro into how those technologies work. That was what Gevent did, but it was like a hack on top of Python, because Python wasn’t built for that, so it did really clever things like manipulating the Python’s stack frames, stack pointers to actually do things like that.

So when I saw Go, I was like, “Wow, this is like that model, like we should just build our code as if all of these threads are blocking, but the runtime should handle that complexity for us.” So that was what got me into it.

It’s nice when that’s built right into the language, as opposed to implemented into the library or added on later.

Yeah, it allows you to do a lot of things better when it’s just built in at that core level. Gevent always had these problems where you would worry about every dependency brought in, because if that dependency did any kind of blocking IO you had to worry, like did it actually block or did Gevent manage to monkey-patch those calls so that it wouldn’t block, or did it do the IO in C somewhere so you couldn’t deal with it at all? And you had no visibility into these sorts of things, and it was a real pain. So Go having all that built in was really awesome.

So I started building a couple toy projects in Go to kind of fool around with it, and then at some point I actually decided that I wanted to learn to actually build a substantial thing in it.

So was ngrok your first big, substantial thing, or did you have some prior art?

Yeah, ngrok was the first thing. Ngrok was originally a project built to learn Go, that was about it. The strategy that I really love for learning a new technology is to take an existing project and port it into the new framework, the new technology or the new language that you’re trying to learn, exactly. That’s what I did with ngrok.

The reason I like that is because when you build things - at least for me, I get all up in my head about what it is that I’m going to build. Suddenly it becomes a product, and I have to make product decisions where it’s like, “Should I do this? Should I not do that?” Whereas if you port something, all of those decisions are made for you and it becomes more like a class project where all the requirements are defined upfront and that entire cognitive load is gone.

Ngrok was essentially that, it was a port of Jeff’s tool, localtunnel, actually.

[28:24] Yeah, I recall talking about localtunnel actually back on #99, if I recall… We talked about Flynn, Tent, localtunnel… That’s really interesting. So was localtunnel in Python?

It was, yeah.

So you were interested in obviously learning Go, but then porting something to it and making it a serious project.

Yeah, that kind if happened accidentally, really, that it turned into a serious project. It was really just a project to learn Go to start with, and localtunnel was a project that seemed like relatively small and well-contained, and seemed from what I knew like a decent fit for Go, because it was pretty network-heavy. So yeah, I kind of picked it up and ran with it; it just kind of spiraled out of that.

Were you surprised by the success of it? I mean, I know when it first came out - I can recall back, it was probably at least two and a half, maybe three years, roughly… As far back as at least I know about it. And it seemed like an obvious problem for a large amount of people, so did it seem like a surprise to you that it was successful?

Yeah, I guess it was surprising that it was successful because it was just a clone; that was kind of what surprised me about it actually taking off. Yeah, it was about three years ago now.

Ngrok is so much cooler sounding than localtunnel. [laughter]

I cannot begin to tell you how excited I was when I found a five-letter pronounceable .com available. It was a beautiful moment.

Yeah, and I’m just telling Jerod in the back chan, I’m really interested in the name… Obviously ‘n’ stands for network, but why ‘grok’?

After I cloned localtunnel and I was happy with it, I started to think about other things that I would like in terms of the product. I worked at Twilio, so I was dealing with WebHooks all day… I built a lot of Twilio applications for testing, and when you’d build a Twilio application it was this really frustrating exercise, because you would write your code, and even with localtunnel or ngrok or an SSH tunnel, you would pick up the phone and dial a number or send a text message to actually cause Twilio to call back to your code and trigger it to run. So you’d do that once, but it’s always broken, right? So it would break, and then you would try to fix it, and then you would do the same thing again, you would pick up the phone. So I’d just end days with a call log with like forty calls to this one random number, just to try to get the application working.

So I started thinking about the kind of things that if I were working on that type of application that would make it easier for me, and one of those was that I wanted to see all of the traffic that was flowing across the wire, that was really important, as well as being able to replay it. The introspection part of it was kind of new, or wasn’t at least in any of the other tools, and I like the idea of being able to introspect things and look at them as they’re happening, so the name is kind of just a play on the word grok, coined by Heinlein, which is ‘to understand’, and ‘n’ because there are a whole bunch of other network tools that start with. It was kind of a play on ngrep.

[32:22] It’s weird, it doesn’t actually mean that to me any more. That was the genesis of it, but to me it just kind of is its own definition now.

A few things about this tool that impressed me… First of all, you do a really good job of explaining its value proposition on the website. It’s one of those tools that once you know you need something, you wanna expose your local development environment to somebody who’s not on your network, you need NAT traversal, all these things… You may not understand exactly how to get all that done, but you’re like “Man, I wish I could just give somebody a URL and they could actually access this demo that I have on my machine.” You do a really good job of explaining, like “Hey, this is what that tool does.” That was immediately impressive.

Then the fact that it is just a single Go binary, you drop it into your path and it runs, which has been kind of a flagship feature of Go for a long time; the easy distribution makes for a good first experience. Then there are a lot of little details that are really nice, and I’ll just give a single example - this might be ngrok 2.0, it wasn’t in 1.0, so you can help me sort that out… But for instance, when you’re actually running the thing, it will show you there, there’s a little display in your terminal that shows the URLs and the introspection if you want it, what’s going on, but there’s even a time where it detects an upgrade (there’s a new binary that you can download) and it just says, “Hit Ctrl+U to update itself.” So while you’re running it, you just hit Ctrl+U and it downloads and upgrades itself, and you then probably have to restart the program for it to run the new version.

Little stuff like that, that you just don’t expect from an open source project… So I’m curious where your attention to detail comes from, why you put so much time and effort into this?

Yeah, the updating thing… I was really psyched about getting that working when I did.

It’s really nice.

It went through a lot of iterations actually. There was an updating component to version one, as well.

I don’t know, as far as my strengths go in terms of building things, I think of myself partly as a software engineer, and the other strength that I feel like I have is product design. So a lot of that is just that attention to detail, of just trying to round every sharp corner, constructing error messages that are helpful and useful. I’m still really not happy with the error messages that ngrok kicks out. I’ve kind of been on a mission lately where it’s my goal to have every error message that ngrok issues have a unique code number that comes out of it, so with a unique prefix for ngrok, with the idea being that if there’s any error that you should actually – it should be like a unique string that you can put into Google, that will only point you at that particular issue. That way, if someone has that issue and tells me about it, I’ll know exactly what it is, and I could write up documentation for each one with like, “This is what happened”, because you only have so much space inside of a terminal to actually tell people what went wrong and how to fix it, and to be able to link to walkthroughs and guides, or possible reasons that this sort of thing happened. In a documentation, that would be really easy to find with a unique code.

[36:25] Yeah, that’s a great idea. I’ve seen a few projects starting to do stuff like that. I know Angular at some point started introducing… Their error messages would have short URLs in it, and it’s like, “Here’s the brief description of this thing”, but in the console I’d click there and it will actually just expand out to a full page that has all sorts of extra information. I think that’s super useful.

So have you been able to achieve the unique ID for every error yet, or is that an aspiration?

I’m getting closer. It’s certainly not all the way there, but it’s probably about half there. I don’t know… I really love building product experiences that people love, and so a lot of the drive that I have in software is to take really hard things and solve them, but make them really accessible to people. There are a lot of projects that I use that I get very frustrated with because they do really cool things, but the user experience is often really poor with them. I’ve always enjoyed making things very polished and very nice, and the auto-updating experience was definitely part of that for ngrok. I spent a lot of time learning how other people did it, researching the best ways to do it, and I still got all those things wrong to start with, but I’ve been iteratively getting better at them. Recently I actually just took all of that learning from building the auto-update experience and kind of spun it out as its own product recently.

I think there are some lessons to be learned here. This doesn’t just apply to open source, but for software development in general. Especially when your target audience is developers, when you sweat the details, I think it pays off in spades, because developers - probably more so than any other people, because we understand what’s happening, we understand what’s going on when an auto-update is occurring… And when you sweat those details, those little things that people may or may not even notice, really set pieces software apart from the pack, even if they are serving the same purpose. Because we appreciate the level of effort that goes into that, even if the value that it brings is minor - and I’m not saying an auto-update is a minor value, I think it’s a pretty big value, but just the polish on the software… Like you said, it was a clone, so it’s definitely not a new idea, it’s not a standalone product that nobody else is doing so it’s like, “Wow, this is singularly valuable”, but it works well and it was crafted with care, and your audience is developers mostly, or network people, and we tend to care about those things.

A lot of success, frankly, in open source, can be serendipity, right place, right time, those kinds of things… Sometimes it is based on merit, sometimes it’s not, but I think if you just sweat the details like you have, Alan, I think you’re giving your particular tool a better shot.

[40:04] Thanks. Yeah, that’s really gratifying to hear, because I care a lot about the details, so I’m always glad when other people care about them as well. It’s a good feeling.

On the note of success, I would actually probably say that because I have some sort of background knowledge on another project that could have been, or probably was just as good - and I’m just saying that lightly, I don’t mean that for sure, because it was a web service behind it, an install of a client, things like that, but… I had a friend - I still have, we just haven’t talked in a while - who roughly around the same time ngrok came out, had solved a problem like this but wanted to make it a paid service, mainly because they wanted to do something similar like you, they wanted to do something serious that was outside their normal job, that could potentially grow some legs and do something interesting. But something about ngrok - I would say ngrok itself, but also the fact that it was open source, so I think my hat tip to you is I think that - and maybe you can agree - open source really was, or it being open source and being the way it was, available to people - it solved a great problem very well as Jerod was saying - was a core underpinning to its success.

Yeah, I definitely think so. I think that a lot of people like open source for a number of different reasons, but there are a number of people who just simply like it on principle, and that’s the only kind of software that they wanna use. A lot of people like it for the hackability, that they can change things in it… So being able to tap into those developers by making a project open source is a really powerful to get adoption for anything that you’re building.

I guess a side note to the other project I mentioned is that that project right now is no longer…

I’m sorry to hear that. [laughter] It’s not a competition, I guess…

I know that, but it’s just…

Sad face.

It’s a sad face, but I think it’s interesting to – at least for me, because I have a different perspective than you, because you didn’t know this before me telling you this, but all along I’ve been watching you two in parallel to a degree, and I’ve seen you rise, and I imagine it’s for the reasons we all know, that open source is easily adopted, is easily contributed to, and you can go in and change it if you want. You can work hard and become a maintainer, become a contributor or whatever, whereas a proprietary software that is to a degree simple enough that it should be open source - maybe even better as open source, because it is infrastructure… I’m just trying to say that I think maybe open source was the better way to go, obviously.

For somebody out there who maybe thinks, “Should I close source this…?” but now you aren’t really open source anymore, so that could be the flip.

That’s true.

I think that’s the perfect teaser for the next segment, which is that you built this tool, made it open source, huge success - even to this day, ngrok 1.0 has like 700 stars on GitHub, lots of people using it…

People even bake it into their products too, into their software; they open source it, like “Yeah, fire up ngrok to do this.” It’s become a thing that people use daily.

That being said, I happily upgraded ngrok 2.0 without even a thought, because I was a happy user of ngrok 1.0. So let’s talk about that after this break. The switch - ngrok 2.0 closed source, what happened there, what was the decision, and all sorts of stuff around open vs. closed, after the break.

Alright, we are back with Alan Shreve, talking about ngrok 1.0 vs ngrok 2.0 - open versus closed, and what was probably a big decision to release ngrok 2.0 as a binary, and keep the source to himself. So Alan, tell us about this decision, this change, where it came from and why you made it.

Yeah, it actually was not like a decision that I made at any one point in time. It was kind of an overarching decision that lasted for like a year as I was building ngrok 2.0, maybe a year and a half. It took a while to actually build that out.

It was a really hard decision, it still is. When I started building ngrok 2.0, the idea was like, “I’m building it with all of the source code in a private repository, and then when I get towards launch, I’ll actually open source all of it.” So part of that was that when I started building ngrok 2.0, I started building it in a more modular way. The intention was to take a whole bunch of different libraries and open source them, so that all of these individual libraries that were used to build ngrok 2.0 would actually be open source. And that actually happened. A lot of the technology that is used to build ngrok 2.0 is open source.

One of the projects that I’ve built for this is a project called Log 15, which is like a structured logging package for Go, which allows you to build reusable handlers and reusable handler structures. So that ended up being open source out of it, and then another piece was the actual network player that does stream multiplexing to actually run a whole bunch of virtual connections over a single TCP connection. That ended up being open source as well, it’s a project called ‘muxado’ (that’s on my GitHub as well). Both of those projects have been successful in one way or another, that they’ve been useful to other people, which I guess is a decent metric for the success of a project.

So a lot of this stuff actually did end up being open source. The actual product that was built on top of those libraries is kind what didn’t.

So it sounds like you’ve modularized the underlying infrastructure, kept that open source, and built what was before open source ngrok into it sounds like potentially an actual product that people will pay for, on top of the open source that made ngrok 1.0.

Yeah. It actually wasn’t built on top of version one, it was essentially a complete re-write. There was almost zero code that’s shared across them, honestly.

Why the big rewrite, instead of just continuing to evolve what you had?

[48:00] Yeah, it was basically a result of a couple things. One was that ngrok was a project to learn Go, and when you’re learning things, you make a lot of mistakes; there are a lot of things that you don’t understand, a lot of things that you think you’re smarter than other people who’ve been working on it for a while… So I made a lot of mistakes, a lot of things that looking back on them I’m not proud of the way that it was built.

The other part was that the scope was changing. The scope for ngrok 2.0 was not this thing where there’s a single server binary that the ngrok client connects to, but I wanted to make it a very reliable service. One of the other pieces of quality from a product design that isn’t so much UX that I care a lot about and put a lot of work into, and that you see very little of from like a user, is reliability and stability of the service. A lot of work goes into that, and one of the pieces that went into that was building out the server component in a way that it was distributed across many machines, so that it could tolerate arbitrary machine failure.

These days - I actually just released this last week, is that ngrok actually now operates in multiple regions around the world. There are actually HA setups in a number of different data centers that coordinate with each other to actually run a global ngrok service. And that’s a very far cry away from “There is one machine that runs the ngrok service.”

Sure.

So actually rearchitecting it for that basically involved a complete rewrite.

So is it fair to say that the decision around this is financial?

Yeah, that’s totally fair to say. It’s mostly financial… There are some other things in it as well… It’s interesting, building a business model around open source is a tricky thing. There are a couple ways that you can do it - there is the “We do support” model. Like, “We give away all the source, and this runs some critical piece of your infrastructure, so you pay us essentially insurance.” That’s kind of what it is. If it breaks, you have a direct line to someone who can help you fix it.

Right.

There is also the “open core” model, where you give away most of it, but you charge large amounts of money for enterprise features, like single sign-on, audit trails, or something like that.

Sidekick is in that model, where there’s the open source community version, and then there’s Sidekick Pro, There’s derivations of that where you have Enterprise, which has enterprisy features, but Sidekick Pro is more like SideKick++. It seems like that model could have possibly worked for you. Did you battle around all these ideas as far as different ways of doing it, or did you just throw your hands up in the air and say, “Well, I’m just gonna keep it closed.”

Totally. The other model that was the one that I considered most seriously was the model that Sentry takes, and Docker as well, for Docker hub at least; all of the code is open source there. There’s nothing that’s like a closed add-on, but we run the service for you. So running the service is the piece that’s too complicated, that you don’t wanna deal with, so you pay someone else to do it. So that was the one that I considered most seriously, and I ended up having problems with all of them.

[52:07] The support model is based entirely around your product being core to someone else’s production infrastructure, that’s the thing that you’re paying insurance for. Ngrok right now is not that. It may be in the future, but at the moment, and certainly at the time, it was a development tool. So if it breaks, your developers are kind of unhappy, but it’s not something where you’re like, “Oh man, we need to have Alan on the line in 30 minutes notice to help us fix it if ngrok goes down.” So that one was kind of out for ngrok, which was kind of validated, because I never really got any interest from people, who when it was open source were like, “Hey, can we pay you for support?”

The other model that was seriously considered was running it as a service. The trouble that I had with it was that it kind of put me at odds with making the product really great from a server standpoint. I feel like a lot of companies that had this, “Well, you can run it yourself, or you can use our hosted service” don’t put a lot of work into usability when it comes to installation on your own service, because they’re not incentivized to. They’re actually financially incentivized to not make that good, so that you’re more likely to be like, “Oh man, I don’t wanna stand up RabbitMQ and a database and all of this other kinds of stuff to make this work. Let’s just pay someone for it.

And ngrok was kind of in this interesting place where it didn’t really require a database, it didn’t really require any infrastructure; it was just a binary. The server installation was such a simple thing; I’d put work into making it straightforward and really easy. So I talked with a lot of customers who were basically like, “Yeah, we could have paid you, but we basically had no reason to.”

The reason that that happens is partly because it was designed that way, but also the product itself doesn’t have any persistent piece. It doesn’t store your requests on the server or do any of those other kind of things that require more complicated infrastructure like a database or a message queue, or those kind of things.

So I wasn’t particularly happy with either of those options, so when I launched it, it was kind of launched in the mode of, “Well, I’ll keep it source closed for now and we’ll see if it makes sense in the future, if there are more pieces that I can open source, or maybe if I can find a way to open source the whole thing without jeopardizing the business model. That was kind of the thought around it.

Another piece of it that went into the calculation was if I made it open source and I couldn’t get paid to work on it full time, if that wasn’t enough money to make it a sustainable business, would it be better for the product and for its users to have it remain a side project, something that got my attention whenever it happened, whenever I had the time? Or would it be better for the product and for the users to be in a place where it was like, “I can work on this full-time and dedicate all of my energy to it.”

[55:48] I have two questions; you can take them in order, or take them however you like. The first one is where does the paid product begin with ngrok 2.0? Because, like I said before the break, I happily upgraded. Adam, I don’t know which version you’re running, but I wasn’t taking advantage of any of the open sourcedness of it, I was just using it as a tool when I need it. We were very casual users; Adam said we use it to expose a web server to campa ign monitor, so that it can suck in some HTML, and I’ll use it to develop a webhook here or there. So we’re very casual users. So tell us where the paid side – what model you decided to go with in terms of the paid stuff at ngrok 2.0 has, and as a follow-up to that, if you will, give us some insight into how it’s going and the status of ngrok as a paid product.

Ngrok 2.0 still has a very generous free tier, and most casual users never really break out of that free tier. That’s been something that I’ve been thinking about over the past years - do I want to consider that lead gen or do I want to actually take more things away from it that it incents people to upgrade. It’s a difficult calculus. I don’t want to cripple the product. I still want that really great initial user experience of like you download it, run a command, and it’s instantly working, and there’s no “Please pay” sort of thing immediately. You just immediately get value out of it.

It’s kind of all advanced features that people that are in the paid tiers today. One of the things that ngrok does that you can pay for is end-to-end encryption; instead of ngrok essentially terminating your TLS traffic at its servers and then re-encrypting it as it transfers over the tunnel, instead ngrok actually inspects the incoming TLS connections and multiplexes those to your backend so that you can actually do end-to-end encryption. Ngrok is just a router at that point, essentially, on the internet.

The one that most people tend to upgrade for is custom domains. Right now on the free version you always get a random domain when you start ngrok, so allowing people to pick a custom subdomain of ngrok.io, or even being able to run a tunnel over their own domain name, like dev.inconshreveable.com.

Yeah, kind of white labeling to a certain degree.

Yeah, that’s part of the paid features as well, and also some additional businessy stuff like IP whitelisting, things like that.

What can whitelisting do?

It basically lets you restrict the incoming connections to your tunnel endpoint to a certain set of IPs.

Gotcha.

Interesting.

Question two, how’s it going? Give us some insight into ngrok as a paid product. You also have Equinox, you can talk about that in light of how you’re doing it, making a business out of this?

The great news is that ngrok is a sustainable business, which is awesome.

Nice.

It means that I get to dedicate all of my time to it, which is really exciting, and it’s why it keeps getting better, really - I actually have all of my time to devote to it.
As far as Equinox goes, the work on that is kind of work on ngrok in itself.

What is Equinox, first?

[59:57] We talked earlier about the auto-updating experience that ngrok has, where it detects that there’s a new version, you press Ctrl+U and it updates itself. So Equinox is all of that functionality around building a self-updating Go program packaged into a service for you. So it does that, as well as packaging and distribution. If you have a Go program, it will package it up into an MSI for Windows, and a pkg installer for OS X; it creates a custom Homebrew tap for you, so that when you release new versions you maintain all of those things.

Equinox was built out of this desire to make the installation experience better. Sure, there are a huge number of developers for whom you can hand them a zip and be like, “Here’s the zip. It has a binary inside. Unzip it and run it”, and that works for a large majority of people, but ngrok’s user base is huge, and it includes a lot of people who are not technical at all; people who have never used a command line before. So being able to be useful to them means a better installation experience, a better updating experience, all of those things. So Equinox is the work that’s been put together to make that better. It’s been packaged up as a separate thing with the hope that it will be useful to other people.

I’m just looking at version one versus version two… It seems like 1.0, the original open source version wasn’t what you wanted it to be in the long term, and now with 2.0 you’re able to open up a web interface to it; obviously, you have custom domains, being able to look at traffic, things like that. I’m wondering if the web interface and the command line interface has a similar or a mirrored experience.

Yeah, so all of those things were actually present in 1.0. There was an introspection interface as part of version one, and there was a dashboard for version one as well. The website wasn’t as fully featured as it is now, but there was one that existed. As far as them being mirrored…

I just mean, are they similar? Do they have similar features? Do you get more if you use the web interface versus the command line interface or there’s things that just maybe aren’t present on either side?

I see. They’re not really mirrors; there are a couple things that are in common there, but it’s not like you can exclusively use the web interface to work with ngrok. If you wanna start it, you do have to use the command line to start it. Once it’s running, if you wanna look at the requests that are going over the tunnel, that’s the only thing that’s really mirrored, the status interface that you see in your terminal and that you see in the local web interface. Those are pretty much the same, except that the web interface is much more detailed, because it just has a lot more screen real estate to work with, and things like CSS and graphics, right?

Of course.

I have a hypothetical for you, and it’s easy for me to say because it’s your livelihood, not mine… But what do you think would happen if you took what you currently have, which is ngrok 2.0 as a product that’s being both used for free by some people and paid for by some people, and you just took that and you didn’t change anything in the model, but you just hit open source on that repo? What do you imagine would be the fallout or the change from that dramatic button click?

[01:04:04.15] That’s a really great question that I really don’t know the answer to… [laughter]

Well, that’s why I’m asking you to hypothesize here.

It’s one of those questions that… I wish I knew the answer to that question.

Yeah…

Honestly, maybe one day I will do that, and I’ll find out what the answer to that question is, because I’m really curious, too.

Well, email us when you do, and we’ll have you come back on the show and talk about what happened.

Yeah. That’ll be a really interesting experiment. But the unfortunate part is it’s one of those experiments that you can’t really run, right?

You can’t undo it…

Yeah.

This probably is one of the first times, Jerod, we’ve had somebody come on this show that started open source and went closed source, and it is to a degree considering going back to open source.

Well, insofar as I’ve asked him to…

Well, earlier he was saying that too, that he was hoping that he can eventually potentially open source even the code, even though paid features will be there. I mean, did I hear you wrong, Alan?

No, that’s totally right. The goal is to find ways to open source parts of it, and maybe eventually the whole thing. I’ve toyed around a long time with the idea of open sourcing just the client, which might be a thing that happens. Part of it might be actually taking just the protocol for actually setting up the tunnel and actually exposing that as an open source piece as well, or finding ways to… I don’t know, maybe experimenting with something like what Sourcegraph has done. They have their own kind of modified open source license, which still requires people to pay them for it.

That’s fair source, right?

Fair source license, yes.

Right, right. Beyang, actually… Speaking of Sourcegraph, we’re gonna have Beyang back on Go Time. Actually, we were gonna record with Beyang, but then it got rescheduled and he hasn’t come back on yet, but we mentioned Go Time before, and obviously Equinox and this is in Go, but that’s a really interesting example because fair source - I think most licenses are written by lawyers at least to some degree, but this one was actually written by a lawyer that’s very familiar with open source and very involved in open source, that was trying to liberalize this license where it can actually provide the right kind of things that open source needs, but also the right kind of things that business needs. It’s kind of interesting to think about that.

Yeah, I’m really excited about those kinds of efforts. They’re really exciting. I don’t know how they’re gonna go, but I’m hoping for the best.

Fair source.

Well, if you’re interested in that as a listener, I guess go to GoTime.fm and hit subscribe, as you’ll be hearing about Sourcegraph and I’m sure they’ll be talking about fair source on that show as well. Alan, let’s get to our closing questions; unfortunately we’re getting near the end of our time here. The first one we have for you, which is one of our favorites, is programming hero - if you had to name somebody who’s been a mentor or an inspiration or a hero of yours in programming, who would that be and why?

That’s a hard question. There are so many programmers that I look up to and respect. One of the more famous people that I’ve looked up to and that I guess I’m really impressed with is John Carmack. I don’t know if you guys have read Masters of Doom, or know too much about him, but…

I have it in my Amazon wishlist, but I haven’t pulled the trigger yet.

[01:07:53.09] It’s a fantastic book… Really fascinating, especially if you’re into video games, or ever played Doom, or wanna know the history of its software. Yeah, just a guy who is a technologist through and through, and is not afraid to tackle problems that everyone else thinks are impossible. And he’s still doing it. I don’t know if you’ve read the story about him and Minecraft, about how he basically got himself access to the Minecraft source code so that he could port it to Oculus, because he was dead set on it running on Oculus.

Wow…

Really?

Yeah, it’s truly fascinating. And this is a guy who’s been doing this for - I don’t even know how many years now, and is still at the forefront of his industry, leading it and doing amazing things. He’s definitely one of the people that I look up to.

Well, for those that listen to this show, this is the sixth time that I can at least tell by our notes that John Carmack was a hero, too.

He’s a big hero.

Yeah. That’s interesting though, to think that he was dead set on porting it over.

Yeah, that’s a fascinating story.

Are there any licensing issues with that? Is that legit, is that a cool thing to do?

That was kind of the crazy part, at least in the article that I read about it. They basically said that Carmack’s lawyers looked at the contract that they handed him and they were like, “You’re basically just doing this work for them for free. You’re not gonna have any rights to it” and he was like, “I don’t care, it just has to exist. I’m so excited about this.”

Hah…!

Wow…

Yeah.

So what came of it?

I’m not entirely sure. I think they demo-ed it at some presentation, but I’m not sure where the status of that is right now.

One other thing that we like to ask at the end of the show, Alan, is radar. We have a weekly email that we call Changelog Weekly, it’s our editorialized take on our radar. So when we have a guest on, we’d love to learn about your radar. So what’s out there - whether it’s a technology, or open source… What’s out there that if you had a free weekend you’d play with it? Something you wanna play with but you haven’t had a chance yet.

I’m really excited about all of the emerging languages, so I’d probably be playing around with those. I would be excited to try working with Rust some more, to try working with Nim, or Elm, or any of those sorts of things.

Rust, Mim… What’s Mim?

Elm.

Elm, okay.

Yeah. All of those new and emerging languages that are approaching problems…

Elixir?

Yeah, Elixir… They’re all pretty exciting to me. So I’d take a look at those. Another thing not in the language sphere that I’ve been meaning to take a look at for maybe over a year now is NixOS, as like a better way to do configuration management. The ethos around it is really appealing to me, so I’d like to spend some time actually trying it out and seeing how well it worked.

I feel like we’ve heard of NixOS, Jerod. Have we mentioned this recently? It sounds familiar.

I think you’ve probably seen it in Changelog Nightly a couple times. I’ve definitely heard of it, but we have not featured it in Weekly, nor have we done a show on it, so it’s a good one for radar… Now it just doubled up on ours.

Alright.

Well Alan, I think that’s the close for this show. We would obviously love to talk to you more deeply about your fun trip down open source lane with ngrok. It’s kind of interesting to see it start the way it did, as open source, go closed, and potentially come back as maybe fair source, who knows…

[01:12:08.14] Yeah, who knows…

…but I guess we can consider that open source. It’s in that grey area.

It depends on how you define open source. If it’s like, “I can see the code, then it’s open source”, if it’s free open source, then I don’t think it is, at least.

Any closing thoughts from you before we close up? Anything you left unturned that you wanna mention to the listening audience?

I just really wanna thank you guys for having me on here. It was such a pleasure talking to you, so thanks!

Cool, man. We’ll have to get you slotted on Go Time too, since you’re such a Go fanatic. You have to listen - GoTime.fm.

Yeah, I will definitely check it out, it sounds awesome.

Alright, cool. With that, that’s the show, so let’s say goodbye, everybody.

Goodbye. Thanks, Alan, for the great tool!

Goodbye everyone, thanks for having me on.