Gerhardās transition to a senior engineer started 10 years ago, when he embraced the vim mindset, functional core & imperative shell, and was inspired to seek simplicity in his code & infrastructure. Most of it can be traced back to one person: Gary Bernhardt, the creator of Execute Program, Destroy all Software and the now famous Wat idea.
Few stick around long enough to understand the long-term impact of their decisions on production systems. Even fewer are able to talk about them as well as Gary does.
Matched from the episode's transcript š
Gary Bernhardt: So Execute Program - very quick summary of what it is, just because some of this is going to be relevant⦠So itās an interactive platform for learning programming languages and other tools. The lessons mix text with lots of interactive code examples, which is going to be important, because that code has to run somewhere. So itās a very unusual sort of infrastructure requirement. Itās been a commercial product for three years, the code has started maybe five years ago in early forums. A maximum of four people have ever worked on it, so this is a small product, although most products in the world are small, even though mostly we hear about the big ones, which is sort of a distortion in the way that we talk about things. And itās a bootstrapped company. So it makes real money, but itās small. Itās not a giant unicorn or whatever. So thatās the product weāre talking about.
Hereās the architecture. The primary database is Postgres. I love Postgres. I think itās great. It can do almost anything you need any database to do, ever, unless youāre at truly huge scale. The backend servers are at Heroku. Itās a monolithic backend. One repo, one server process, thatās it. It has some workers, with a queue, and the workers auto-scale as needed to accommodate load, just like the web processes do. And the workers in the queue are used for things like transactional emails, reminder emails, interacting with third-party APIās where we just want to sort of shield ourselves from those APIās if theyāre flaky, or slow, or whatever. We receive one type of incoming WebHook, and thatās from Stripe. And the reason that exists is if we create a subscription and the underlying credit card is later expired or something, then we need to know that, so we can remove that personās access, because theyāre no longer paying. So Stripe hits us with a WebHook for that.
And then we get to the weird part, which is how do we execute the code that the user is putting into these exercises? So we have a fleet of executor VMs that exist only for this purpose. They scale up and down as needed, to handle whatever user load we have, because you know, if thereās a peak, it could be a lot of these VMs, and theyāre expensive. And itās a very difficult process, because they have to be security-hardened, because theyāre running completely arbitrary user code, ultimately. And if we donāt harden them, and you know, firewalls and all that stuff, and sandboxes, then people are going to send spam, or mine Bitcoin, or do all kinds of nefarious things.
They also have the wrinkle that as the code is executing, theyāre putting tracing information into the queue, which ultimately gets aggregated into the database, so that we can debug things when things go wrong⦠Because itās ultimately a distributed system executing arbitrary code; itās quite a complex problem. And this, of course, is the most difficult part of the architecture.
So itās Postgres, a single backend, workers with a queue, Stripe WebHooks coming in, and executor VMs. Thatās basically the architecture⦠Which I think is like ā this is pretty normal, I would say. Well, the executor VMs are weird, because itās a specific property of our problem space. But I think this design is pretty normal and not particularly complex.