Human scale deployments
Lars is big on Elixir. Think apps that scale really well, tend to be monolithic, and have one of the most mature deployment models: self-contained releases & built-in hot code reloading. In episode 7, Gerhard talked to Lars about “Why Kubernetes”. There is a follow-up YouTube stream that showed how to automate deploys for an Elixir app using K3s & ArgoCD.
More than a year later, how does Lars think about running applications in production? What does simple & straightforward mean to him? Gerhard’s favourite: what is “human scale deployments”?
Matched from the episode's transcript 👇
Lars Wikman: Yeah. That’s generally what I aim for. Some people really, really get excited about trying to solve problems at scale. I really, really don’t like what I see of systems at scale. All in all, it tends to be sort of a big challenge of making layers upon layers upon layers of people, and tech, and bureaucracy interact in a somewhat useful way. And there are certain things you cannot do at a small scale.
For example, the post office system, or the power grid - there has to be large-scale coordination in place, and then there also needs to be a lot of smaller systems that play nicely within that larger one. But I’m not interested in solving like a 100-engineer or a 200-engineer problem, in general. I like small teams, I like small organizations, and I trust smaller organizations more. There’s a lot of idealism in what I do, and I also optimize for my own enjoyment, which is why I’m not at like a FAANG, or whatever. I don’t think I could be bothered to pass those interviews anyway, but… I’m looking for things that I think can work at a particular scale.
And sometimes small teams can run large things. For example, WhatsApp is a pretty good example of that. Now, I bet they had a lot of orchestration going on, because they had to, because they were at an immense scale. But they also did a lot of things that are not commonly done. For example, [unintelligible 00:29:29.23] code updates all the time. So I think there are atypical ways of doing almost anything, and you can make it work, and you can probably make it efficient. And I don’t think you can find sort of a competitive advantage compared to other more general-purpose organizations. Choosing Kubernetes today is probably not a competitive advantage, because it’s so common. Doing well with Kubernetes, and sort of having a good org, and a good team, and all of that - that’s a competitive advantage compared to companies that are doing Kubernetes poorly. But almost everyone is doing Kubernetes, so I guess there’s no advantage to be found there.
[30:12] It’s a little bit like – I consider Elixir a competitive advantage for many companies, or a potential, at least, competitive advantage, compared to, for example, all the companies that run Java. You cannot win a competitive advantage by choosing Java, because that is not an outlier; it has no opinion, it is the most general choice. So there’s not a lot of advantage you can glean there. But if you go sort of off the beaten path a bit, either because you go sort of “Oh, we’re going to own all these details ourselves”, or “We’re just not going to bother doing half the work that everyone else considers critical.” There’s advantages to be found there.
For example, Apple likes to ship half-finished features and services… “No, no. We just removed a lot of buttons, and it’s so simple, and so straightforward.” Yes, but you could also add some options, so it’s more flexible. But they don’t. And I think that’s part of their plan for sort of shipping more things, even though they’re an incredibly large organization.
If you make decisions, and you sort of take chances and go in particular directions, I think that’s where you can find interesting things. I don’t have what I consider a complete plan for my operations. I don’t have all the tools figured out that I would like to. I’ve got some recommendations for some nice tools for sort of picking up WebHooks and just running commands off of it… It was like “Oh, this is written in Go. It’s going to be one binary. I could set it up with systemd, and it would run there.” That could probably be what picks up my final artifact from CICD and puts it on there, and then I have some scripts then to manage the deployment. But I don’t have a final idea that I’m like “This. This is how it has to be done.” And right now, often it’s done manually, for my personal needs, because that’s good enough. I script the most annoying parts.