Steve Yegge: Because we have to pay for GPUs for you, basically. So it was a bit of a logistics challenge to actually get Cody out to everybody. Next step is we’re going to make it an even more lovable experience. I mean, like I said before, a lot of your experience with any AI coding assistant, including Copilot, is going to come down to your tolerance for it making mistakes… So we’re putting together an unbelievable world-class AI team, that’s pairing up with our code intelligence, our code graph, which is the thing we’ve spent ten years building. It’s the external store. Remember I mentioned Google is going to be fine, because they have all this incredible search infrastructure that the LLM can use? We’re in the same boat. We’ve got fantastic technology, so we can start improving Cody’s quality.

Because ultimately - look, Cody is great right now if you have a question about your database, about some stuff you haven’t seen, it’s new code you haven’t seen… But for day to day, where you already know the code, and your IDE is helping you, and stuff… And maybe Cody doesn’t come to mind as your very first stop, so my goal is, in very short order, we want to make it so that Cody comes to mind first when you’re thinking of a much broader class of problems. And we do that by making it really, really good.

You would be shocked at how much low-hanging fruit there is. I mean, you look at Cody and it does make a fair number of mistakes, but the ability for us to fix these things is extraordinary. Everything that we’ve tried has made it better. So in other words, we’re starting with kind of the worst possible implementation of Cody. It’s OpenAI, generic, sort of Wikipedia embeddings… I don’t even think they’re code-oriented, you know what I mean? So yeah, it’s gonna go out better.

Where are we going with it? I mean, long-term… Look, I want Cody to be your first stop when you’re coding. Look, one of the things that I was always jealous of, truly envious, about VPs with executive assistants, as I was going through my career, was that they were always like a team of two people. And they were always incredibly productive, and the VP never had to go to the printer, never had to do any of the expense reports and stuff that engineers are always having to do. The VPs were always freed up. I mean, honestly, that’s what Cody should be for a developer. It’s like “Oh, man, I’ve got some stuff I need to get done, that I really need to get done that I’m struggling with. Let me just have the AI help with it.” Honestly, that’s where I want to take it. And it’s so early days right now. I mean, it’s really raw right now. It adds a lot of value if you play with it. But still, I mean, you probably find that it also has a lot of mistakes. But the way I see it - I mean, this is inevitable. This is an inevitability. It’s going to happen. It may take a year, it may take five years before we’re all using it every 10 minutes… But it’s inevitable.

Jerod Santo: [55:47] “You’ve now left the ‘90s…” Maybe I should explain why we’re using local storage. Some of it is actually just technical debt. This was a decision I made when building the platform back in 2015, around how we handle uploads. Not image uploads, but mp3 uploads… Which is one of the major things that we upload and process. And these mp3’s are anywhere from 30 to 100 megabytes. And once we have them, we also want to do post-processing, like post-upload processing on the mp3’s. Because we go about rewriting ID3 tags, and doing fancy stuff based on the information in the CMS, not a pre-upload thing. So it’s nice for putting out a lot of podcasts, because if Gerhard names the episode and then uploads the file to the episode, the mp3 itself is encoded with the episode’s information without having to duplicate yourself.

So because of that reason, and because I was new to Elixir and I didn’t know exactly the best way to do it in the cloud, I just said “Let’s keep it simple. We’re just gonna upload the files to the local disk.” We had a big VPS with a big disk on it, and were like “Don’t complicate things.” So that’s what we did.

Jerod Santo: “You’ve now left the ‘90s…” Maybe I should explain why we’re using local storage. Some of it is actually just technical debt. This was a decision I made when building the platform back in 2015, around how we handle uploads. Not image uploads, but mp3 uploads… Which is one of the major things that we upload and process. And these mp3’s are anywhere from 30 to 100 megabytes. And once we have them, we also want to do post-processing, like post-upload processing on the mp3’s. Because we go about rewriting ID3 tags, and doing fancy stuff based on the information in the CMS, not a pre-upload thing. So it’s nice for putting out a lot of podcasts, because if Gerhard names the episode and then uploads the file to the episode, the mp3 itself is encoded with the episode’s information without having to duplicate yourself.

So because of that reason, and because I was new to Elixir and I didn’t know exactly the best way to do it in the cloud, I just said “Let’s keep it simple. We’re just gonna upload the files to the local disk.” We had a big VPS with a big disk on it, and were like “Don’t complicate things.” So that’s what we did.

Jerod Santo: Right. Well, let’s loop back around at the JAMstack bit, because that is interesting, and the conversation has been somewhat ongoing on the show. Maybe even more so on JS Party, but I wanna loop back around to that. Let’s just start with SQLite in production. First of all, I’m a SQLite fan. But I do tend to reach for Postgres when it comes to production… And I don’t know if I do that because I just feel like SQLite’s just not made for production… We do use it, I guess, in one production capacity for Changelog Nightly. It’s what backs Changelog Nightly. But that’s basically a batch process that runs Nightly, and does some processing and sends out emails, and persists its state in SQLite. But it’s not like a web server that’s getting hit by hundreds of requests a second, and all that…

And I always thought SQLite was cool and all, and for specific things, like in your phone it makes sense… But would you run it on a VPS with a web server frontend?

Jacob Hoffman-Andrews: …they do have a box you can check. I don’t know if that’s – it sounds like you have a VPS with shared login, so it might not work on that.

Mitch Wainer: [12:22] I would say I was splitting time between both companies, both entities, but it really shifted when I dug into the market research… And this is a really funny story. Basically, I had a conversation with Ben and we were just kind of ideating on ways to find out how next generation businesses and startups are running their applications online. Are they leveraging virtual machines and virtual servers, or are they entirely on a manage dedicated machine?

So what I did is I basically bought a bunch of pizzas and I knocked on the doors of every single startup in Soho, the “Silicon alley of New York City”, and I asked the person at the front desk “Hey, can I spend 5-10 minutes with your head of engineering, CTO, VP of engineering, just to ask them a few questions about how their infrastructure is currently set up? In exchange, here’s a box of free pizza for your office and your team.”

So I would say I had probably around a 50%-60% success rate, getting just a quick conversation on the table with these VPs and the CTOs… Which was great. It was immediate validation that everyone was moving to the cloud. So I spoke to about 30 companies, so a lot of pizza… And every single company either had their entire production application and even staging environment all in the cloud, or they had a hybrid model of cloud and dedicated… But no one was purely dedicated, and Server Stack was a managed hosting company predominantly focused on managed, dedicated hosting. So I came back with all this insight and information. It wasn’t like an enormous dataset. There was only 30 conversations being had, but nonetheless, it was validating… So that was a point in time where I think the decision quickly shifted on where we actually focus our time and effort, like “How do we prioritize the business?”

There was obviously employees working at Server Stack and there was employees working at DigitalOcean, but I think slowly over time we started to shift the focus and shift the priorities of the business to focus on DigitalOcean… And we took a few swings and we placed a few bets along the way and we were slowly growing DigitalOcean, but it really wasn’t until we dropped the pricing of the lowest-tier server to $5 and doubled the RAM and doubled the hard drive space with SSD drives in the server, and we became pretty much the first cloud to offer SSD virtual machines online.

David Heinemeier Hansson: I think in the past you were thinking “What would it take for me, if I wanted to have my own server?” I call up Dell and say “Hello! Can you send me an Epic box with 48 cores? And here’s the address to the data center I want it delivered.” The jump from getting that piece of iron able to host your application was kind of tricky. It was quite different from the Heroku path. Like, anyone could set up Heroku; far fewer people would feel comfortable setting up a physical box. I want to shrink that gap, such that the bare metal deployment scenario looks virtually identical to the cloud deployment scenario. That it’s not a different category. It’s using the same tools, it’s using the same configuration, it’s using the same method of thinking about it… But also, most people don’t start by phoning up Dell to order a $5,000 piece of hardware. They go “Oh, I have an idea. I’d like that idea to get online.” “Oh, Hetzner has a VPS for $12 a month, with a couple of cores and enough RAM for me to run it… I’m going to deploy there first.”

So I think still the path for most developers will go through the cloud first. I want to make sure that when they do that, they don’t end up getting locked in to a commercial platform, which I’ve seen time and again happen, including on Heroku, where some relatively large services suddenly have some very, very large bills that are making things quite difficult for them, and they kind of don’t feel like they can get out. They feel like they’re in too deep, and that would be a huge project to move off Heroku. I want to make it a trivial project.

So as I said, you can start in the cloud, because I think most people should. Now, what is cloud? I think there’s a discussion there, too.

Jerod Santo: and deployed it all to a $4 VPS of an unnamed provider using Docker with docker-compse.

She then used K6, an open source load testing suite that generates virtual users who continuously run test scenarios using JavaScript, to see what the machines was capable of.

Austin Gil: [50:02] Exactly. On top of that, on top of the trouble with distributed systems is what happens when you start receiving large files and you have a server that’s maybe like a lightweight server, and totally enough for your application to run, but it has smaller disk space, and all of a sudden, large files are taking up a lot of the disk space, you run out of disk space, you have to upgrade your server. Now, if you have to upgrade your server because you want to add more compute, you’re getting sluggish, or you need more CPU - excellent reasons to upgrade a server. If you are running out of disk space, terrible reason to upgrade a server, because you may end up doubling the cost at a not very cost-effective way of essentially doubling the disk space. And before you know it, you’re running out of disk space again.

So Kball, you mentioned S3… S3 stands for Simple Storage Solution, I think. S3 was created by Amazon, along with their service. They created their own protocol for storing files, adding files, managing files, deleting, updating, whatever. And it’s been great. And that protocol has been adopted by a lot of other file storage solutions partners. I do work at Akamai, I feel like I should interject that we have an object storage service that we offer, that is S3-compatible… And the article that I write, again, speaks to fundamentals. Yes, I work for a company that offers object storage solutions, yes, in my examples I use that… But everything that I write, I try to do, again, at a fundamental level, so you can take this and apply it to anything that is S3-compatible. So anything that uses this protocol, take it and put that knowledge where you want it. Very important to me.

But one of the big factors for object storage, S3-compatible object storage, is that they are designed for these files. So they’re highly available, easy to access, they store it in one location, and they’re very, very cost-effective. So for the same price that you would get 25 gigabytes more on a VPS to upgrade your VPS, you can get 250 gigabytes, at least with Akamai’s cloud computing services. So that’s a 10x difference. And that applies – and again, that would be across different service providers. I’m only saying the numbers that I know off the top of my head, but… Not trying to turn this into a sales pitch.

Richard Hipp: And it’s hosted by Roy Keene. But the thing with Fossil - it’s really designed for self-hosting. We make it really easy to set up your own Fossil server on a $5/month VPS or on a spare Raspberry Pi that you have lying around. It takes very little hardware to run Fossil. I know some of these other systems, they say “Oh, you’ve gotta have at least a $40/month VPS in order to support this, it’s so heavyweight.” But it’s very low resource. So you can just plop this up there. It’s a single executable, you plop it on your machine… A two-line CGI script gets it running, and it just does everything for you.

So the motive for having a service like GitHub for Fossil is greatly reduced. Because if you were to just take raw Git or raw Mercurial and wanna set up a collaborative development site like GitHub, that’s a lot of work. GitHub provides a very valuable service. With Fossil, the amount of work to set this up is greatly reduced, and so the need for that is also greatly reduced.

Now, what people have told me though is that for some people who live in other countries, coming up with $5/month in hard currency for a VPS is a hard problem. And for them, having access to a free repository like that is a big deal. But for those of us who are fortunate to live in the U.S, or other Western countries, it’s probably easier just to set up your own.

And let me just – I’m coming back to this subject, but think with me just a second… If you talk with people that like to go backpacking - do you like to go backpacking yourself, or do you have friends that do that?

Jerod Santo: …VPs at large corporations that we have to deal with PR, and it’s very corporate and official… And we’re covering this much broader demographic, really, than we were before. The community has broadened quite a bit.

Jon Kuperman: Yeah, I’d say there’s the big things, like to check out – if you start a new Pages app, or if you move your site onto Cloudflare, just playing around in the dashboard, which we’ve been working on design tweaks to, but… We basically have a lot of cool stuff about automatic image optimization, making your images smaller, or reading the device size and serving images… So that’s like Polish and Mirage.

[43:59] Then we have the Rocket Loader, which will turn off unnecessary [unintelligible 00:44:02.07] So we’ve got a lot of these options, and these are all one-click options for the most part. So if you start a new Cloudflare Pages app, or if you move your website over… Sort of looking at these things - automatic imagery sizing, automatic compression of JavaScript and CSS and HTML, serving images by the device width… Or with the automatic resizing, every time we have to do a resize, we save it, too. That’s like a paid service. And so then you just upload one image, and then as devices come in, we start resizing it based on their width, and then we save all those cached in the edge… So you end up with 10 or 20 copies of your image. Rocket Loader for speeding up JavaScript-heavy sites…

And then the other one that I wanted to talk about which I think is really cool is we have this tiered caching initiative which is now available in the dashboard… So the idea being let’s say you have 200 data centers, and you use our CDN. Or we have 200 data centers, you use our CDN. So your stuff gets cached in all 200 data centers, but then every time you do a cache purge, or a user requests something where there’s a miss, it has to always go back to your application.

So a hypothetical, I guess, would be like you have an app, you go buy a DigitalOcean VPS, you put it in your New York City, and you put your app on there… And it gets overwhelmed, so then you sign up for Cloudflare CDN, and poof - all of a sudden there’s all this caching stuff. But you can still have this problem where if somebody’s hammering on an end point, or just users are trying to request an end point and it’s a miss for the cache - all the caches will start going back to your one DigitalOcean VPS that you bought. It’s almost like a mini-DDOS, an innocent/accidental one…

But you see this – like, Facebook went down the other day… So all of these big sites, like Facebook, Twitter, Google - they take in more traffic than they could possibly serve cold; like, if all the computers went off and all went back on again - they’d just get hammered. So they rely on these caching mechanisms. So one of the problems for these companies when they have an outage and things go actually down - you’ll see this all the time, where they keep trying to go back up, and it’s like up and then down again. And then it’s like, they go up and they go down. So they have to come up with these clever ways of like “Okay, we’ll go up, but we’ll block any users whose names don’t start with A”, or they’ll come up with these ways of doing these slowly booting back on, so they don’t get just decimated again.

So the same thing can happen with just anybody’s application, where you have like this one VPS that can handle (whatever) 100 or 1,000 hits a second, and then something gets purged from the cache and all of your users are trying to hit it - boom, it takes it down again.

So we came up with the idea of tiered cache, where we take those 200 and we mark, let’s say, 25 of them as top-level cache CDNs. Like, cut top-level endpoints. So what happens is now the other 175 are no longer allowed to hit origin when they have a cache miss. What they have to do is they have to find their closest top-level cache system and ask it for the file.

So it’s like, “I need a JPEG, and I hit this one – I’m in Florida, so I hit the Miami one. Miami doesn’t have it. Miami is no longer allowed to just hit New York.” [unintelligible 00:47:10.19] it can’t do that anymore. It has to find its closest one, let’s say like in South Carolina, and be like “Do you have the JPEG?” And only if that doesn’t have the JPEG, South Carolina has permission to hit New York now.

Suz Hinton: I’ve always sort of messed around with rented Linux VPS‘es… And when it’s your own box, you tend to do all the dodgy stuff. You do all the cowgirl stuff on the server, you modify your NGINX config live, and things like that. And usually that’s done in the shell session, so really the most practical options that you have are something like Nano, or Emacs or Vim or Vi… And it’s installed out of the box on so many Linux machines, so it’s just always there.

So usually I would pick it up if I needed to use it, and it was usually within that context, of just like messing around on Linux without some kind of display drivers or GUI. To be honest, nothing about the experience itself made me wanna pick it up. I knew how to go into insert mode and get out of insert mode, and sometimes, depending on the machine, you could actually get away with using the arrow keys instead of J and K and things like that…

I’ve been curious and felt that it was valuable to learn properly, and I knew other developers and colleagues who used it, but I just never really felt motivated enough at the time, which is why I really only picked it up four years ago.

Jerod Santo: Imagine Fly.io on your own VPS

Sidekick creator, Mahmoud Mousa:

I’m tired of the complexity involved in hosting my side projects. While some platforms, like Fly.io, stand out in the crowded field of Heroku replacements, I believe a simple VPS can be just as effective. That’s why I created Sidekick: to make hosting side projects as straightforward, affordable, and production-ready as possible. You’ll be surprised how much traffic a $8/month instance on DigitalOcean can handle.

Grab a VPS (or your own hardware if you prefer, just need a public IP) with Ubuntu on it, set up SSH access for yourself, and let sidekick init take you from there to a deployed production application “in minutes.”

Jerod Santo: Yeah, 100% agree. I have self-hosted many things throughout my life, and I learned so much doing so. Both production, small business, and friends and family stuff, whether it’s self-hosted on machines in my house, which I have done for a long time… I had a Linux server that ran in my home, and I ran all kinds of services off of that. And then also VPS, self-hosting, shared hosting, self-hosting… You know, just managing your own things. And yeah, the amount of learning is really not to be compared with. You can’t fake it, it’s just real.

But the learning comes through trial, and it comes through things going wrong. And this post that he writes is like “Here are the –” One of the sections is like “Things that broke in the last six months”, you know? And it’s like, yeah, you’re going to have that kind of stuff. And it’s similar to any sort of maintenance. I don’t know how many large appliances and vehicles you own, Adam, but you and I are both old enough to know… A house, a car, a laundry machine, anything that’s significantly expensive and significantly complicated breaks. And when you own it, you own the break, you know? And you’re going to either pay to fix it, or learn to fix it. And for me as a guy in my 40s with a pretty large family, I don’t have the patience or the time to self-host anymore, but I think everybody else should. [laughs] Not everybody else, but given your circumstance, I think everyone should try it, and some people should do it. And there’s certainly things that I’ve thought about self-hosting, even though I don’t have the time, because of privacy and autonomy, and the independence that Christian speaks of. For me more importantly than the learning, because I’ve already learned enough, I think, in the category, is there are things where I want the independence and the privacy. And so that’s why I consider it, even though I don’t want to do it, because I know how much of a headache it can be.

Silvia Botros: I did not understand how VPs think about things, how to get VPs to care about certain things, and why are they in so many meetings… Junior engineers will always say stuff like this, “They’re just in meetings all the time.” I’m like “It’s not just meetings. There’s an overhead to communication when the company grows.” So that book was so helpful… Shout-out to Camille Fournier. So helpful in understanding that mindset. “Oh, this is the stuff they care about.” Because if you don’t understand their incentives, you can’t work with them. It’s that simple.

Anurag Goel: I think that it’s always very enticing to go to on-prem or bare metal for a company like Render, because yes, CPUs are cheaper there, bandwidth is cheaper there, you end up either having better margins, or giving people a better deal, or both. We, as a startup have a lot of constraints on how many engineers we can hire, and what we can do with the limited time we have, and how do we grow revenue by building things that people want. And our choice has been to continue to focus on the higher-level products and features that people need from their cloud. And over time, that will change.

[00:44:14.17] So Render now has an engineering team that does focus on building stuff on top of AWS. But for the next stage, you can imagine us hiring a team that is already experienced with running infrastructure, let’s say Kubernetes even, on prem; or running, say, a hypervisor on bare metal. And at that point, I think we will expand across the sort of cloud versus hybrid versus on-prem spectrum.

My learning is to just focus on what gets users value the fastest as a startup, and not try to be technologically cool, or say “Oh, we’re going to do bare metal because it’s going to increase our margins when we’re tiny.” When you’re tiny, margins should not be what you focus on. When you’re tiny, you should be focused on creating customer value. And so that’s something that we definitely experimented and learned the hard way, and that’s something that I would do differently next time.

And then instead of focusing on “Oh, let’s do bare metal” - and this is a debatable opinion, but for us it’s important to focus on providing value. And I do see now a lot of people saying “Oh, you should run applications yourself on, say, Kamal, Rails” and DHH is spearheading this sort of “Let’s not –” Even though I know him, and he likes Render, I think that ultimately people have to decide what works best for them. And you can totally run a Rails application on a VPS using Kamal. And you should if it’s just you building your app. But as soon as your team grows, and as soon as your needs evolve, and as soon as you have, frankly, a lot of paying customers, reliability starts becoming much more important than how you’re running your app. And your users don’t care; they absolutely do not care if you’re running on bare metal, or AWS, or Hetzner.

Eric Clemmons: I really appreciate Kball’s framing recently… Drawing that line I think is really interesting, especially with the shift to serverless… One of the things that’s been on my mind is “When was the last time I deployed to a VPS? When was the last time I ran my own server, and what is the complexity there?” And that opens up so many doors to running other runtimes too, like PHP, and having it available. And now it’s like, I have forgotten that; I may have lost that skill. So it’s kind of a shame to be limited now, with like the latest and greatest, and lose all the possibilities that I used to take advantage of. So… Food for thought. Thanks, Kball.

Adam Stacoviak: My thought is this, is that you don’t have to choose one or the other. I think you can do both. I think that Gertrude, the hosted version of it is for “the soccer mom”, or somebody who just doesn’t seem to be that connected to technology, spinning up a Linux box, or managing a VPS, or whatever it might be. These things that we know, because we’re technologists, and we play with this stuff because it’s fun for us. That’s where I think that comes into play.

Linux is widely distributed because of its model, its open source nature. It’s many distros of many flavors. pfSense is widely used because it’s open source. It’s also a security tool. Imagine if they kept pfSense behind the scenes and said “Okay, this is not open source. This is only a paid for product.” Would it be the same pfSense? It is probably arguably one of the most deployed firewalls out there, aside from say UFW, because that’s like free, and an apt-get away; it’s pretty easy to install that and configure it. But pfSense is a tried and true router, firewall, all these different things rolled into one. And you can deploy it on cloud, you can do it many different ways. It’s the core software, I think… And again, back to how we got there - it wasn’t me advocating, “Hey, open source this”, but more so the security measures. Can you be more secure? Or can you have potentially better software that meets the needs of the world if the world uses it, if the world can give back to it and care just as much as you care? …but still give you the freedom to spin up Gertrude, the hosted version of it, which is for the everybodys, essentially; the non-techies.

Jerod Santo: One thing you said at the beginning there, Steve, was that it’s more about where you run it, because we have all this code we talk about the copy and paste thing and the dead code on the internet… And I think that’s really powerful, not just for beginners, because I definitely think that mindset… I used to teach coding bootcamp-style web development 101, and getting people set up and running, and troubleshooting environments was such a time suck for that. And that was full stack, it wasn’t merely in the browser, frontend. So Val Town and other options like it are just a boon for teaching people and getting people started. Most of them, like we’ve been talking about, are frontend oriented, whereas this is more of a full-stack serverless option.

In a way it’s really cool, because on the website you have some use cases right there on the homepage that even though it’s great for beginners actually speak to longtime developers as well, because there’s certain things that we have to do, and it’s easy to code those things up, but then the question was always like “Well, where is this going to run?” And then we all have our solutions, like “Well, maybe it is on Lambda, because I already am an AWS expert, and I can just get that thing going.” Maybe I have a VPS that runs Linux, and I put all my random crap on there… Maybe I have a few strewn about; you know, home network, home lab… All these places. I have some cron jobs that run on my laptop. It’s not a server, but it runs most of the time, and so I just cron job on my laptop, and I let it run things. But then we ask ourselves, “Well, how do I email myself? What’s the easiest way to get myself a notification?” And I’ve been doing this lately, looking at like “Well, how do I push-notify myself reliably, from anywhere in the world, on a script?” Just like pipe it into something that sends it to a thing, versus me having to do a bunch of other things to get something done. Just like simply set up something small that scratches a particular itch.

And you have cron jobs, you have email yourself… So you have scheduled, you have notifications built-in… And for me, as an experienced developer even, when I think about where, it’s just like “Well, you just throw it on Val Town.” I have an account there, throw it on Val Town, and let it email me. Schedule it with Cron. And so I like that that focus is there from the beginning, where it seems attractive not just to people trying to get started, but also people just sick of having to manage where their code runs all the time. So I have a bunch of stuff that I could easily port, being it’s JavaScript and TypeScript… I could just create a Val Town account and start moving stuff up there, and not worry about “Did that crontab run because my laptop is closed?”

Jeff Geerling: But they just basically threw a nuke into the downstream ecosystem, coupled with the fact that there were so many – in the first two or three days there were a lot of articles… There was one in particular in LinkedIn that tore into Gregory Kurtzer, who I’ve never met; I’ve had one email with him ever, and that was after all this stuff happened… I don’t know much about him, I don’t know much about his history, besides what I’ve read online, but they were like tearing into him, saying he’s like this terrible person, and CIQ is horrible, and the way that Rocky was structured… I’m like “Where did this come from? Nobody ever asked about this, and they’re posting about it.” And then I saw other stuff, and it leads me to believe that this was – I don’t know, internally, I’m thinking that there must have been some conversations with SAs, with marketing, with maybe PR, I don’t know, about like “Here’s why this is happening”, but they didn’t release that stuff publicly. And then there were comments on Reddit from Mike McGrath about some underhanded business decision, and he’s not going to talk about it because he doesn’t want to throw someone under the bus, or whatever… I don’t know, it felt very weird, the whole way that first two or three days went, especially. And then at this point, Red Hat, if you talk to the VPs and see all the interviews, they’re always talking about commercial side interests, and how it’s bad business to let your competitors just take your stuff for free… But they don’t talk at all, they address none of the trust issues, the community issues, the open source and free software philosophies… And that’s, I think, the big difference that I see with the current kind of thing at Red Hat. Versus 10 years ago, everybody was like “Red Hat is the open company.

Simon Willison: That is such a sophisticated example. I love – I feel like systems administration tasks are a particularly interesting application of this stuff. So I’ve been mucking around with Linux on and off for like 20 years, and yet I still have a fear of spinning up a new VPS for a new project, because I’m like “Okay, well, do I have enough knowledge to secure this box? and all of those kinds of things. That knowledge is very well represented in the training set. Millions of people have gone through the steps of securing an Ubuntu server, and all of that kind of thing. I’m just not familiar with it myself, but I would absolutely trust ChatGPT in this case to give me good step-by-step instructions for solving problems on Ubuntu, because these are common as muck problems, right? This is not some esoteric knowledge.

All of this sort of like very detailed trivia that we need to understand in our careers - it feels like that’s the thing that this is automating. Stack Overflow did this originally for all sorts of problems that you’d come into. This is that times ten, because it can come up with the exact example, for the exact problem that you throw at it every single time.

Adam Wiggins: Yeah, absolutely. I think the players in, my mind were… You had classic shared hosts, which everyone knew to be – you know, they were cheap, but they were just really kind of… You didn’t have a lot of control, and CPU was bad, and it just wasn’t good for anything other than a really tiny hobby app.

[17:57] And then you had - VPSes were a bit of an emerging thing. Slicehost was another inspiration for us. I had been using Slicehost, one, just because it was a great product, they just did a really good job with it… But they used virtual private servers to make it – basically, slice servers into smaller pieces, to make it more possible to get started with a small app.

I mean, I spent a lot of my time during that consultancy, and actually for the rest of my career - you know, when I wanted to get a new production app spun up, the thing I would do is order a rack mount server, and then it arrives weeks later, I install Linux on it, I physically like put it in the trunk of my car and take it to a colocation facility… This was the state of the – let’s not call it the state of the art, but that was the standard thing to do. And at least a VPS was a step up from it; it was a little bit more on-demand, even though you still had to wait like 24 hours for your thing to be provisioned, or something. But that was lightning fast, and very low effort compared to racking and stacking your own servers.

So certainly Slicehost I think was an influence for us, at least in the relative ease of provisioning something, and the fact that they could cut the servers into smaller pieces. So you’d get the $10 a month one or the $20 a month one; you didn’t have to go straight to the big, beefy server that was probably in the hundreds a month, let alone the upfront cost of buying it.

Then you had Rackspace, which was also in that same kind of business of racking and stacking managed servers, and had some other offerings… I didn’t fully track what they were, but they were traditional hosting, as far as I was concerned.

And then you had EC2 as the new big player, and that also was a piece of the origin story of Heroku. We had been playing with it as part of trying to compete for the Netflix prize. I don’t think we ranked particularly high, but just for fun, we were competing in that, and used EC2 to spin up some compute, and saw that there was something pretty interesting there, for as low-powered and simple as it was at the time… You triangulate that and Slicehost, you see there’s something interesting happening with this whole virtualization thing, even though it’s very kind of toy-like in the moment.

So I think those are the things going on. And then Engine Yard, yeah, was probably, again, also more on the traditional host side of things, but they had a very high degree of Rails and Ruby focus; they employed people who were luminaries in that community, they did a lot of contributions to open source work… So you went with them because they had the expertise. So more like Red Hat, probably would be the closer thing. But ultimately, I think the thing they offered you was - apologies to any engineering people that might be listening, if I’m getting this wrong, but I think it was basically managed hosting, and had this extra layer of “We’re part of the Ruby community, we’re contributing, and also, we’re experts on this. So when you hit challenges with your app, we’re going to be in a really good position to help you.” And notably, the Engine Yard office was down the road from us, and we hung out with them quite a lot, and we were part of the same communities.

Adam Stacoviak: Yeah. Like I was saying to Toby the first time, if in the future this evolves to the point where when we SSH into let’s say our VPS, for some reason… Like, if I wanna do that, I’ve got my identity attached to it, it’s got a particular UI. Maybe there’s the common everyday command line, but what if we know our stack and so there’s something with the Kubernetes painted glass; we wanna see what’s going on ad nauseam across the deck. Maybe there’s something like that where it’s aware. I don’t know, maybe that’s a thing.

Adam Stacoviak: Yeah. Like I was saying to Toby the first time, if in the future this evolves to the point where when we SSH into let’s say our VPS, for some reason… Like, if I wanna do that, I’ve got my identity attached to it, it’s got a particular UI. Maybe there’s the common everyday command line, but what if we know our stack and so there’s something with the Kubernetes painted glass; we wanna see what’s going on ad nauseam across the deck. Maybe there’s something like that where it’s aware. I don’t know, maybe that’s a thing.

Adam Stacoviak: So we were on The Changelog together around probably somewhat that timeframe. And this is 2019. We published that show August 2019. And as you may know from that show and maybe know of my background, I’m more on the front end, more on the business side, more – a different side of tech. I’m still a software developer, but just on a different space of it. And while I’ve deployed, or I would say, built Linux boxes in the cloud, VPS Linux boxes, I’ve never observed or monitored anything, really. So if I’m being honest with you, I could empathize with that designer so much, because on that show, I was barely hanging on in terms of what observability was. And only now, going back and listening to that show as part of, one, preparation for this show, and just kind of going back to catch up with where you’ve been, because that’s where we’ve been too, I’m like, “Okay, now I get everything Christine had said in this show.” But then, I was like, “What is she talking about?” I mean, not fully, but for the most part I was like, “What is the unknown unknowns? What does it mean to be monitoring a system? What do you mean by metrics, and that’s not enough, and logging is not enough?”

I get the basic premises of those things, but this space you’ve been forging has very much been like an uphill battle, because you’ve coined the term, and you’re telling the market what they need, and it is what you’ve made, and it’s a brand new thing, and you’ve got competitors sort of shifting and saying, “Okay, well, now this is observability, too”, so you sort of have muddy water almost even to this observability space.

So even then, 2018, I was like, “I love you, Christine, but I don’t know what you’re talking about for the most part, in terms of where you’re going.” So now I’m getting it. Hindsight is always 20/20, and you can always connect the dots better backwards than forwards… So I can empathize.