How WebMD ran in the year 2000

Hello, and welcome to Ship It, the podcast all about everything that happens after you git push, and when code becomes software. I am your host, Justin Garrison, and with me as always is Autumn Nash. How’s it going, Autumn?

It is Sunday and I have lots of coffee. So very well.

How’s it different than a normal day?

Because on Sundays I do fun nerd work, and not like corporate America work. It means that I have so much more time to do all the – no, no, you’re right. It’s the same thing. We’re parents.

Okay. I’m just making sure. I’m not throwing shade, I’m just…

I thought about it and I was like “You’re doing the same stuff you do every day.” Dang it, Justin, don’t ruin this for me.

It’s okay, you can like Sunday. It’s fine. I’m not trying to throw shade.

I was like “I have coffee, and I get to do fun nerd things.” And I was like “Wait, my fun nerd things are a lot like my corporate America nerd things… And parenting.” Dude, my kid had his first soccer game yesterday, and it was a bunch of three and four-year-olds just like running around with soccer balls. It was the cutest hot mess ever.

We use game in a loose term there.

Dude, the dads take it so seriously… Like, I spent half the time making fun of the… I was like “The kids are cute, and all of the dads need to calm down.” They were so into it. I was like [unintelligible 00:02:11.27]

Watching my kids play sports, one thing I realized is it doesn’t matter – like, it is cool to see someone at the top tier athleticism or whatever, but like, if competition is equal, or even if they’re just having fun, it’s still fun to watch. It’s just. I’ll just go and I’m like, it doesn’t matter if you’re a pro athlete or not. It’s cool.

Dude, three and four-year-olds are hilarious. They are like the end of the good things in the world, okay? They don’t know enough to try to act cool… Like, everything is just – it’s like [unintelligible 00:02:43.15] It’s great.

Yeah. There’s no pressure except for just to live in the moment.

Yes.

And it’s just like “Hey, guess what?! Taxes are coming for your, kid…!”

And they think everything is hilarious, and they have like the best jokes, and they say the most ridiculous things. They’re my favorite.

So that’s what’s going on in our lives today… And we are recording on a Sunday, but this episode is for Rich, Rich Burroughs. And Rich is a friend who I invited on because he has a wealth of old history, lore and knowledge about how things used to happen on the internet. And this episode is all about how he used to ship webmd.com.

Which is also super-cool and has the best glasses.

Yes. He is a really cool guy just in general, and has great glasses. His beard is amazing.

Before beards were cool, Rich had beard. Oh, and not just that, but like, his Bluesky tweets are on fire. And he makes infrastructure cool.

Yeah, so we had a great conversation with Rich, and we hope you enjoy it, too. It’s just fun. Again, this one of our retro things, of just like looking back on how things used to work, and there wasn’t all this complexity and things that we do now… And maybe it was a simpler time, maybe there was less money involved, and risked, but also maybe we look back on it as more nostalgia than we might today.

I love that he got into it because he thought it was cool, and then his interest in it landed him into like a job. And the fact that just all his stories are like “You had how many boxes, and you ran a whole website?” [laughs] Insanity. I was like “How did everything not die?”

And the secret is it always was dying, but people will prop it up.

Basically.

So to start the show today, like always, we have some links of the week… And my link is from theverge.com and it’s called “The cloud under the sea.” I love this article, because it was more of a visual experience than an article. It’s a very long article. It is it is extremely long. But the design on the page is so great. Like, it’s just more of an interactive magazine… Which I know The Verge does quite a few of these long expose pieces.

I love interactive… I think it’s so much fun.

Yeah, but just in general, the visuals, they were really good. This is actually something that I wish was like a documentary, because I would have loved to see more visuals around it. I know it’s kind of hard to show fiber lines under the ocean in a video, but just what you’re dealing with on ships. And so “The cloud under the sea”, a great piece just about how a lot of the infrastructure that connects datacenters and countries together is really just these small garden hoses under the sea, and where they go, and how people repair those. Specifically how the ships kind of repair them, and work.

There’s also a book called Pipes, which is another great book about this same network, in book form. And it does a lot of how a datacenter connects to a different data center. So they walk the whole path, which is fun, because they have like – when it comes up on a beach, what do you do? How do you actually terminate that and like make it go somewhere? Do you connect it in the ocean water right there? It’s kind of shallow… What do you do? So the book is also really good, but for the link I’ll go with “The cloud under the sea.”

[00:06:05.09] I like to read stuff like that to my kids, because I think the magic in the world is learning that these invisible things are things that you take for granted, so much work goes into it. I used to love that show - what was it called…? How Things Work, as a kid… And just like learning the secrets, and… I think it gives you more appreciation for different people in different jobs. Also speaking of appreciation for different people in different jobs… You can tell that a designer, and other people had input on this because it’s visually beautiful. And the visuals really drive the point home in the article. So not only is it like technologically interesting, but it visually puts the dots together, and also it’s just… I don’t know, I think it makes it richer. The experience is richer, and a lot of times visual arts and designers are always like “Well, we can get rid of them”, but it’s crazy how much it adds to the experience, you know?

And this is one of the pieces that I love reading on my iPad, but usually it’s just text… And this was one of those – it felt like what reading on an iPad should be.

The whole intro part is just like “What’s gonna come next?” It’s so pretty.

As you scroll, things happen with the scrolling, and things come into view… And all that was a lot better than reading it on my computer even, or even on my phone, because it’d be too small. It was like this perfect moment of like I read that whole thing on my couch, on my iPad and like “Oh, this is great.” Like, this is close to a novel. And I already knew a lot of the technology behind it, and again, reading the books and seeing other things, but it was still fun just how they followed these people that were on the ships, doing repairs.

See, I have too much ADHD for TV, because I have to stop and start things a million times as a mom, and it gets so frustrating, that stopping and starting and stopping and starting… You remember we were talking about on TikTok about how many times you have to get up when you have children…

Yeah, I was gonna say…

…and you’re saying this would be a great documentary, but for me the fact that I can get that visual experience and put it down, and pick it up in between making my kids macaroni and cheese - you know, sometimes those are the only really visual cool experiences I get, because it’s like you’re running around from work, and soccer, and this, and that… So I am so here for it. I really appreciate all the effort, the little details that people put into it to make it that cool and visually stimulating.

So “The UK introduces its first IoT security laws.” This is IoT Tech News. I thought this was interesting, because the fact that technology has advanced so much in the last few years, or just overall the last century or so… Our laws have not caught up to it. And I don’t know if our lawmakers understand enough about technology to make laws for it. Did you see Nancy Pelosi and all the other people talking about algorithms not too long ago, about TikTok, and we’re like “She has no idea what an algorithm is”? And it’s crazy that this is such a big deal, to do IoT security… And it’s the first law, but it’s really, really simple things, that you would think that you wouldn’t have to tell people, but we all know those are the things you have to tell people, because usually the biggest security hassles are something that they just walked in the front door because people didn’t make good enough passwords… So it’s interesting that some of them are “Just don’t have easy to guess passwords, don’t use 123”, but that had to be a law, because people will do it, you know…

The other ones are like publish vulnerability disclosures, state minimum periods for providing secure updates, and provide mechanisms for securely updated software… But what I’ve found interesting is the password section of it came from the Mirai Botnet incident, which is a DoS attack that was super, I guess infamous… But I just thought it was cool, because I loved the Georgia Tech write-up that was done about it. So it’s just crazy to see people go and write these great post mortems, and to see actual change in laws come from them.

[00:10:05.05] Of those bullet points in the article too, like “Publish vulnerability disclosure policies for reporting security flaws.” They didn’t have to do anything. They didn’t have to have a blog post to say “Hey, by the way, this is vulnerable.” And the second one, “State minimum periods for providing security updates” - again, they don’t have to do anything. They just have to say “We are going to have 12 months of security vulnerabilities”, or something.

And then that last one on there is “Provide a mechanism for securely updating software.” That’s the first one that like “Oh, they have to have a way to update it.” Like “This is the thing that’s wrong, this is our guarantee for this thing.” I wonder how this applies to things that happen… Like I know California is pushing out a really big law for right to repair, and right to be able to own some of the hardware and stuff, and be able to fix it, and be able to “Hey, if this is vulnerable, I should have a way that I can get into – even if you don’t provide security updates, I should be able to push in my own security updates if that is something that is outside of your cycle of maintenance anymore.”

And so I’m wondering how this kind of plays into that as well, where it’s like “Hey, on one hand they need to be responsible for some amount of time”, and then beyond that, a consumer or someone smart enough, or a community should be able to take over and provide the other security updates and things… Because I have plenty of IoT devices that are long past their lifecycle, but I’ve been able to flash them with different firmware, or close them off from the internet, and do things that I have some control over, but not necessarily like route them… Not all of them, at least. I can’t take full control. So…

I remember when I first had my kid, and it was the baby monitors that you could kind of set up on your own network got really popular… And even the ones that you couldn’t, people were driving through with the remote ones and connecting to people’s baby monitors, and it got to a point where there was one article, I think, where people were talking to people’s children…

When they had a two-way, yeah.

Yeah. And I’m just like “Guys, you cannot just set it up with the default [unintelligible 00:12:00.09] Like, I had to tell all mom groups to stop doing that.

Well, like the 900 megahertz… It was completely unencrypted in the clear, and if you had a way to pick up that – you could just use an SDR, Software Defined Radio, and pick it up and just “Oh, I can send information to this same channel.”

Dude, I have cameras still in my garage in the boxes… And they were a bunch of like nanny cams. And I was like “This is great. They’re super-cheap. Why are they super-cheap?” And I got them, and it was like “Cry detection software”, but it’s also a bunch of spyware on them. And I got halfway through the setup and I was like “Nope.” [unintelligible 00:12:36.02]

I’ve been wanting to do this series on YouTube or TikTok about just reading the open source licenses in some of these products; just like going to the manual… Because being me, I’m sitting in my car waiting for my kids at school, and I’m browsing through the licenses that are in my infotainment system. It’s just like, it was there, and I was like “Oh, what software is in here?” And I’m like “Oh, there’s 18 different Android [unintelligible 00:12:59.21] Oh, look at this SSH. Oh, wait, I know that version of SSH. That had a bug, right?” And so I’m looking at it, because it’s the versions, the dates, the license and everything else in it… I’m like “Someone should expose –” Like, not expose it, but just like talk about it. Like, this exists in everything.

Yeah. I want y’all to know, these are the gems of being friends with Justin. When you get a random text, and he’s like “Did you know this is in there?” And I’m like “No. Tell me more.”

[unintelligible 00:13:21.19] screenshot…

Yeah.

“What is this doing in my refrigerator…?!”

No, but dude, I got some warnings about updates that came over my refrigerator, and I now that’s like a record feature… And I was like “Whoa, whoa, I didn’t ask for this.” Like, I wanted a Twitter fridge with a screen on it and Alexa. Now you’re getting too deep. We’re too deep now. I was like “Oh my God, I’m turning into one of those people in tech who’s like “Let’s take all this stuff out. I don’t want any more smart home stuff.” I was like “I’m becoming one of those old, salty tech people. I didn’t know I was gonna get here.”

I’m at the point where I still want them for the convenience side of it, but I don’t want them connected to anything. Like, I want to own both ends of that.

[00:14:02.26] I remember when I first got into tech seriously, and everybody was like “I don’t have any Alexas”, and I’m like “How can you not?” I was obsessed with IoT, and smart homes, and all of the lights and stuff… And now I’m like “Oh, we’ve got to find a happy medium.” Because they just keep updating more and more stuff, and just sending you stuff without even really telling you what these updates are doing, and you’re like “Whoa…”

My youngest child’s first word was Google. Literally. Because we used to say “Okay Google” all the time - I’m sorry, I’ve just triggered my own - for music and whatnot. And that was his first word.

It’s wild… Between my first kid talking to Alexa, and my third kid, the way that Alexa understands voices is like a game-changer. And then like just the fact – like, it’s part of their life. They grow up talking to Alexa, and… Can you imagine if we grew up – like, little us’es would have been like “You did what?!” It’s amazing. Like brain melting. So it’s crazy; the way that kids can use an iPad, the way that kids can use Alexa’s… When we were talking to Mandy, I was thinking, that was one of my first hardcore exposures to the internet. You could talk to all the little nerds that had the same interests as you.

They all got an AOL CD from the grocery store, and they’ve put it in for their 90 minutes free…

My son is going to a robotics camp, and he’s gonna build like those battle robots… And I’m like, “This is where he finds his people”, you know what I mean? And I remember being that kid, and like being the nerdy kid, and then you find people that are into art, and technology, and these things that you’re into, and that was like one of my first exposure into finding kind of like my group of people… And it’s crazy - as adults, we still use Twitter or Bluesky, and it’s kind of like, they were exposed from it from day one. It’s insanity.

So speaking about back in the day… Let’s jump into our interview with Rich and all about how WebMD used to work.

Break: [00:16:09.15]

Alright, thank you so much Rich Burroughs for being on the show today. I am super-excited to talk about this. Rich, you’ve been around technology for a little while, and you’ve been doing a lot of stuff…

A little while.

A little while. Because when we reached out – like, this is one of our retro episodes, and I’m really excited talking about what you used to do with technology. So tell us what you did, what were you doing, I don’t know, 20 years ago, 25 years ago… When was this?

I mean, I started off in the mid ’90s.

How did you get started?

It’s kind of a funny story… So I had a friend – I didn’t have a computer at the time; this was like early ‘90s. And I had a friend who had a Mac 2E, or something like that, and he would connect to a local ISP through dial-up… And at that point, there wasn’t even pictures, right? It was just all like text-based stuff. And he was playing [unintelligible 00:21:04.27] which if you’re not familiar with those, they’re a text-based game like Zork, or something like that, but multiuser, right? He was playing one of those, and I got kind of addicted to it… And I finally got a computer and was playing it, but sometimes the ISP server would get laggy. I was dialed into like some old Sun server… And I heard about this thing called Linux that was pretty new, where you could actually run it on your own computer, and then run the client on that, and so the lag wouldn’t be as big of a deal. And that was how I got into Linux. I was involved with that for a few years.

It’s kind of funny, because back then being a sysadmin was like a dream job for me… Because there just weren’t many of them. To be a sysadmin, you probably were at a big company, or especially at a university, to be at a place that had internet access… And I kind of ended up falling into it sort of backwards. I got a job as just a tech support person at an ISP. We were doing things like changing people’s modem init strings, and stuff like that, just helping them with connection issues… And after a while, they just realized that I knew a lot more about Linux than they thought, because I’d been just playing around with it so much. And that was the very early days of Red Hat, and I was using Red Hat, which is what they used… And my boss knew more about Unix in general than I did, but I knew more about Red Hat… And so I would be showing him stuff about RPM and things like that. And so finally, I got promoted to be a sysadmin, and that was where I started. That was probably like ’96.

That’s awesome that you got like your dream job.

Yeah. I mean, it’s funny looking back on it, because I would never take that job today. [laughter] But back then I just thought that sys admins were just the coolest, right?

I was talking to my friend about that… Isn’t it crazy? You get like this dream and you’re like “I want this job”, or this thing, and then you get it, and your dreams change over time…

Oh, absolutely. And things were so different back then. At the ISP that I was at, I think that when I showed up at first we had like maybe five servers, and there was like no load balancer or anything… They were just like these individual servers sitting on the network.

And you were web scale, right?

Yeah.

For what the web was at the time, that’s all you needed.

Yeah, for sure. We didn’t have a firewall.

There was no firewall, no such thing. Like, Windows didn’t – Windows, whatever, like 98 or something like that… No firewalls until – like XP Service Pack 2 had a firewall. It was just like “Yeah, you just open ports all the time.”

[00:23:51.01] Yeah. So when I showed up, there was no sysadmin, and there hadn’t been for a while, because my boss had fired the last sysadmin. When I say my boss, he was the owner of the company. Because the previous sysadmin apparently would do this thing where he would say that he had stayed up all night doing maintenance, but he was just playing Duke Nukem or something… They came in and he was like asleep at his desk, with like the game still up… So they fired him, and there was months where they didn’t have anybody patching the servers, and stuff like that. And one day I’m talking to my boss, and he says “Oh yeah, I’m looking on our mail server and somebody’s FTP-ing off the password file.” And for those of you who know about password files in Unix, this is like the very early days, where the passwords weren’t even hashed yet… So I mean, it was literally the plaintext passwords in the file. And this guy is downloading them all.

Oh, no…

No shadow, no salt…

Yeah.

Wow.

So I was like “Man, you’ve gotta let me do some stuff.” And it ended up like – you could do shadow passwords, it just wasn’t the default… So I ended up doing that, and I ended up doing the Linux firewalling, and that thing called the TCP wrappers, where you could also restrict what could connect to what, and I nerded out on security stuff for quite a while then. We had collocated servers. So a customer would pay us to have a server on our network. And a lot of times, they would pay us to actually build the thing, too. So I would build them a FreeBSD box, or a Linux box or whatever, but we didn’t manage them. [laughs] So these boxes would again just sit there and not getting patches or anything. And so we had a number of customers who had like security incidents, where I would end up being the one who had to clean up afterwards, and reinstall the OS, or whatever…

I feel like that has gone in waves throughout the entire history of the internet, and people were having services and whatnot… Because early days it was just like Linux and Windows boxes directly on the internet, with no firewalls, no security around them… And you could literally see dollar signs, someone else’s computer in your ISP. My neighbor’s file systems were just exposed to whoever else was on the network of the ISP. And then it went through waves of like Wordpress [unintelligible 00:26:19.03] things that were like hosted services, that like “Oh, you are kind of responsible for this.” It was your responsibility to do something, but no one knew how to. And I feel like we’ve gone full circle again with things like Mastodon and “Hey, I want to run this as a server, and this is my data.” It’s like “Yeah, you’re responsible.” There’s this whole thing that people forget.

That’s because developers all of a sudden – my little brother’s building his own computer, and him and my kids are building their own raspberry Pi, and their friends are building their own like Linux computers on like crappy old computers they’ve found… And it’s so funny, people are obsessed with having options. And I’m like “There’s a reason why we got rid of – and got things to be managed for us, and more automated, because it sucks having to think about all those options.” But every wave of like wanting to be able to touch all the buttons, and people get so into it, and you’re like “Dude…”

I mean, there’s definitely excitement in building and learning new things. And the maintenance side of it is not one of those usually exciting parts.

So when I worked at that ISP, I had my own server on the network. That was one of the things my boss let me do. There were no restrictions on my bandwidth, or anything like that… And one of the things I did on that server was email. And it’s just so funny, because nowadays I would never want to own a mail server. I’ve been paying people to do that for me for so many years now… But back then I just loved it, and…

So you were at this ISP for a little while, you’re learning a bunch of stuff… And at what point did you move over to WebMD?

That was like about ’98… So I learned my first big lesson in grown-up pants, big boy business stuff, which was “Get everything in writing.”

[00:28:06.00] Oh, no…

So our boss had been telling us at the ISP for years, he’d been telling me – because I was a really key employee there. I was like the sysadmin. And he’d been telling me for a long time that “Oh, someday I’m gonna sell the company. I’ll make sure that you get something. You’re gonna get enough money to buy a house, or something like that.” And he sold the company and I got offered twice as many stock options in the new company as a new employee would have. So a new employee would have gotten 1,000 options, I was offered 2,000, with no vesting at all. The clock would start that day. And I was just like “You’ve got to be kidding me. I’ve been working at this place for three years, and done so much…” So I started looking for other opportunities, and I happened to meet one of the best recruiters that I’ve ever worked with in my career. She was a third party recruiter, and she was just fantastic. She was like “What do you want to do?” And that’s not a conversation that recruiters have all the time, really. Like “What exactly are you interested in? What kind of companies do you want to work at?”

And she ended up giving me an in there at WebMD. And I was really excited to work there. And it’s kind of funny, because a lot of people still know about WebMD. They still go there and google whatever ache or pain they have that day… But for frame of reference, it’s kind of hard to understand now, but that company was so big then. They were taking out Super Bowl ads, there were people who described WebMD as potentially being the Microsoft of healthcare… And the reason why is it wasn’t just the consumer stuff. It wasn’t just the website. They were going to work with physicians, there was a different site that physicians could use, and patients contacted them… It was basically going to be like MyChart is nowadays, if you’ve ever used that… But that, of course, didn’t take off at that point in time.

It was really early to digitize that information, right?

Yeah.

That was still – late ’90s it was like “What does the security internet look like?” Dan Cohen had the first purchase, like credit card purchase on the internet - it was like around that time.

That would have still been when I was at the ISP, because even there we were dealing with – it’s kind of funny, when the whole cryptocurrency stuff happened… Because in like probably like ‘97 or something we were dealing with this thing called cyber cash, that was like one of the early encrypted currencies… And I was big on – there was this email list called The Cypherpunks email list, that was like the crypto nerds, and I was very big into that stuff. I would go to like PGP key signing parties, and stuff… And so it was funny when the cryptocurrencies stuff came up, and everybody was like “This is brand new.” And I was like, “No, it isn’t.”

Which is like “You’re not old enough. This is –”

Yeah… I mean, the thing that was brand new was the blockchain part… But yeah, so I go to WebMD… Like I said, it’s this super-big company… I come in as like – you know, even though I have a few years experience, I’m still sort of a junior-ish sysadmin, like maybe mid-level at best… And it was a Solaris shop. So that was brand new to me. So I had done all Linux stuff up until that point… And we did have some Linux servers, but it was mostly Solaris. And that’s a whole different world, right? We would have these big sun boxes that had like 10 CPUs, and every CPU was like a whole shelf… You could pull them out, and hot-swap them, and… Yeah, I didn’t touch any of that stuff; they wouldn’t let me touch those big boxes.

[00:31:50.03] But yeah, and when I started off there, I was doing a bunch of different things, sticking CDs in sun boxes and building them… I ended up building like a little kind of cluster of Linux boxes that we use for like load testing, that would just run the load testing apps… But then eventually, I got into doing deployments. They just had me do one or two on the physician site, which wasn’t used as much… And then I ended up doing it on the webmd.com site. And that was kind of crazy, because that was such a high traffic site… Due to deployment, it was like a big deal. And when we would do a big release, we did them in the evenings, and like everybody would come in, and we’d all be sitting at our desks at 9pm, doing this big deployment.

Right, because you had a maintenance window, right? Did you take down the site? You’re like “Can we put up a splash page that points somewhere else?”, or…

No, we didn’t have to take things down. We would do kind of a rolling thing. It’s funny, because – and I’ve talked about this some. I have a talk that I do, where I mentioned this… People nowadays, they think of a website like that and they think of like tons and tons of hosts, or containers or whatever that are taking traffic… We had three sun boxes that were in the farm for that site. They were all in one load balancer form. And the deployments at that point were running some shell scripts, and I would literally take one of the three boxes out of service and run the shell scripts on it, and then move on to the next one, and roll through them that way.

Yeah, it was a fun place to be at. There were a lot of really smart people there. One of the guys on the team, one of the – kind of a senior engineer… Well, we were kind of like a Java shop, right? Not kind of. We were a Java shop. It was –

Solaris and Java, right?

Yeah, exactly. And one of the guys there, the senior guy had actually been on the Java team at Sun, and so he knew tons of stuff about Java. But also, we had kind of an in with Sun… So sometimes we would like report stuff to them that would get patched, and things like that. But the craziest part of that was that we had our own web server that was written in Java. So the guys there had written the web server that we used.

We just had Mandy from AOL on… Well, a couple episodes ago. And AOL had their own web server, which is now open source, which is awesome. So early days that’s what you had to do, because there was no generic “This is going to serve generic traffic behind the scenes.” And Apache came out, and as that expanded into having all these other plugins and modules, it got a little complex. But like these single purpose sort of web servers were seen like they were prolific. They weren’t shared, but people just like – it wasn’t even open source at the time. Open source in the late ‘90s wasn’t like a big thing. There wasn’t a place to find all this stuff. So your bug reports to Solaris and stuff is like “Here’s an email with this patch diff on a memory dump”, or something.

Right, exactly.

People weren’t really using GitHub in the ’90s, right? Was GitHub even a thing?

That’s what I was saying.

Oh, no.

That’s crazy.

This wasn’t even like – Sourceforge wasn’t available at that point yet, because –

When did GitHub get really like popular, I guess, with like using and sharing open source? I’m trying to think…

I can’t remember.

Also, another thing that’s cool is in WebMD, and Yahoo, and AOL were such a big thing in the ‘90s… And it’s crazy to see how these giant companies are – I mean, they still exist, but they’re not what they used to be now. And then you wonder what’s gonna happen with the companies that are so huge right now, that seem like they could never fail, or never be lesser than they are… It’s amazing.

Yeah. I think it’s kind of different now with some of these companies. They’re so entrenched, something like Facebook… Or Meta, excuse me. I still can’t call it Meta.

It’s like when people call Twitter X, and I’m like “I can’t do it.”

[unintelligible 00:35:46.11]

It’s Twitter forever.

Git came out in 2005, GitHub was 2008. So I mean, were you using like CVS? What was that deployment script doing? Was it just like –

[00:35:59.27] CVS, yeah. So for those who aren’t familiar with it, CVS was one of the early version control systems. There was a thing called RCS before that… And yeah, the big difference between CVS and Git is that when you make a branch in CVS, it literally makes a copy of all the files. So it’s not like Git, where it just knows the differences between branches. If you have a really big repo, and you made a branch with CVS, it was like go to lunch, go take a two-hour lunch or something, because it’s gotta copy all the files over.

The merge conflicts that must have happened…

Yeah, yeah…

But the fact that you were using version control at that time –

That’s pretty cool.

Because that was not common for a lot of shops. This was like “Here’s a directory. We all either SSH to the server, or SCP these files around”, and that’s how you managed source code back then.

I mean, I think that we did SCP the files around for the deployments, but those were like the tar balls, or whatever. We did have a build process.

Yeah. The artifacts of whatever the release was is like SCP-ed…

Exactly. Yeah, no, I think that for the time it was a pretty advanced shop. So the office that I was in here in Portland had been a startup that WebMD acquired.

Oh…

And so, you know, a lot of sharp people there… And they made all the technology that the website ran on. And when I showed up already, this stuff was in play, where – Microsoft had invested a bunch in the company. I can’t even remember the number, but it was huge. And they got a seat on the board for that… And the deal was that the company was supposed to move over to use Microsoft for all the stuff, like running the website, and things like that. And for anyone who lived through that time period, you will probably be chuckling pretty hard at that idea, because doing web surfing with Windows - it was just a nightmare back then.

IIS came out – was that with a 2000 server, or…?

No, it was before that, because again, we were using IIS even at the ISP that I was at. But it was that running on NT4. Oh, I have a funny IIS story. So when I was at the ISP, there was this team at Microsoft that was created specifically to market to ISPs. And so they wanted ISPs to make an install disk that had Internet Explorer on it, and all that stuff. And so they were saying “Oh, you’ve got to get this new version of IIS. It’s got this new stuff in it.” And my boss got really excited about it. And so I was going to upgrade our IIS box, or one…

Yeah, the one box.

Microsoft web server… And I go and I run the upgrade thing on it - and we did declare a maintenance window for that. And so I run the upgrade on it, and at one point it does the thing where it checks for disk space, and it’s like “Oh, yeah, you’ve got enough disk space.” And then it runs for a couple of hours and it gives up and it says “You don’t have enough disk space.” And it turned out when it was checking, doing the check, it was literally just saying “Do you have enough room on the file system for the files?” But IIS had this internal database that kept the state of everything, and that database itself had to get migrated, and there wasn’t enough room to like do the migration.

And so it crapped out, and I had to go to tape, and restore everything from tape… By the end of the day, I’d worked on this for like 10 hours straight, and I was just completely fried. And I get home, and my girlfriend at the time - we’d lived together for a couple years… She wanted to have the relationship talk.

Oh, geez…

No, not after all that… [laughter]

And I was like “I just can’t do this. I can’t do this right now.”

[00:40:02.07] “Let me tell you about IIS, okay?”

Yeah, let’s talk tomorrow, or whatever. She went and stayed with her parents for the first time that night. She had never been like “I’m going to my parents”, and she dumped me the next day.

Oh, no… I feel like that’s one of those –

Rich never ran IIS again.

It’s one of those unfortunate things that line up… Poor Rich.

I mean, obviously, there were things that led up to that, right?

But still, that’s like –

I can’t blame it completely on IIS. But yeah, so that’s my story about how Bill Gates ruined my relationship.

Which is funny, because that story has like played out again and again. Especially Microsoft – like, Microsoft, a lot of investment in Open AI. And guess where Open AI runs? Azure. And that play has been pretty common in the industry, of like “Hey, we’re gonna give you some money to do stuff with our products, and promote our stuff.”

Yeah, there may have been money involved, too. I never heard about that part of it, but it’s very likely that that marketing team was offering people money. So I’m with WebMD, and there’s this push to get everything onto the Microsoft stuff. And we were all just laughing, because we just knew that it was just going to be brutal.

There was a whole separate team down in Santa Clara or somewhere that was like working on getting everything up and running on Windows… And I was there for, I want to say two and a half years, and they didn’t get it done during that time period. And one of my friends told me later on that they finally got it migrated maybe like two or three years after I had left… So yeah, it was just ridiculous. A crazy idea back then.

Funny enough, I’ve heard a story - and I can’t say if this is true or not… But when Elon Musk got kicked out of PayPal – so you know, he had his own company, which was called X…

Yeah, because he owned the domain…

…and got bought, or merged with another company, and then they call it PayPal, right? And so he eventually gets ousted as the CEO. And Peter Thiel was involved, and there was a bunch of politics and stuff like that… But apparently, one of the factors is that Elon wanted to move everything over to Windows, which was that same time period. Yeah, it was just, again, a ridiculous decision. So even before the Twitter stuff, when I would hear people talk about how smart Elon was, I would just think back to that.

But yeah, so I was there for like two and a half years, and it was fun, it was a really great environment. A lot of really smart people. Like I said, really fun people on my team. We would do stuff like ping flood each other, and screw around, change people’s desktops to be something stupid, and things like that… It was a fun time.

And then the company was actually kind of on the frontend of the bubble bursting… Like I said, there had been this merger right before I showed up, with this other company, and we didn’t know it at the time - they definitely didn’t tell us - but it was less of a merger than an acquisition. WebMD was just in trouble financially; they had just spent too much money, and the physician thing hadn’t panned out, like I said… And we merged with this other company, and there were co-CEOs for a little while…

That’s how you know it’s a great merger.

Yeah…

No good can come from that.

Yeah. And then eventually, the WebMD CEO went away. This guy named Jeff Arnold. And he started a company after that, that was like – I can’t remember the name of it, but it was like you would get a soft drink at McDonald’s or somewhere, and the top that went on the cup had like a disc in it, a little music disc that you could pull out and play…

[00:43:58.05] What?! Like the lid had like one of those mini CDs in it?

Yup. So that was his big idea after WebMD.

But why?

It’s an excellent question, Autumn. It’ an excellent question.

Some of the startups that you hear… I’m like “Somebody funded this?”

I’m sure it had some really goofy name, too. I can’t remember what it was. The other funny story about Jeff Arnold is that – so the company was based in Atlanta; he owned a mansion that for a while it had been described to me as being the mansion from Gone With the Wind… And I later on – I don’t think that was true, but it was basically that mansion. It was very much like it. And at one point after he was ousted I think, and when things were kind of rocky with the company, his mansion burned down.

Geez.

And people were like “Huh… That’s kind of coincidental, that suddenly a bunch of your money went away, and the mansion burned down…”

Rich, you have the best Tee for WebMD. [laughter]

I’m sure that was accidental…

Accidental…

Any rumors about arson definitely are not true.

Okay, wait, I have to ask like a total petty tech Twitter question. When you saw the video for WebMD, when they were trying to force everyone back into office, did a part of you die on the inside? Because I think we all cringed so hard…

I’m not actually sure if I even saw that.

Oh, my God, Rich, you have to go watch – it is so bad.

Was this like a video they made for the employees?

Yes…!

Did it accidentally get published?

Yes!

They published it publicly on YouTube or something by accident, and…

I don’t know if they published it on accident or what, but I want to hope that it was on accident, because I want to hope nobody saw that video and said “Let’s show the world this.” Because if they did, then I have questions.

It was very cringe executive-out-of-touch…

I need you to find that video ASAP.

I’ll have to watch it, for sure. Yeah, it’s kind of funny… So at one point - this was years after I had worked there - they did open up another office here in Portland. And this was after they were on the Microsoft stuff, and everything. Probably like mid 2000s…

So they actually got all the way on, and migrated it?

Yeah, yeah. So they did open up another office here, and at one point somebody – there was a job there when I was looking for work, and somebody was like “You should go work at WebMD.” And I was like, “Nope.”

“Sorry. No.”

You’re like “Hard pass.”

Yeah. So like I said, we were kind of on the front edge of the bubble bursting… Because we’d had this bad merger, and everything… The company wasn’t doing well… Part of the merger process, for anyone who’s ever not been through it, is that usually one of the first things they do is like they do the efficiency studies and everything, and then they decide which offices they’re going to close… So there was a legendary website back then that was called A F****n Company…

[laughs] Can we bring that back? Someone buy the URL for this.

Oh, I think somebody else [unintelligible 00:47:20.12]

Yeah, I’m sure…

So the site isn’t there anymore, but I think that he still has the domain.

We need a petition to bring this back, because this is great.

And it was basically, you know, [unintelligible 00:47:31.21] about companies that were in trouble, right? Startups that were in trouble. This was before the bubble burst, and…

I feel like tech needs this website right now.

[00:47:43.04] I agree completely. It would have been very valuable in the last year or two. And people would do stuff like they would leak internal memos and emails about which companies were laying off people, and what terrible things they’d done… So yeah, things were already crazy back then. And one of my favorite companies in that period was this company called Kozmo. It was basically like DoorDash, or something, but they had actual employees. So it wasn’t random people. Like, they had employees that they paid, who I’m assuming got benefits, and all of that, and they would deliver your lunch to your desk, or whatever. And so they were another company that went under because apparently that business model did not pan out.

What’s something you learned from those days running that infrastructure, or trying to do things at that scale at the time, that you kind of keep with you today whenever you’re kind of evaluating technology, or just running systems?

Huh, what did I learn…?

It’s like, there was a lot of smart people, there was a lot of stuff going on, and you were kind of at the forefront of a lot of it.

Yeah, I mean, I think that one of the big things back then was scaling, right? And especially running at the kind of traffic that we got - I don’t even remember how many requests it was a day, but it was in the millions. And so back then, the scaling was all vertical. So like I said, we had these giant sun boxes, and there were three of them, and that’s how you scaled. And then probably in that period, in like the late ’90s, was when the horizontal scaling started taking off more. And that was the – we had actually told the WebMD people, the higher-ups there that, you know, instead of doing the Microsoft thing, that what they should be moving to is a fleet of Linux boxes. But yeah, I think that the importance of the scaling and the load balancing and all of that stuff was probably the biggest thing that I took away from that.

It was funny seeing – when I was starting as a sysadmin, and like config management was coming out, and I saw all these companies that had moved to Windows stacks… Like, Open Table was a notable one, and WebMD… And there was no config management for Windows, ever. They were still in this like “Oh, we have to do these golden images with like some bat scripts.” That was like as much as you could do. And then it was a lot of manual work, a lot of clicking, because PowerShell wasn’t a thing, and they didn’t have APIs, or command line scripting to do that stuff.

It’s crazy that there was no PowerShell.

Oh yeah, that was many years before PowerShell. Yeah, I’m trying to remember the name of – what was the thing that preceded like Puppet and Chef and that stuff?

CFEngine.

Yes, CFEngine. So I actually played around with that, someone I was at the ISP that I was at, and ended up just having some hard coded config files like the host file and things like that, that I would use CFEngine to deploy out to like–

No reliable DNS, right? Like, host files is where you lived if you wanted to do that stuff reliably. So many years I did not work in a shop that had a reliable DNS infrastructure, and so it was just host file management.

That’s a super funny point, but totally true. You didn’t want to rely on the DNS.

And most of the time it was because it was an Active Directory DNS server that never worked for me… But even if it wasn’t that – if it was like a bind server or something, when that has maintenance, everything else breaks, and I need to have some reliable way of replicating what’s in bind, and storing it locally, so I know what IP address I’m gonna hit. And that was like the longest time… And even today. I mean, we’re kind of doing it today with Kubernetes. IP addresses for like the Kube API is like hard coded in there. Like, you don’t want to rely on this discovery system. At some point, you just want the actual like [unintelligible 00:51:34.17] ARP table to say “Hey, who on the network has this MAC address? Let me go there”, and that’s what you do.

[00:51:42.28] Yeah, there’s something else, too… We were talking earlier about the early days of Linux and stuff like that, and one of the things that’s a pattern that’s repeated itself a lot in the industry - or in technology in general - is that things start off open… So the early versions of Unix had basically no security at all; you could just do all kinds of crazy stuff. And like I said, even with those early versions of Linux, there was no shadow passwords, or anything like that… And so I think that things tend to start out open, and then as a response to the stuff that the hackers do, they get locked down more and more.

Well, it’s hard. That security side is very difficult, and it adds more barriers… Because managing password files that worked with the shadow file and weren’t hashed was a lot easier as a sysadmin. Because like [unintelligible 00:52:30.05] and there you go, your password’s changed. That was the process.

Yeah, exactly.

And that changed a lot, where you’re like “Oh, I’ve gotta run this password command. How do I do that?” You have to look at the man page, to do it for a different user… “Do I switch the –” All that stuff became more difficult. And at first – it was funny, because I thought you were gonna say things started more open, and then IBM buys them. Because they [unintelligible 00:52:50.12] Oh, sorry. That was a bad joke…

Oh, wow…

Oh, no…!

Too soon. [laughter] Too soon.

The wounds haven’t even healed yet, Justin…!

That where I thought you were going, but…

I did actually have a thought when I saw that news. I was wondering “Huh, I wonder if that process was already in flight back when the licensing changes happened.”

Instead of “sips tea”, we’re gonna say “Justin sips Dr. Pepper” after he says [unintelligible 00:53:20.02]

Well, Justin, when you were talking about that, about the password files and everything - there was a book that I read back then… This one thing has stuck with me forever. There’s this guy named Wietse Venema, who’s the guy who wrote Postfix, which is a very popular…

Mail server?

…SMTP server. Those times were funny, back when that came out, because there was another rival one, and they were feuding a lot…

Yeah. No one remembers the mail server wars…

No, there were some serious mail server wars. But he wrote this book called “Practical Unix Security” or “Practical Internet Security” I think it was called… The title changed at one point, but he had – the metaphor that he used was like a seesaw, and on the two ends of the seesaw were, on one side it was security, and on the other side it was convenience. And so those are the two ends of the spectrum. So something could be “The most secure thing is shut all the computers down”, right?

[unintelligible 00:54:21.18] to go do something, yeah.

But then, you know, it’s incredibly hard to use. And if things are completely open, then it’s incredibly easy to use. So that’s a concept that stuck with me for a long time, is that it’s always kind of trying to balance those things, like how secure can we make something without completely screwing up the experience for users and their productivity.

And the fact that just the threat model has changed drastically from the early ‘90s, when the elite few that could afford either a second phone line, or a modem, or whatever - they were the ones [unintelligible 00:54:57.26] internet. And now you plug anything in that doesn’t have a firewall that’s on the public internet, you are getting scanned in seconds. It’s just like –

I just wonder if being a hacker in the ’90s was more fun because everything was open… But then it was probably harder, because you needed more tools to do that. And now…

I don’t think it was harder at all. There were script kitties back then, you know… And even when I worked at the ISP, I used to hang out on IRC in like these hacker channels and stuff to try to learn about that stuff, and what people were doing…

I feel like it had to be way more fun in the ‘90s, because everything was probably open, and there wasn’t as much investment.

Yeah, I mean, I would have done things that would have been considered hacking at the time because I just learned about it. The hard part for me in the late ‘90s early 2000s was like just finding the information. Information was so scarce. Things were open, but you had no idea how it worked, or what you were doing. When I popped up someone’s [unintelligible 00:55:51.27] and I’m like “What does this mean? What am I looking at here?” That’s their files. Like, how did this work? And understanding how, and how you get to that point, and that thing actually exists… And now all the docs are open, and all of these things know how this stuff – and the information side of it is really, really free, but you have to go really deep to find the underlying thing that you still get around.

[00:56:13.20] Yeah. Now there’s so many more automated protections, and best practices…

There’s a lot of better defaults, for sure.

Yeah.

You don’t get computers without firewalls anymore, by default, unless you’re doing it on purpose.

I think the way that you’ve had such a perspective, Rich, with the fact that WebMD was kind of like around when there was that bubble… How do you feel now that you’ve gone through seeing these huge companies kind of come and go, and also seeing that first tech bubble? Do you have any advice for people, like for me and other people who this is your first time seeing this sort of like a tech bubble, and recession, or whatever you want to call it in tech? How do you look at it?

I have to say, I think things are really rough right now. I exited my last job in like mid November, and I intentionally took a couple months off… But I’ve been looking for probably like two or three months now, and it’s really hard. There aren’t that many openings. I mean, I think that the thing that over the last few years in general has kind of helped me a lot is that I have that perspective of what it was like back then when the bubble did burst… So I didn’t talk about this part, but like I said, there was that merger that WebMD had, and they were looking to close offices… And at one point they decided they were going to close our office. And things were still pretty good at that point. And so they paid us a retention bonus, which is something that I don’t know even happens nowadays, but they basically gave us double pay for like our last three months. They paid us, plus at the end of it you would get another three months’ worth of pay as a bonus if you stuck it out till the end and helped spin things down, and migrate your tasks over to other people…

And so we were in the middle of that period, and they were also laying off people from our team in Atlanta. And so at one point this new VP comes in above us, and he realizes that basically all of the people who run the production website are about to get laid off at once. [laughs] And so at that point, they actually tried to hire us back. So they were like “Hey, we’ll let you stay, and you can keep the retention bonus.” And I think pretty much all of us were like “Nope, that’s okay…” Because things were so – I mean, back then I just felt bulletproof. I felt like I could find another job in like 10 minutes, you know? And so I took some time off after that. I took a couple months off, and just chilled… And then by the time I started looking for work, it had completely changed. And at that point, the only people that were hiring here in town were the people like Xerox, and they needed you to have a bunch of experience with these enterprise Unix tools that I hadn’t worked with before, a lot of IBM stuff, and… Yeah, it was really rough. And at that point I ended up being unemployed for 10 months, and I was at the point where I wasn’t going to make rent in a few weeks. And so I ended up taking a job that was like a 40% pay cut, and it was much worse responsibilities compared to the kind of stuff that I’d been doing. And it was rough, but I did what I had to. I sucked it up and I worked there for a while, and then I finally ended up getting another job that kind of put me back more like where I was.

[00:59:43.02] One of my friends from WebMD actually went to another company and ended up getting me hired there. But I mean, I think that the biggest lesson that I’ve had from all of that is that I know things are scary right now for a lot of people, but the stuff really does go in cycles. Do what you need to take care of yourself now. Make the best decisions that you can, and try to keep the faith that things are gonna get better, right?

Do you think this is just part of tech, where we’ll just continue to go through these cycles? Is this just part of like the territory, I guess?

I think so, yeah. I mean, unfortunately, one of the things that’s contributed a lot to this stuff over the last few years is the fact that a lot of the investment is going into places that just really don’t make sense. The amount of investment that went into blockchain stuff… Like, every VC was throwing money at these blockchain startups… And a lot of us just knew that they were just throwing money down a hole, and that stuff was never going to work out. I mean, I can think of maybe a handful of like legitimate use cases for it where having a public ledger that’s only writable make sense. But there’s so few. And everything was like “I’m going to take this thing I’m already doing and rub a little blockchain on it, and suddenly I can get some money from venture capitalists.”

Well, don’t worry, they’ve sed replaced all the blockchain with AI… So we’re fine now.

Exactly. [laughs]

The amount of money that they’ve spent – the last couple of conferences I’ve gone to, and you just walk around the floor and it’s the same startup with a different logo, over and over. And you’re just like “This is not sustainable. There’s no way.”

Yeah, absolutely. You know, there’s two things. There’s the companies that are the new startups, that their whole identity is “We’re an AI company”, right? And then there’s the companies that were already around, that are like “Oh, now we’ve got to have some sort of AI offering, right? We’ve got to add AI into our tool, even though it makes absolutely no sense to do that.”

It’s just wild, after crypto, and NFTs, and blockchain…

You would think people learned their lesson.

But it’s like they went even harder for AI. And then they put it in everything, and I’m like… Like, the security problems we’re gonna have… I’m just like “Oh, goodness.”

Looking back on when I first saw blockchain – like, I read the white paper from Satoshi. A long, long time ago – I was doing Bitcoin from 2009, or something like that. I had Bitcoin. I was like “This is fascinating.” Then I saw the shift of like everyone jumped in for the money side of it… I’m like “No, no, I wanted the tech side. How did that [unintelligible 01:02:18.19]

Yeah, sure.

And I got out of it. I’m like “I’m done.” But my first experience with “Oh, cool, I mined some coin”, versus my first interaction with ChatGPT were completely different of how my experience was. And so I understand the hype cycle of like “Oh, this is just a money thing”, versus “Oh, this had a conversation with me.” Those are different mindsets of people that are attracted to them.

See, I felt like I had the interest in the other way. I looked at blockchain and I’m like “It’s an immutable data structure.” You know what I mean? I was just like “I don’t get it.”

It’s a slow database. [laughs]

Yeah. NFTs were like a bad DNS receipt for a lot of –

Exactly. “Here’s an HTTP lead to a JPEG.”

Websites never go away. [laughs]

But AI - that was something I legitimately, before the hype cycle, wanted to learn more about the ethics, and I felt like it could really change the way we do things… But it’s so odd to see. I feel like we’ve scaled back on so many AI, I guess services, or offerings, or startups that could have actually been different, and we’ve put so much money into generative AI, which is like… Dude, we’ve had chatbots, and Ask Jeeves, and… You know what I mean? It’s just odd, even where we’re choosing to put like this huge – they’ve closed so many other realms of machine learning and AI and different projects to see this huge race where everyone’s doing the exact same thing. And now you are treating developers like crap, and I’m just like “But who’s gonna build it? Who’s gonna maintain it? What are we doing?” I’m so confused. This doesn’t even make sense. We just did this, and it was worse, and then you’re like “Let’s go even harder for a horrible –” Like, I don’t get it.

[01:04:06.21] Yeah… So working in operations for many years, I was concerned with things like monitoring… And 8-10 years ago there were these companies that were adding “Oh, we got machine learning now in our monitoring software, and it’s going to do anomaly detection, and you’re gonna get less false positives, you’re gonna get woken up less at three in the morning”, and it just even all felt like snake oil back then. And it doesn’t seem like things are much different. It’s just like the size of the datasets, right? Like, I’m not an expert on this stuff at all. I’m very far from that. But it’s like, I know they’re more complex models and things too, but I mean, it still feels to me a lot like madlibs; it feels to me like it’s just like the predictive text on your iPhone when you’re writing something…

Which is AI-driven.

Yes.

And it’s still so weird that they can’t just – and I obviously do not know a ton about hallucinations and stuff… But I felt like we could solve a lot of the problems with it just saying “I don’t know”, instead of making something up. That’s the scary part, when it completely makes something up, and you’re just like “Uhhhm…”

Yeah, I mean, it wants to give you an answer, right? And if it can’t find the real data, it’s just going to… There’s the stories about the lawyers, who – it’s happened at least a couple times, some lawyer had ChatGPT write a brief for him…

That didn’t exist…

And he goes up in front of the judge, and it’s like full of cases that aren’t real even…

Well, not just that, but it’s wild… Like, we can’t decide what goes in our kids’ textbooks. There’s people who will sit there and be like “That huge thing in history didn’t happen. We don’t know what you’re talking–” So how are we going to sit here and agree what we’re going to put into ChatGPT? It only goes back to 2022, or something like that… So can you imagine like just that fight in general, of what is the source of truth that we’re putting in here? It’s going to be wild.

Yeah. And I mean, we know that these data models tend to be racist, and misogynist…

That’s what I’m saying. And then if they only put in – I’m just like “Oh, this is gonna go wrong in so many ways.”

Yeah… Google firing all their AI ethics people… I mean, not all of them, but –

The good ones?

…so many important AI ethics people… It was a pretty big sign as to where things are heading.

Well, the G in GPT stands for Gullible.

I was sitting in a talk at a conference that we will not name, and they were talking about how their AI was going to do all these – it was going to stop all of these cyber attacks, and it was going to make DoS attacks like not happen, and it was going to do all these amazing things… And then someone behind me was like “We’re using it and we’re getting a ton of false positives”, and she just like murdered his whole spiel, and I was like “I wonder if this is basically our future for a while.”

Well, this has been a roller coaster of positivity on what the internet’s gonna look like in the next few years… But Rich, I want to thank you so much for coming on the show and talking to us about what things were like back in the day, and kind of how you’re seeing things repeat over time. And just FYI, I know you are looking for work, and I looked up, WebMD is hiring tech leads…

Oh, wow…

Did you see that video? Why would you do that to Rich? He’s our friend now. Why would you do that?

He’s in Mumbai, so… You’re gonna have to relocate, but…

Alright, I’m interested.

Yeah. [laughs] I’ll send you the link.

Justin and the dad jokes. It should be like a whole part of the show, like the one Justin dad joke an episode.

You need to make a supercut of them.

I know.

[01:07:52.16] All the terrible jokes. Thank you so much, Rich, and we’ll talk – oh, actually, if people want to reach out to you or find you online, where should they go?

Oh, yeah… I mean, probably the best two places at this point, because Twitter - and we’ll call it Twitter - is such a dumpster fire… I mean, I’m on there, I check it occasionally.

I know… It makes me sad, because it was like so amazing, and then I go there and it’s literally just a bunch of like tech bros talking crap to every woman in tech, and then like Elon…

It doesn’t even load on most of my devices anymore. I don’t care.

It’s like you just walk in and it’s familiar, and you recognize it, but you’re like “This is not the same place. What just happened…?”

But yeah, I’m @richburroughs there, with no dots or anything. I’m on Bluesky, and I’m RichBurroughs.dev there… So that’s probably a better place if you want to just get more like my casual tweets… And then the funny part about it is I think the whole Twitter thing has led to like a resurgence in LinkedIn… LinkedIn is really like one of the best places now to stay in touch with people and post content and things. And so if you search for my name there on LinkedIn… Or it’s just linkedin.com/in/richburroughs, it’s me. So feel free to follow me there or connect if we know each other.

That’s crazy. I feel like I’ve gotten more engagement on LinkedIn. It’s weird, because I feel like tech social media has dispersed into all these different corners, and I’m waiting to see where we all end up at some point.

Yeah, for sure. But yeah, thanks so much for having me on. This was really fun. It was good to see you again, Justin, and nice to chat with you too, Autumn.

It was nice to finally meet you, almost in real life, virtually in real life… I don’t know.

Thanks so much.

Thank you again, Rich, for coming on the show and telling us how things used to work. And if anyone else has interesting stories from their past, we would love to hear from you. We love these retro episodes, and learning just how things used to be… So feel free to email us at ShipIt [at] changelog.com, and then we can figure out if there’s a way we could get you on the show and talk about it.

Also, someone hire Rick, because he’s amazing.

Yeah. For today’s outro, this is going to be just kind of a little discussion about something that happened on the internet, which someone who has controversial opinions about some things posted. So David Heinemeier Hansson, if you don’t know, posted a blog post about “We’re moving continuous integration back to developer machines.” And he was talking about this, and saying “Hey, this is–” Hah, his company is Hey. Actually, his company is Basecamp, but the product that he works on right now is Hey. They had CI systems. CI and CD are, if you don’t know, continuous integration and continuous delivery. And continuous integration is specifically responsible for running your tests and your builds, and saying “Hey, how does this software work?” And validating it, making sure that all of the tests you have pass. CD is all about pushing those bits out to the world to actually deliver them so that customers can use them. And in this case, he’s arguing that the CI system that they were using - I don’t even remember if he mentioned what it was - was slow. And it was slow to the point of annoying him, or something. And I think in his tweets he was saying it was taking like 10 minutes or something like that to run CI, and on his local machine it would take like two minutes. I was like “Okay. Yeah. Computers are fast. He’s arguing that local computers are fast.”

The thing that I’ve found interesting was that he replaced the CI system with this 50-line Bash script. And a lot of this to me screams of comments on places like Hacker News, where someone says “I could rebuild that in a weekend.” And once you get to some level of proficiency with tools - and DHH’s tool bag is probably Ruby and Bash - you’re like “I could replace that with a small Bash script.” And he did, for some amount of his use cases. And I’ve found it interesting. And Autumn, I would love to hear your opinion on CI/CD systems.

[01:12:02.27] I have so many opinions…

Yeah, because this is something that he did, that solved his problem. There are lots of other problems that this creates, right? He has this sign off Bash script, which again is 50 lines, and all it does is it runs five different Ruby scripts. It’s just like “Hey, we’re going to figure out what our SHA is, and then run these scripts that do tests.” It’s like “Okay, if that’s all you’re doing for CI, where does that have to happen? And where do we make that repeatable?” And a lot of that comes down to like “Do we trust the developers?” Because if I had this locally, and one of them caused an error, I could just comment it out and run it again. And I’m like “Oh, cool. We passed.” You could do these sorts of things. But you have to kind of trust the environment, you have to trust the company, you have to trust people you work with, and at large companies I absolutely know that this would happen, that people would just say “I skipped those things, because they were annoying.”

I think that’s just humans, but I don’t even think that’s the biggest point there. First, anything that comes out of that man’s mouth is – he speaks in absolutes, which I think we all know that when we say “It depends”, it’s for a reason. It depends on what you’re building. It depends on what kind of software. There’s so many caveats that it drives me wild that he has such influence, and he really influences people, and he knows, he’s aware of the influence he has, and he just makes hot takes to be controversial, and for people to bolster his position, or whatever it does for his ego, or whatever.

But I also just think that this sounds great, right? But isn’t everything small that we run locally faster? It’s almost like “No s**t.” But at the same time, if you’re building for multiple architectures and multiple versions of something, that’s not going to work. It’s like saying something works on my machine.

It tests on my machine. That’s exactly what this, right?

Like, just working in environments in the past, where you were building for different architectures. You cannot test on one architecture and know for sure. Think about how many kinds of Linux there are. You have Mac and Windows. There are so many different kinds of… And I just don’t think that it’s great to speak of it in an absolute of “You can do this. And look, I reinvented the wheel, and I made it so much better.” And it’s like “Cool, you did for your use case.” I think if you really wanted to teach and you don’t just like the sound of your own voice, you’d say “I did this, and I did it because this was my use case”, and then you give people information, like a post mortem. Or when you’re doing a tutorial, you give them the background.

He’s also said “Go on prem, because it’s cheaper, and cloud’s dead.” But it is for some use cases, it is cheaper, and some it isn’t. But he never gives the information to be educational. He gives the information to like get someone to retweet it.

Well, I mean, in that specific use case too, he has like a follow-up post of like “Here’s our numbers”, or like “This is how much it costs us, and this is why we said it was cheaper.”

He’s constantly only giving certain information. Do you notice that? He is very selective. It’s always to prove his point. And it’s never – like, there’s tons of people who’ve done things with obscure work… Like, not even obscure, but every kind or type of like – if it’s like a retail vertical, or gas, a lot of things are reusable, and it’s actually great when people share those, and they’re like “Hey, I saved money doing this.” I’m all for it. But there’ll be like “And I did this and this. And reasons related to this, and this is how we got here.” But he gives out very specific information, and it’s almost just kind of to hype himself up and his opinion sometimes. And I’m just like “Dude, you actually build cool stuff. You could really give out and help people and be a voice that we could all learn from.” But it’s like he is just – I don’t know, it just seems like he only gives out very one-sided information.

[01:15:51.11] It’s like when Elon said he was going to close one data center, and they closed a whole data center. But what else did you open? He’s like “I deleted all this code.” But how did you do that? You know what I mean? That’s cool, teach us, but actually give us the information we need to make decisions based off of what you’re promoting, I guess.

Yeah. Well, and for this specific implementation of like a local CI – there’s like 20 things I thought of right away. Like “Oh, well, you can improve this by running these things in containers, having reproducible –” Because if someone’s on a Linux machine versus a Mac, or versus Windows, versus whatever… Like, how do you make this a little more reproducible? But then also, I think that – I’m constantly running tests locally as I’m developing things, or we’re writing some tests, we’re verifying those tests… I’m not doing the whole suite, because that takes longer. But then also, when I git push, at that point I run the system – like, the system runs again just because it’s designed to run again. And whether that’s saving time or not… For some tests that can save a lot of time. I’ve worked on plenty of systems where it was like half a day to build the software. And it’s like “Hey, if that’s the thing, if I could speed up 20% of that, then that’s a big win”, depending on how it works. We have a whole system for building software that would like distribute it amongst everyone else’s developers machines. It’s “Hey, we could like distribute this in containers, and make sure that it ran consistently, but it also reduced our overhead for how long it took to run.” But it did take time to set up that environment, and do this other stuff. So it’s like wherever your bottleneck ends up being.

And this specifically, it sounds like “Hey, I have a really fast machine. I can just get rid of some of this bottleneck by just running it locally. I don’t have to push it…” And that does help with debuggability. I hate git-committing something and seeing Jenkins fail 20 minutes later, and I don’t get the notification… That cycle really, really sucks. But if it is reproducible enough locally, where should CI run? Because CI and CD have always been coupled, just because historically Jenkins told you to. Signor Jenkins was like “Hey, I can do both. Just run them here.”

I think they’re all valuable though. Like, even when you have automated CI/CD, there’s some times when you can learn more by testing a portion of it locally when something fails, right? I think you can always have both in certain situations; like, something will fail, and it doesn’t make sense, and you go test it locally, and it passes, and then you have to figure out why. But I just think that he didn’t give enough information. Like, there’s certain use cases where maybe you only have to test it locally. Maybe you’re building something that’s very Mac-specific and you’re testing it on a Mac, and it’s only for that thing. But how much software is really that small? I don’t know. It depends, I guess. It depends on what you’re building.

I guess it depends on the target, a lot of times. Because to your point, if I was developing something on a Mac, I would build it on my Mac, I would test it on my Mac, and I would push it somewhere else, and I would like reproduce the Mac environment. And that’s really, really hard.

Well, not just that, but think about it - there was a time where things worked on Macs, but when they came out with an M1 chip, they had to then start making patches for it to work on an M1 chip. So if that was the case, and you had an M1 laptop, testing it locally would be the right thing, because you’re testing it for that specific system to get it to run. But I think this just goes down to like “It depends.” And I feel like you always should leave the world a better place than you found it, and when you have knowledge – especially right now, people in tech are struggling, they want to find jobs, so you’ve got like new people coming in… Give them the background to learn from you. Like, dude, you’ve contributed to a whole language; you’ve got a whole company. Pass that knowledge down, play it forward, and don’t just hype yourself up all the time.

Well, so what situations do we think that CI locally makes sense?

If you’re doing it specific for the architecture that you’re running it on, a patch for something that maybe has been tested everywhere else, but now you need it to work for that specific architecture, maybe…

And that’s still difficult, because it’s like your environment is different than your customers.

Exactly. So I still think that you have to still run it through some sort of automation and have it tested on more than one machine. So that’s what I’m saying… I always want to give the benefit of the doubt, because even when you may disagree with people in some areas, I think people – like, everybody has diverse, valuable perspectives… But this is just one of those – like, there’s so many ways that this could be bad advice.

[01:20:02.11] Yeah, yeah. Absolutely. And that’s why I wanted to talk about it. Because this isn’t necessarily good advice for everybody, like “Oh, we should do CI locally, because they’re doing it.” I guess when it’s homogenous at the developer experience – like, if everyone in my company was doing GitHub workspaces… Those are virtual, they are reproducible, they are tightly controlled in what they install, and how they work, and I could see that being an environment where local CI makes sense, because the environment where it’s developed is tightly controlled enough that it’s going to be consistent. And it’s going to be the same as if I spin up another container and have Jenkins do the same steps, right? That makes sense to me, is if the –

Maybe something that’s a small tool… But I think at this point, especially like in open source, and like at scale, it’s probably not a great – it might be a great way to debug something, or to double-check something… But at any kind of scale that makes it worthwhile, I don’t know what the use case would truly be where you could use this.

That was the other thing I was thinking, was like when your development environment’s tightly controlled and when your target system is tightly controlled. Because he’s deploying in containers on a deployment system that he wrote. Literally – I forget what it’s called. [unintelligible 01:21:18.09] that’s what it was called. It was like, it’s just Docker containers at the end of the day. So it’s like “Hey, if you can control one end”, and you can deploy in one specific area, where again, you can control the processor and the operating system, or the container environment, all that stuff, that environment seems like “Yes, I can see how you can come to this natural conclusion that you can save some time and do it this way.” But if you don’t control either ends of that spectrum, then this probably is going to cause more complexity in the verification and testing side of it, in the integrations.

But even when you do control those things, eventually you’re releasing software into the wild for people to run it in different places, and you could still end up with a problem, depending on what the use case is.

Well, this software specifically isn’t open source. They’re not letting other people run this.

Oh, okay.

This is this is the backend for Hey.com. There’s a web interface and the backend. So it’s their systems. Who cares. That makes sense to me. It’s like, yeah, if I’m releasing it – like, if I only ship a container… But even that architecture, and…

That’s what I’m saying, there’s different architectures. That’s just an image, but what kind of image?

Yeah. But if you can control both ends of that spectrum, you can simplify a lot of the middle.

But how much software do we really control both ends of that spectrum? I mean, I’m not saying that it doesn’t happen; it does happen. But at scale, it is unlikely for it to be – like, I think you can do intense test coverage and you still end up with edge cases that you have to go account for and do a fix for. So…

And a lot of that though is user behavior. A lot of those edge cases are not down to the system.

Sometimes it’s not, though. At this point we are running very complicated infrastructures in a lot of places. Phones are complicated. Just regular laptops are very complicated. There’s such a variety of different kinds of phones, different kinds of computers. People are using MacBooks, but they’re using it on a dev [unintelligible 01:23:17.07] I feel like there’s just too many variables.

Yeah.

Which gets us back to the famous saying “It depends.”

Yeah, I’m trying to think what environments have I ever worked in where we control the both ends of that spectrum.

I was thinking maybe if you’re writing like an automated script somewhere, and it’s very, very, very basic… But for all the use cases in which this would apply, you’re not doing intensive CI/CD pipelines and automation anyways. It may be a part of that, but… I don’t know, I can’t think of anything.

I just wanted to figure out if there’s a use case for it. If anyone has opinions and would like to talk to us, again, you can email us. But I think actually the Changelog Slack would be a good place to – there’s a Ship It channel…

I love the Changelog Slack.

Yeah, feel free to jump in and talk about it, because we’d be interested… I’d be interested in knowing what other things are we missing? If you don’t control both ends, would it make sense to do something like this? Because it’s just a conversation around… I’ve done it in the past, and it was always seen as an anti-pattern. Before CI/CD was a thing, that’s just how things worked. But now it became an anti-pattern. And is that anti-pattern always also true?

I also think that there’s so many great tools, like GitHub Actions and different ways of testing, that does so much for you that unless you’re just trying to do a bug fix or you’re just trying to debug something, and get like a kind of a quick way to do something locally before you then put it to something automation… Like, there’s so many cool ways to do testing that you get more coverage for.

Yeah. Okay. Well, thank you so much for listening to the show this week. Thank you again, Rich, and… I forget what we have for next week. We have a couple of things scheduled, but I don’t know what order they’re going in anymore.

Oh yeah, I can’t think of the order. I can just name a bunch of guests, but I don’t know about the order.

Yeah. So I’m sure we’ll have someone great, because we’ve had a string of great guests…

I can’t believe we get to talk to all these people, and it’s a podcast; it’s like nerding out with like the coolest people.

It is definitely a life hack, of being on a podcast and inviting cool people to come talk to you.

So much fun.

Thanks, everyone. Have a great week, and we will talk to you again soon.