Even the best rides come to an end

Kelsey, welcome to Changelog & Friends. Happy to have you.

Yo, happy to be here.

I would consider Kelsey a longtime friend.

Yeah, I’ve been rocking with the Changelog for a long time. Longtime listener, and then I’ve had my share of episodes…

You’ve been on like every podcast we produce, pretty much. You’ve been on Go Time, you’ve been on Ship It, you’ve been on the Changelog… Probably not JS Party; that might be that one hole in your resume there, but…

We’ll have to make you a founder someday, get you on Founders Talk.

Well, we could do all of that.

Well, I first met Kelsey when we worked with CoreOS, way back in the day. I just remember because it was just audio, not video. And you described yourself as somebody who was short, but with the last name Hightower, so it was a misnomer.

Ah.

That was just the best. And we’ve been doing these ad spots that way for a while, like have somebody on from inside the company, and Kelsey really helped me pilot how that will work, and that was kind of cool.

It sounds like I need royalties. I should have gone on that [unintelligible 00:01:33.22] strike!

Well, you’re back, aren’t you? Aren’t you back? We’re getting you back…

I think the first time I met you, Kelsey, I gave you the shirt off my back. Do you remember that?

That’s true. Gosh.

It was OSCON.

The last OSCON.

That was kind of weird…

That was weird.

Because you think about all the sweat, and hair, and all of that stuff…

It was early. I was fresh and clean.

Yeah, I made sure I washed it. I was like “Let me wash it first.” [laughter]

I saw Kelsey’s face when you did that, Jerod, and it wasn’t like bad, but it was like “This is odd.”

He wasn’t sure about it. He was like “This is happening?”

I took the shirt it was dope. You’ve just gotta wash it first.

He took the shirt. So we were at OSCON, we were giving out T-shirts, and we didn’t have your size. And it happens that you and I wore the same size. And I was wearing the shirt that we were handing out, and I had another shirt there… And so I was like “Hey man, we don’t want anybody to leave disappointed, leave empty-handed… So I’ll hook you up.” And he’s like “You don’t have to…” I was like “No, I insist. I’m gonna go change my shirt.”

Wow, Jerod. That’s in your brain, man. That was exactly how it went.

Well, you can be the first person to say you literally gave me the shirt off their back.

That’s right. Just selfless. Just pure selflessness.

Just aiming to please around here. Aiming to please. Well, you’ve got some big news in your world. Should we start there?

Big change. Holy cow.

Yeah, I mean, I announced my retirement. I was doing dry runs for a year… I did my first set of two-month vacations last year and at the top of this year… I did it right. I mean, you get the corporate laptop, you remove all the corporate accounts from your phone, you slide that laptop under the bed, and you just don’t check it for months. And I did that.

It sounds nice.

And what I realized is that I have so much going on outside of Google, it almost feels like a second job, in some cases. Like, you’ve got the advisory, you’ve got the speaking engagements, I’ve got family… And so I’m just like “Dude, let me just tune everything out and just kind of focus on what’s in front of me.” And I realized it’s like “Yo, it’s probably time.” And you get to some point in your career and I ask “Why am I still doing this?” And the bigger question I used to ask myself probably the last five years - if I could buy my time back, would I? And I started to think about the relationship I had with companies. They value my time in a certain way; they give me stock, they give me base salary… So they say, “Hey, Kelsey, your time is worth X to us.” And I said, “Well, one day, if I could buy it back, would I? Could I think of something better to do with my time if I could afford to leave the money on the table?” And so when that time came, I felt like “Yeah, if I want to splurge on something, why not it be my time?”

Yeah.

Love it.

That’s pretty profound to think about that, could you buy your time back? If you could, would you?

Yeah, I mean, it’s a lot to think about, because if you think about it, most people go to school till they’re about 18, maybe a little longer, and it’s all structured: do this, learn this, here’s the requirements to graduate. And then from that, most of us will pivot into a job, and roughly the same kind of thing. “We need to learn these skills. Here’s what’s required to get promoted.” And a lot of people do that for 30-40 years straight. And so you don’t have a lot of time in between that to figure out who you are, and what you would do if you didn’t have to do that. And I get it, it’s such a luxury to be able to be in a position where you get to decide whether you want that route or not… So I’m 42, and taking a step back, I was like “Man, how come I haven’t put a ton of thought in what I would do with my time?” Like, I put a lot of thought in programming languages, I put a lot of thought and how platforms work, I put a lot of thought in how to get way better at those things; like learning all the keyboard shortcuts… But boy, I didn’t spend a lot of time learning life shortcuts. There are better ways to cook. There are better ways to clean. There’s better ways to go on hikes. I didn’t spend any time on that kind of stuff. And so I was like “Yo, what if I put the same effort in? What happens?” And so when people ask me “What’s next?” I’m like “I want to find out if I took the same creativity, the same level of focus into the nuance of life… What do you end up with?”

Is this a – it’s not a crisis, but is this a midlife thing, potentially?

Well, he is midlife…

But think about it like - if we frame it as like a midlife crisis, what’s the crisis?

Right.

Yeah, that’s why I pulled crisis out, because this is not a crisis. You’re not in a crisis moment. You made a full fruition decision. Obviously, you’ve put some thought into it, you did it deliberately…

You know, it does feel like an awakening. I remember watching Westworld for the first time. No one told me the plot…

[Spoiler alert]

I’m just watching this thing, and I’m like “Why is this show popular? Like, these people are going to this thing, doing nonsense in this real-world simulator…” And then one of the robots found out that they’re a robot, and they woke up.

Ah. Spoiler.

And they were just like “What the hell is this? I thought it was real, but I’ve been in this loop the whole time, and it turns out I can get out of the loop.” Some people - they don’t want out of the loop. Like “Hey, I’d rather stay in the loop. This is much easier to deal with, it’s easier to process.” Getting off the loop is dangerous. There’s no script. What do you do when you’re off the loop? There’s no programming for that.

You’ve gotta follow a protocol. You got red-pilled, in a way.

But it does feel like a crisis, because when you get off the script, that means you’ve got to write the script. You can’t complain as much anymore, because it’s your script now.

You’re in control. Yeah.

Yeah. So I think - yeah, it does feel like a sense of urgency. If you wake up, and you’re like “Yo, that’s an option”, it’s hard to keep telling yourself that you can’t take that option. That’s a hard thing to say, “Yeah, I can afford it” and then you start making these excuses, why I love my job. It’s like “Well, what do you love about your job?” Well, I like working with people, I like working on cool technologies, I like leveraging this tool set to nudge the world in a certain direction. Are you telling me you can’t do that without reporting in at a certain time? That’s not true. It’s just that I couldn’t afford it before. So I removed the main barrier, which was the financial piece, and it turns out I can still do those other things without necessarily having to be on someone’s time schedule, or their list of priorities.

Right. This is day one then, roughly? I mean, this is like the first day of the rest of your life kind of thing. Kind of a moment.

I think anyone who retires, mentally you have to retire months or years before.

Okay. Yeah.

It takes prep. This isn’t something you wake up and do the next day, at least for me, because you have to think about basic stuff like “Where are you gonna get health insurance from?” All of that stuff has to be – what kind of coverage do you need? You have to go through all of these things, and then you realize, “Oh, wow, the company was paying quite a bit of my premium.” You put all this calculus into it, because now you’re going to be solely responsible for these decisions. You’ve got to learn a lot more about the tax law. Where do you live? How will your income be treated?

[08:19] You also have to think about runway in a whole different way. When you have a job, you kind of figure yearly you’re gonna be getting a certain stream of income, you can kind of live based on that stream always being there. But when you’re kind of on your own - and one of my conditions for retirement is I need to not have to do anything. No podcast, no speaking, nothing, and it still has to work. If it doesn’t work without that, then I will feel a little bit uncomfortable knowing that I will still be grinding in a different way. And so there’s a lot that goes into that. So this is kind of like, I don’t know, year two.

Gotcha.

You’ve got the plan, you’ve seen the plan work out, and you’re like “Okay, now I’m ready to go and execute the last part, which is to resign.”

You’ve been sharpening your knife. This is an Abe Lincoln quote. I don’t know if this is true, but I’ve heard this before… “If I had eight hours to cut down a tree, I’d spend the first seven sharpening my axe.” It’s kind of what you’ve been doing, right?

100%. Because you need that boost of confidence to know that “I don’t want to swing this thing and not–” [laughter]

Right. Work smarter, not harder, in life.

What’s your runway look like then? I mean, did you calculate out 30 years, 20 years, 40? What are you thinking? …because you’ve gotta make it to –

Oh, I calculated it at my daughter’s daughter’s lifetime.

Okay.

Yeah, because like my lifetime - I’m 42, and I was like “You know, lots of things can happen.” And so whatever number I used to come up with, I used to triple it.

That’s how assume about software estimates.

So here’s the number I need for the family, multiply it by three, and I think that deals with – I mean, so here’s the thing that I as a person that’s humble and realizes the value of people… In order for someone to retire, that means everybody else has to keep working. Because your water needs to run, trash needs to be picked up… All these things need to occur. And so in order for that to work, that means other people have to respect this thing called money. So the money you amass, you need to be able to use it.

And so I thought about, “Okay, what’s the average salary of a person that has a decent life?” And to me, my budget, I never spent more than $100,000 in a year. I mean, after taxes. Vacations, everything. I’m debt-free, everything is paid off, house included; everything is paid off. And so in that mode, I was like “Okay, what would I need to earn, maybe through interest, that the average salary of the average person that’s doing pretty well in my eyes?” And everybody’s metric is different. And so when I looked at that number, I said “Alright, if I can triple that number, and get it through interest payments from like Treasuries, that is my criteria.” Because I knew I would be okay based on the lifestyle that I would like to have, plus cushion, and weird stuff like inflation. And so that was the mental thing I needed to get comfortable with making that jump.

Well, congrats on getting there. I mean, that’s awesome. It’s got to be a great place to be.

We’ll see. [laughter] I take it one step on this journey, and I know friends that have done it. Some of them only lasted a few months, and they needed to go back. And I understand why. So now I’m trying to make sure I’ve done the mental prep to figure out what to do with my time.

Right.

Well, I appreciate you doing this podcast, because you said no podcasts here in a bit.

You didn’t have to be here. he does not have to be here with us.

I never have to be here, but I like being here.

[laughs]

Well, I’m glad you are here. So when you say retired, do you mean just from Google? Do you mean from software? In your eyes, how do you map out true retirement for this?

Just no more nine to fives.

Okay.

No more HR, no more OKRs… Profit Loss… That’s the part I want to retire from. To me, tech at this point is like any other skill. I got pots and pans. That’s technology. The things I’ve learned to cook, that skill, that’s cooking. So writing software - it’s just a tool that I have in my tool belt. I’ve been doing it for so long that if I ever run into a problem that can use software to solve it, I want it to always be there. You don’t throw away your expertise either in retirement, right?

[12:00] I guess what I mean by that is, are you gonna be talking about things like Kubernetes still yet? Are you going to be still – I saw you respond back to Scott Jonathan from Docker, “I’m still gonna help out with the Docker world.”

I’m an advisor to Docker. He and I talk all the time offline, so… I think a lot of what I do is not predicated on my current employer. So that expertise you have - I think that sticks around for a while. And my attempt at staying relevant is by kind of mentoring people, and seeing the challenges they face, getting my hands dirty, the prototypes… I’m always going to be curious, and I think I’ll still be curious about this realm. And I think that if I do that well, I’ll still add some value. And then when I don’t add value, then I’ll also be done with that.

What are you the most curious about right now? I’m thinking, what’s his next couple of weeks look like? Is it “Okay, vacation time, and then I’ll figure it out”, or is it like you’re hitting the ground running on hobbies, or you’re learning to cook better? What’s your first couple of weeks planned retirement look like?

I mean, last month I remodeled my fireplace. Tore it down, learned how to reframe it, two by fours, Japanese saw, got it the way I wanted it… I just didn’t want to mount my TV over the fireplace. And I used to do low-voltage work back in the day, so my tool bag is all built back out… I bought a really nice tool bag. I got all my client tools, cable testers…

Nice…

Looked at the code for running electrical… You’ve got to put staples at least 12 inches from the – like, I learned all the things, and I finished it. And I had a guy to help me finish the drywall, and put the shiplap up. And I stood back from it and been like “Yeah, now I’ve got those skills.” I learned about pulling permits, when you need them, when you don’t… I just took the slow road to learn some new skills, because I also want to have skills like in the physical world. And so two months ago, I finished it, and remodeled it. It’s now the way I want it, the TV is where I want it… I have a little access door on the side, I pulled all my Ethernet cables through… And so I’m just going to figure out whatever skill I want; I want the patience just to sit back and learn it the slow way. So if it’s cooking, remodeling, no matter what it is, that’s the approach I want to take to it.

TV over the fireplace. It’s a controversial concept. I have one at our house, it’s over the fireplace…

Yours is over there, Jerod. Yeah.

It is. There wasn’t a great other spot for it, but… I don’t have a problem with that, but some people are like “No, you never put a TV over the fireplace.” What’s your guys’ take on the topic?

That Reddit thread, “A fireplace too high” is hilarious.

I don’t know that I’ve seen that one.

I was about to do it, and my friends sent me that thread. I was like “I can’t unsee this now.”

[laughs]

So there’s a whole Reddit thread about “Your TV is too high.” And you see people on the couch just looking up with neck pain. I was like –

Necks bent, yeah…

If you got a big living room, it’s okay.

You can cope all you want… [laughter] There’s no reason for the TV to be six feet off the ground… Unless you’re at a bar.

Yeah, I do agree with you on that, TV to high kind of stuff. So my take on it is I still agree that it is a little too high. But you can augment at least the look with a Samsung frame TV.

Yeah, those are nice.

Yeah. So if you put that there, it’s almost like a painting when it’s not a TV. And you can even buy some cool frames. You can build your own frame for the frame TV if you wanted to, and make it look super-cool. We used to have our TV over the fireplace, Jerod, before we sold that house, and this new house is not over the fireplace. And that’s not by choice, it’s just - it doesn’t make sense over the fireplace. It would actually be off angle, in the wrong place if it was over the fireplace. I’m still team over the fireplace, honestly. I kind of like the look. Kelsey’s like “No.” Drop it, Kelsey. What is it?

Just too high?

Nah. My neck, man. I have to take care of the family, man. We’ve got to make sure our necks are good, our posture is good… You can’t do that watching all three Lord of the Rings in a row. You do that –

Right. He’s paying his own premiums now, you know? [laughs]

Yeah, you’ve gotta think about all this stuff.

Here’s my solution then - that’s not your only TV.

That’s true. I mean, if it was me – my daughter took the media room; the room that’s designated for the big wall and all that. One of the biggest rooms in the house. She’s like “That’s my room.” So I lost my media room. Because if I would have had it, there would be no TV in the living room.

Sadness…

I lean towards that philosophy; like, have a viewing room for all that stuff, and keep the living room for living, chatting, talking and discussion.

[16:05] Right. Yeah, that’s my wife’s take. It’s like, no TV in the main living room. But there is one, because we have an open floor plan, so - I mean, it’s kind of one big room: the kitchen, the dining room and the living room. It’s all one big room. And I’m all about conversation, and sitting down, and enjoying time together… But I’m also about the football game is on, and we’re at the island, cooking up stuff, and we’re gonna have that thing on. So she lost that particular battle, but she’s won plenty other ones. And I have to put it over the fireplace not by choice. I wouldn’t want it there, but it’s just in the design of that particular space. It just doesn’t make sense elsewhere.

If I had to design the room though, I probably would not design the room with the fireplace and the TV over it. I’d probably actually design the room to not – if my fireplace room had to have a TV, I just probably wouldn’t put it there. I’d just be like “There’s no TV in this room.” That’s kind of how I am about my bedroom.

You live in Texas. Why do you need a fireplace?

Well, it’s an aesthetic thing, right? It’s a design thing. We did not need fireplaces, unless it is the great freeze, which was two years ago, and the more recent great freeze here in Austin… We literally had icicles on everything. I don’t know if you saw that, Kelsey, but it was intense. Everything in Austin was frozen. You couldn’t even drive. The roads were literally ice. Never in Texas I’ve ever seen that, ever in my life.

Yeah, when I saw that, I was like “Yeah, that’s – things are different.” I guess they’re bigger in Texas now, including the ice storms.

Gosh, it was the most intense storm ever. And it wasn’t even like bad. It was just like, everything was frozen. It was like Elsa when she was upset. But here’s a different side of this, this TV endeavor… I do not want a TV in my bedroom. Bedrooms are for pleasure and sleep.

There’s no pleasure in a TV? TV gives you no pleasure?

Well, it’s just too many TVs all over the place. I say that… I’ve lost this battle. My wife has won. There is going to be, and is a TV in our bedroom.

She wants a TV in your bedroom.

It’s going to be there, but I do not like it. So right now it’s not installed, and I’m loving it, and I’m going to use these – because we’ve just moved into our new house, Kelsey, like two weeks ago. Three weeks ago.

You’re slow-playing it.

Yeah. So the TV is not mounted yet. Like, it’s not installed properly. Other things are higher priority than this TV being stood up in our bedroom. And I’m going to use these few weeks as evidence and data to say “You know what? Have you been sleeping well? Have you come to bed and just gone to bed? Yes, you have. Okay, that’s because there’s no TV in here.”

So I’m anti TV in the bedroom, personally. And every doctor who thinks about sleep and the importance of sleep - I’m sure you think about this, Kelsey, as you retire… Like, “You know what? Now I can actually sleep a little bit better, because I have less things over me, and I can get my true eight, or my nine”, if you’re a nine person… But it’s a lot easier when you have no TV in the room.

Yeah, I like as few TVs as possible. I actually lived a long time with no TV. Maybe three or four years with no TV. You just end up doing other things… You’ll cheat and watch some YouTube videos or something like that on your laptop or your phone… Bedroom - I agree with you. I kind of keep the TV out of there. And honestly, I only really watch TV as a family activity. Like, we’re all doing it. Shared experience, talk about the show afterwards…

Yeah.

So I’m kind of with you. I don’t want to have TVs in every room in the house. Like I said, for me, I’m kind of like “Put a TV in one place”, because it’s like going to the movies, or watching the show, shared experiences with society. I kind of use TV for that kind of thing. But yeah, I can argue. I don’t want to go from one TV room to the bedroom and continue watching. It’s like “Hey, at some point, talk to each other.” There’s so many other things to do. I’m pretty sure people can get creative doing something else other than watching TV.

Yeah. It’s bad for your eyeballs, too. Blue light before you go to sleep is just not good for your eyeballs. Your sleep, circadian rhythm - it messes with all of it. Now, I say that as a person who probably looks at my phone within the few minutes of actually going to sleep, so that’s no different…

You’re probably watching YouTube on your phone, right there…

I believe there’s TrueTone stuff on your phone to like block the blue light, and stuff like that. So I think I’m at least getting that. But yeah, I’m still – I am anti-screen before bed, although I do screen before bed. I prefer not to; I don’t always do as I say I should do.

Break: [20:13]

So Kelsey, one of the things you said in your announcement was that you’ve spent 25 years learning how to work, and now you’re ready to learn some more about how to live. You’ve kind of referenced that offhanded earlier… For those of us still in the game - you’ve been highly successful at everything you’ve endeavored to do, that I’ve seen, in your work, and you spent 25 years thinking about it a lot, working hard, succeeding, probably having some failures as well… For those of us still playing, have you got any tips, man? Have you got any tricks? How do you work well? How do you succeed? How do you not suck?

I mean, thinking about tips like these - they don’t really work out of context. For me, there’s a lot of context into it. I pivoted a lot in my career towards influencing people. So if you think about it, if you write code on your way up, or you’re doing system administration on the way up, you really get good at what the computers need. You really get good at “Hey, here’s how the Linux Kernel works. Here’s how you patch this, how you automate everything.” You get real good at that. And then at some point though, you realize that you’ve got to convince other people either how to get good with that, or when to move on from that. And that’s a whole different set of skills.

So I would say mid-career, six, seven years in, I was like “You know what, I’ve gotta figure out…” There’s that phrase, “Some people have ideas, and some ideas have people.” And so I want to get a little bit better at saying, “Look, I can see a vision in the next two or three years, all of our infrastructure will be fully automated with Puppet. Now, Puppet just came out and it’s not even 1.0 yet, so I’ve gotta show you.” And I learned a bunch of ways of making people see themselves in the future. And so when I would give these demos working in companies like financial institutions, everyone’s afraid of big change, and I was asking for humongous change. And so what I would do is I would show people the technology and they’d be like “Yeah, whatever. I can do that with Bash scripts.” And it’s like “Damn, do I really want to fight this person? I’m doing something wrong. It’s not Bash versus Puppet… That’s one route, you could go.” But I decided to do something different.

I remember, I was in this all-hands, executives from security, risk management, operations were all in the room, and I was like “Hey, I’m gonna show you something. Pick any server in dev right now… I promise you that this thing can come back, as it was, in about three minutes. Everything.” And I was like “Alright, let’s pick this encryption service.” I was like “Yeah, just delete it. Just click on VMware and just hit Delete, and then just bring it back.” And in three minutes, PXE boot the OS, laid down all the right things, Puppet did its things, I converted everything to RPMs instead of tarballs and WAR files, and it just came up. And I was just like “Delete it again. As a matter of fact, go in the file system and just do [24:18] everything, and then watch what happens when we restart it.” And the team’s like “What’s that?” I was like “It’s dope, ain’t it? If we could do that for all the environments, the concept of provisioning just becomes less of an issue. And I’m going to show you one more thing. The challenge is, we get all these requests, right? Every day, someone will say “I need a new server that does this thing” in one of our 500 environments.” Because you don’t really have three environments, you have like 500 of them. And so you’re getting these JIRA tickets, and you’re processing them… I said “Let me show you something that I think would be dope. You’re in a live demo. What if you could open a ticket, and then from the dropdown you see all the available servers that you have permissions to see, and then there’s another dropdown where you can pick the app version that has been signed by QA, and once you hit Save, and it gets approved, that that server will just come online, and in the ticket there’ll be a custom field with the IP address and your SSH keys will be already there, because you opened the ticket? Wouldn’t that be dope? And I’m just walking through and I hit Save… And that works.” And they’re like “Yo, yo, that’s cool. What kind of trick is this?” I’m like “This is very real. It’s a combination of Kickstart, Puppet, Red Hat, all these technologies… But that’s less important.” And when people saw themselves using that tool, they’re like “Yo, you know how much time I would save if I can just point people to this type of ticket type, and that has to be in the way of doing that kind of work?”

And so then I think I’ve learned how to actually speak to people and inspire them. And so that was a big a-ha moment, that I had to go from just like “Hello, world. Here’s how this low-level tech works” to “I’m going to show you what you look like in the future if we were to take our problems in context and solve them.” And then my whole career took off from that.

You’re pulling an Alladin.

[26:01] I can show you the world… Shining, shimmering, splendid…

He says “I will show you the world.”

Show them the world.

You’ve got to paint the vision, right? You’ve gotta paint the vision big time. You can’t just say “Here’s a new tech, and how it works.”

Not just paint it though, but he actually demos the vision.

Right. Exactly.

He doesn’t just paint a picture, he actually shows you at least an aspect of that future working today.

The world. He shows them the world.

I like that analogy with Aladdin, because you do have to take people on the magic carpet ride… And you’ve got to take them there, and you’ve got to be quiet, and you’ve got to let them see what it feels like, you’ve got to show them the good and the bad… And then I think it’s easier for people to make a decision once they’ve seen it themselves. You know you got it right when they can explain the vision to other people.

Oh, yeah.

Like, when they leave that meeting, it’s like “Yo, Kelsey just opened a ticket with custom fields, and checked every box that we do during the change management process, and it came up reliably every time. We’ve been trying to build that. He has that. How do I get on board with that?”

Yeah, that’s cool stuff. You’ve done a lot of very powerful demos at conferences well, publicly, not just inside of your particular organization… I know that, and I saw you address it at some point - this is probably years ago at this time, but I’m thinking about it again as you tell that story… About your demos are very magical, and – I guess a criticism of some of your demos has been like they’re so happy path, they’re so good that maybe they’re too polished, too happy, not realistic… That’s been levied at you, hasn’t it? How did you usually deal with that?

It has, and I think people think that people who can present and talk, they can’t do the work. They get this misconception that all you can do is – like you’re a scripted actor. And I get why they believe that, because you can get real far as a scripted actor. And look, society needs some of those. I love movies. But I think the reality is what people don’t see - when you see someone like Steve Jobs on the couch, pull out a MacBook Air from a vanilla envelope… Do you know how much work had to go into the fact that they had to make it that thin, that it would fit in there? So imagine the number of product meetings that Steve Jobs was in and said, “Yo, that’s too thick. We can’t do that.” “Well, if we make it any thinner, we’re gonna have to get rid of the ports.” It’s like “Yeah, people are going wireless anyway.” “Not yet, though.” So there’s a lot of decisions that go into that, and the number of prototypes that go into that.

And so when I put together a demo, I’m having to learn all of these technologies. I have to learn when they break and fail… And then what a lot of people don’t know is a lot of these demos, if you go look at GitHub before I do the demos, I’m sending patches to Open Policy Agent, so that it actually has native integrations with Google’s IAM and BigQuery… Because the demo ain’t gonna look good if I can’t really authenticate the cloud-native way. And since it doesn’t support GCP yet, I’ve gotta stop the prep, go and write some open source code, unit tests and documentation, go on Slack, get to know the founders and the people who maintain that project, push the code, get it merged, get a new release, and then use that release in the demo. That’s what goes into those demos when they say is highly polished, because I took a long time to polish it.

Right.

I look at all the rough edges and I don’t lie to you.

You shaved all the yaks.

The demo - you can’t lie. I can lie with PowerPoint all day. But when I do that demo, that’s reality. So I write all the code, I try to patch the things that need to be patched in a way that benefits the rest of society and the community… And so when I give you the demo, I’m showing you how the world is. Not how it could be. Now, you’re right, I do focus on the happy path, because there’s so much sad stuff to be sad about, you know what I mean? A lot of people struggle at work with all the stand-ups, and trying to get promoted, and things breaking all the time… When people come to a conference, or you go, you want to feel a certain way. And so I choose to inspire.
[29:56] And I usually remind people of the pain. It’s like “Hey, isn’t this what we’re dealing with?” and you get all the laughs, because people know “Yo, that’s legitly what we deal with in our jobs.” And then my job, I believe, or the route I would like to take is, “But here’s what it could be. Here’s what it could be, and I’m going to show you that this thing isn’t something that’s in a distant future. I’m going to show you that this is real right now, and you can go check out the code on GitHub.” And that level of inspiration that comes from that - you’re right, that happy path gets people out of their chairs, they’re going back to the computer, like “Yo, I’m gonna try it right now.” Come on… I’d rather have that, and I’ll take the bit of criticism that comes with it.

A hundred percent.

Who’s criticizing this? I haven’t heard this, this anti happy path Kelsey’s put out there? Who’s got the sentiment?

Oh, it’s just a few curmudgeons, I don’t know…

No, no, he doesn’t even have to give examples. I’ve seen it directly on Twitter.

You have?

Yeah, I’ve seen it as well.

Yeah. People will just be on Twitter like “Hey, I’m showing a thing”, or someone will share a video that I did, and they’re like “Oh, that’s just the happy path.” Actually, it was the most cool presentation I got to watch personally. His name is Gregory; there was one year at Monitorama there was a whole talk, it’s on YouTube right now, “Everyone can’t be Kelsey Hightower” is the name of the talk. And it was basically pulling on this thread, but in the most honorable and respectful way possible. And he was so hilarious, the talk is hilarious. He’s like “Look, when Kelsey has to do something, he can do all of these things, it’s like magic. But you ain’t Kelsey.” And then he reminded people of the realities that this stuff can break in day two and day three, he was kind of showing that there is a lot of complexity underneath the hood that Kelsey isn’t quite showing you… And I appreciated that, but he was really just being very honest that there’s a lot of work that goes into building systems like this. “So in order for them to feel like magic, there’s a lot of work Kelsey didn’t show you, that he’s actually doing. Let me show you some of it.”

The thing is, that’s criticism that I was open to, because he’s right, I am making it look too easy. And I think it is clear. But I’m also the same person that writes Kubernetes the Hard Way, that shows you every command and keystroke behind a system like Kubernetes. And it’s tedious. If you’ve never gone through Kubernetes the Hard Way, I make you generate every SSL certificate by hand. No scripts, no tools. So I kind of know both sides of that equation… But yeah, a lot of times people just say, “Someone else does his demos. Kelsey can’t even write code. He’s just an entertainer, he’s just a performer.” You get this sometimes live at the Q&A. Some people will say, “Do you even know what CAP theorem is?”

Dang…

And then what I tend to do is say, “Excellent statement. Not a question. But let me break through CAP theorem for you, and show you how Etcd implements it through the Raft. So here’s the Raft paper, but here’s the shortcomings of the Raft paper. In the real world, I’ll show you Etcd and the Raft log, and I’m going to show you what happens when the Raft log breaks. And the reason why I know this is because this is what we work on at CoreOS. You can just run a git blame to answer your own question, but you haven’t gone that far. But I’m gonna break it down for you, because I’m not just talking to you, I’ve got to talk to the camera, so that the next person doesn’t get the illusion that the magician doesn’t know how to put together the trick.

Right. Is it because you’re just that good, people gotta find your cracks? You think that’s what it is? Because you’re very talented, and you’ve been very successful, as Jerod said. You know that, everybody knows that.

Yeah, I’ve learned that – when I watch commentators be like “LeBron James is trash.” How the hell can you say that? Whether you like him, whether you like the teams he’s played on… You can’t say LeBron James is trash?!”

Right.

That’s how sports works, though. You have to trash talk. It’s like the name of the game.

Sometimes that’s how you make your name, right? Like, you make your name by going up and calling LeBron James trash. Or maybe you’re trying to make your name by saying “Kelsey Hightower ain’t got nothing.” It’s weird, but it’s a thing…

You know what, that’s the thing… The world deserves critics. As someone who gets praised, I accept the criticism. But that don’t make the criticism true.

[33:51] That’s true. Yeah. Well, what I mean is we have to – when we find somebody that inspires us, that can cast this vision, that can shave the yaks to get there, that can patch the kernel, that can get the releases released etc. when you find somebody who’s that much of a renaissance person to be able to do all those details, to do a demo, to cast vision, it makes you think, there’s gotta be something not right here, because this person is just too good, and so you’ve gotta find the cracks.

But you know what - I actually like it. I remember getting criticized a bit, and I was like “Okay, I hear you. So let me tweak in tune.” So what I would do then, I remember - I just did the code from scratch on stage. I said “You want to see it all? I’ll show it all. Step by step. First, you start with the empty canvas, then you do this, then you do that.” And what it turned out to was - it was also helpful for my style, because it gave people more context. So now, when I’m showing things off, or even when I’m telling the story, I’m very patient now. I zoom out and give context. “Hey, my personal journey was this. The technologies I was exposed to at that time were this. And so now I’m about to answer your question, but hopefully that context gives it more shape.”

And so I think that criticism that I was getting, they were actually helping me fine-tune. So if I was just being like a nice person, and I hear the criticism, I’m like “Man, you’re inadvertently trying to make me better in some way.” Now, some stuff you’ve got to ignore. Some stuff is just meant to try to tear you down and hope you fail. I can’t do a lot with that. But when someone’s like “Yo, you need to learn how to type.” Seriously, someone – I was working, getting into software development early in my career, and I wasn’t good at touch-typing. And so I’m doing the thing, and I can, most of my fingers… But he was like “Yo, bro, you’ve gotta learn how to touch-type. You’re too slow. It’s inefficient. This is hurting your productivity.” And I was like “What are you talking about? I’m writing code, it’s in production…” But I thought about it, I was like “Yo, let me just go and learn how to touch-type.” And this was like 2006-2007, and I remember going to go get – what was that, Mavis Beavis Teaches Typing? …like, a little thing you install on your computer..And I’m sitting there at night and I’m like “I’m gonna learn how to touch-type.” And I did. And I remember it took me about a good six to eight months to kind of get to the point where I could let people see me type… Because I was worried at that point, I was like “Wow, people looking at me like “Why are you doing this? You don’t know how to type.” But boy, once I learned how to type, I’m like, at live demos, “We’re typing everything. Put the camera on the keyboard. We’re good over here.”

“Put the camera on the keyboard, yeah. Look how fast this is moving.”

“Watch my fingers! They’ve gotta go!” So cast vision is a big part of success; be able to not just see the future, but be able to demonstrably put the feature out there, and get people to re-explain what you think that feature is, and believe in it.

Yeah, I mean, I was telling this group about – they asked me what technology I was most excited about learning and getting better at. And I asked them “Yo, you know what the number one programming language in the world is?” And they were like Python, Ruby, whatever. Java. I was like “I think it might be English. It might be one of the spoken languages.” Because if you really wanted people to do something, how do you get them to do it? You’ve got to tell them. You write it down, and then people see the instructions, and that’s what they do. And then if you construct that correctly, it becomes law, it becomes religion, it becomes culture. And so when you take the spoken language and you serialize it into any of those formats I just talked about, watch how people move. People dress a certain way, people talk a certain way, people share and exchange ideas a certain way. And the way we encapsulate ideas from the mind is we put it in language. And then we can actually like transport it to the future through books. You write it down, you print the copy of the book, and five years from now someone finds the book, and they get everything you were thinking at that time in this nice, neat package. And guess what it does? Then it goes into their mind. And then it may inspire them to think a certain way.

So when I think about like tech, you’ve got to zoom out and be like “Yo, spoken language is one of the most powerful tools that humans have ever created. This ability to exchange thought…” That to me is pretty dope. So using that programming language to get all these other things to work - and probably engineers listening to this are like “Kelsey is in no way talking about it.”

“He can’t even touch-type.”

[38:06] And I say, “Hey, what do you think is an important part of the development process?” And I usually use this very quick analogy - you take a software developer and you give them the computer, their favorite IDE, everything, and you just walk up to them and say “Alright, build it.” And what’s the first question they’re going to ask? “Build what.” And that tells you that software programming languages alone are not enough to get something built. There’s a whole process that stands before the keyboard even gets into the action.

Yeah, you need to have the problem itself, which takes living, interacting in a lot of cases… You don’t have a problem in isolation, you have a problem in groups. And you have empathy for people who are feeling that pain. You’ve got a large majority or a small minority that feels the pain, and you want to exercise your ability to innovate, to solve for that pain. And that’s life. That’s how it works.

Well, that’s one of the problems that we have often, is we don’t even have a problem sometimes. We’ve got a solution. That happens a lot… “Here’s a cool solution, here’s an awesome new technology” and it’s like “Yeah, but what problems is it solving?” Or even just one. Name one problem that it actually solves. “Well, we’re still looking for the problem.” That’s a challenge as well.

Yeah, I got a question on Twitter… Someone’s like “Hey, what do you think the root causes to a lot of the problems we see in the software and DevOps world? This stuff gets brittle, it gets complex… What’s the source of complexity?” And I said “Man if we were honest and we thought about it, think about how software gets built, the very nature of it.” You start, maybe you get a little bit of planning before you start… But once that ship starts going, it’s a nonstop, constant stream of new features, bug fixes… And you get like a week or two to think about the bug fix, and to fix it. Same thing is true for some major feature sometimes. People just want it now. So you get someone that just comes up with a basic design… “We can always change it later.” It seeps into the process. And after five years, what do you think that software’s gonna look like?

Right.

You bolt on Kafka, you bolt on Redis, you bolt on the cloud, you bolt on Kubernetes… And you zoom out like “Yo, what did we do?”

You just keep bolting.

Like, “Was this our intention?” And I just think it just happens over time through the flexibility of it all.

The best MacOS release in my lifetime was Snow Leopard. You guys remember Snow Leopard?

I was gonna say Snow Leopard, as soon as you said that. That was the release where they said, “We’re not going to do anything but clean this thing up.”

[40:29] “You know, MacOS 10 is made up of a lot of projects, over 1,000 projects. And for Snow Leopard, we’ve decided to refine over 90% of all those projects.”

No new features, because Leopard was a big, feature-field release, and then 18 months of just fixing and refining. And Snow Leopard was just rock-solid, reliable, good software. And I thought, “Yeah, they’re on to something here.” And then they haven’t done it since. [laughter] So maybe their focus groups told them otherwise… But man, Snow Leopard.

Do you have any thoughts or opinions on System Initiative then, considering this reshaping and rebuilding DevOps from the ground up that Adam Jacob has valiantly, deep in stealth for like years, basically, comes out the cut last week with us, and announces System Initiative? And it’s gonna be open source, and it’s a new world. Did you dig into this at all, did you pay attention to what was happening there?

Yeah, I mean, I know Adam since the Chef days. I was at Puppet Labs. He’s the co-founder of Chef, and I’ve spoken at many of the OpsCode’s, events, the Chef confs… And I remember when they showed me habitat for the first time; that was kind of their take on a modern approach to automation. Chef was one way of doing it; they learned a lot since Chef had came out…

[41:49] I remember speaking at that Chef conf, and they showed me Habitat the night before. And I was like “Yo, this is pretty cool. Let me go redo my keynote.” And I worked with the team, and really excited about Habitat… And it was definitely a new way of working. In many ways, it was kind of taking a similar approach, like this infrastructure as code model… And so it gave people arguably better tools, but it was still working at the same level of abstraction. And so I kept up with Adam over the years, and he showed me lots of prototypes of System Initiative over the years. I remember he was so frustrated with Kubernetes, because you can tell they were doing some integration work with that side of the equation… I was like “Adam, stop fighting Kubernetes. You keep wanting it to be something it’s not.” And I remember we jumped on a Hangout, and I showed him, “Look, this is why Kubernetes does what it does. Whether you like it or not is a whole different story. But you’ve got to come at it from a place of understanding of how it works.” Kubernetes is not infrastructure as code, Kubernetes infrastructure as data. Very different things. And if you check that approach, then you realize that you don’t ever have to write the YAML by hand, you can generate it. You can use any frontend you want.

And so he showed me some early prototypes of it, and then they released it. And I pinged him, “Hey, let’s do a Twitter Space real quick.” So we did a Twitter Space last week; it’s still up there, I believe… And I asked him a bunch of questions about it. And mentally, I’m like “Yo, this is another infrastructure as code approach. There’s this visual component, and in many ways, it feels like if Pulumi had a visual designer, what would you end up with?”

There’s some cool things I think it does, which is it makes validation not an afterthought. And so what Adam has done with System Initiative is they start with a digital twin. So typically in software we start with writing the product or code first, and then we figure out how to test it. And if you look at most people’s unit tests, they try to emulate how people would use the system, especially if we start doing end-to-end integration tests. But no one really makes that a core part of the process. Now, System Initiative works the other way. They start by creating a digital twin of the real world in a way that you can just kind of run it before you start touching the APIs, like TerraForm would do. So a dry run is like the first thing you do. You model the world, and it gives you a place to hang all your policies. So this is cool.

But then the question I asked was like “Yo, this looks real cool.” It’s always on, so it gives you that kind of Kubernetes feel of when you change something here visually, or do the kind of programming language… I ask him “What’s the programming language though?” Because he was all about Ruby back in the day. And since Ruby, there’s been a lot of languages that become popular. You’ve got Rust, you’ve got Golang… But none of those are really great in terms of building custom DSLs… And he said TypeScript. And I was like “Adam, I’m allergic to JavaScript and TypeScript.” [laughter] “I’m gonna need to take Benadryl to use this. And I was like “Yo, man, I don’t know about TypeScript.” So I had a prediction. I said, “Hey, I predict that, like Kubernetes, you’re gonna have to have multiple frontends to this. Pulumi has probably done a good job of saying – people have their favorite programming language, so Pulumi tends to work with whatever language you want, and then you can kind of use Pulumi as a core engine to manage those resources.” And I’m like “TypeScript?!” And he made a good case. Like, fair enough. Most frontend developers, it’s probably the most popular language in the world.

Exactly. Yeah.

But I don’t know if it’s popular with the people who do that kind of work. And will they learn TypeScript just to do System Initiative? So my prediction is they’re going to ask support for other languages, and he said, “Yes, people don’t know it yet, but System Initiative kind of already at its core already supports multiple languages. It’s just not what they’re leading with. They want to focus on TypeScript.”

So my overall thoughts are, while we automate the world, we’ve got to make sure we don’t make the same mistakes we did with config management, and the initial DevOps movement. We were all about automating everything. So we have Puppet, Chef and Ansible. And then Docker comes along and says, “There’s one thing better than automation, which is abstraction. Why are you all still messing with tarballs, and apt-get and yum, when we could just have a better package that makes those things go away, and move them to a different level of abstraction, so then we get Kubernetes?

[46:08] Those two things didn’t come from the infrastructure as code community, right? You had to have someone step back. So when I look at it, I think they’ve taken a step back. We’ll see if people like this visual designer, but I do think the biggest innovation that I saw there was the digital twin. Start with the thing that lets you do the analysis, policy and dry runs as a first-class citizen. I think that part will be a game changer.

Were you bummed to see their focus on AWS first, versus GCP, given your past?

No, I think – look, I worked at CoreOS, and we had the same focus. And I remember it, before I left CoreOS, I was a product manager, and we had two core beliefs at CoreOS in the very beginning. We basically built our own tech stack from the ground up. Etcd was the distributed configuration system, we had CoreOS, the operating system that was kind of tailored, focus to just running containers. So the only thing we installed on there was like Docker and nothing else. And I remember being the product manager and just like “Yeah, that’s very dogmatic, because in the real world, when you have to manage like an HP ProLiant server, in order to do firmware updates, you’ve got to be able to actually run the RPMs, or the Deb packages that they supply, and CoreOS doesn’t do that. So it’s gonna break down on-prem, where people still need these utilities.

And I remember pushing that “We’ve got to support Ubuntu, at least. We’ve gotta support maybe even Red Hat, at least, with some of these CoreOS tools.” And so when we had our cloud product, we were asking ourselves, “Yo, number one, where are our customers at?” And it turns out AWS is probably like the best place to start in terms of sheer volume… But I’ll tell you one transparent thing. AWS also needed the most help, because of how complex the product offering was, and what the native tools allows you to do, that we felt like “Man…!” With Kubernetes - for example, at CoreOS, we had a Kubernetes distribution. Why would you go to GCP with a Kubernetes distribution, given how good GKE was? Why not start on a provider that didn’t have a Kubernetes managed service? So that was our calculus. So when they said they’re going to focus on AWS, as a startup, that’s what you’re supposed to do. The question though - will they get the same community that HashiCorp got, that can actually fill in the gaps for all the other providers? Because they’re gonna need that to be really competitive.

Yeah, that was one of the things that I asked Adam about specifically, was the community side of it. Obviously, there was a big community built up around Chef. He’s been in the open source world for many years, and so he’s been on the show before, talking about more community-oriented topics, and it seems like he’s well skilled and suited to hopefully get that community going… But these things are kind of ephemeral, or what’s the word…? Like, you can’t just conjure up a community… And you’ve done it once, it doesn’t mean you can do it a second time either.

You are 100% correct… Because the power of these tools now is the network effect. When you ask someone how you should do observability these days, they’re going to push you towards Prometheus or OpenTelemetry. And what happens with that then is that a lot of the sharing and knowledge and investment of time goes into that area, and then you create the snowball effect. And the same thing is true of TerraForm. If you want to start today, right now, automating something, TerraForm has such a breadth of knowledge and community and people that know what they’re doing… It’s really hard to decide not to use that, and do something now.

So you’re right, one thing he’s going to have to demonstrate is that 1) the community has all the right extension points, APIs to do what they need to do… I think that’s one thing that helped Kubernetes adoption so much, was you could actually just build your own extensions without dealing with the core, or feeling like a second-class citizen. OpenStack didn’t really have that. Mesos made that mountain really hard to climb… And so whenever you get these kind of tools or frameworks that allow you to extend them and then the ecosystem can blow up, I think now – I don’t think you can have an infrastructure tool or automation tool without a community. I don’t think you can survive.

Yeah.

Whatever became with CoreOS? I didn’t follow the path.

Me neither. I went to Google. [laughter]

[50:07] Okay. We don’t know. Something happened…

It was acquired, wasn’t it? By Red Hat.

Yeah, they got bought by Red Hat.

Okay.

One thing people don’t know is – you know, people know that CoreOS did Etcd, kind of the storage engine behind Kubernetes, key-value store… We kind of helped push the idea that the container runtime could be swappable. So the container runtime interface, the thing that lets you use Podman from Red Hat, or Docker - back in the day we were pushing Rocket.

Rocket. Yeah, I remember Rocket.

RKT, wasn’t it? RKT?

Yeah, RKT.

That’s right. I remember that.

Yeah. We had the T-shirt.

Oh, yeah.

Lots of people had the T-shirt. It had a dope logo, too.

I loved that T-shirt.

That’s how you get the community started, is you get the community T-shirts out there. Step one.

That’s right.

Swag is the ignition to community bootstrapping.

That’s right.

And so we also did the Container Networking Interface, CNI… And [unintelligible 00:50:53.03] came up with the term operators. And operators came from this idea that we’re not just going to have a Kubernetes offering, but we had this thing called Tekton, which was basically thinking about a shift in the landscape. So we brought in Kubernetes, but we had this UI, we had this dashboard… And we had this marketplace of these operators; these little automation tools, because Kubernetes was so extendable.

And when we got to that point, I felt like we now had a pretty decent enterprise strategy in terms of what people would pay for on the product side, just versus pure open source and support… And so at that part, that’s when I went to Google, and a few years later they got bought by Red Hat, and got integrated into that whole thing… And just keeping up with some of the folks, I think CoreOS got sunsetted in favor of Fedora, and its way of thinking about a container-optimized operating system. Etcd is still here, it belongs to the community, Red Hat continues to be a great store, Kubernetes is as big as ever… CNI is the de facto standard there… But I think the thing we used to think about in terms of CoreOS, I think that’s gone.

Still, lots of contribution; long-standing contribution that fell out of that effort, so that’s cool.

Don’t we continue to use a CoreOS image on Fly? I thought we did recently.

No, not anymore… But we used to back –

We just talked about that recently. How far back was that? 2022?

I don’t know. Time is a vortex…

Yeah, I think some people went to Flatcar Linux, which was I think a pivot from CoreOS… So I think there’s a few CoreOS things out there like that. But if you’re on CoreOS right now, you probably should just stop, I don’t think it’s being updated. [laughter]

Yeah, I know we were, probably like 2020, maybe 2019… But I think at some point – because it got end-of-lifed, and it was like “Hey, get off of this”, and we did. I don’t know, Gerhard would speak to that better than I could, but… Yeah, we ran it for some years.

Can we talk about a different area of the cloud? I want to talk about 37signals’ choice to move away from the cloud. So back in October last year, DHH says “Why we’re leaving the cloud”, explains all the pain in the cloud. And then, I think about a week back, or like days back, basically, seven days back, “We have left the cloud.” What are your thoughts on this? What are your thoughts on… At what point does this make – or does it make sense?

So I guess now I don’t work at a cloud provider.

You can be free.

So let’s be clear. At the time of this recording, Kelsey has retired from GCP.

[laughs] That’s right.

I cannot speak on behalf of Google, because I don’t work there.

Okay, great.

So we’ve got to make sure we lay that part down. So it’s not a disclaimer, it’s a statement of fact now.

Correct.

When he did the first post about rethinking the cloud in general, I remember DMing him and saying “Hey, we should probably do a Twitter Space, because I have lots of questions.” So the first thing I think is - in order to do what 37signals is doing, the amount of skill you have to have… Because remember, they built their own orchestrator to pull this off. Most companies are going to the cloud because they don’t have the bandwidth or skill to do any of that. This is the same company that created Ruby on Rails. So for anyone confused out there, this is not your run of the mill dev shop. This is a tight-knit group of people…

Right. This is innovated here/invented here, situation.

[54:10] Yeah. Well, I mean, extreme level of focus and talent. Just look at their product strategy - extreme levels of focus and talent. So if any company in the world can pull this off, it’s gonna be 37signals. It’s just a choice. Now, the interesting thing about this whole conversation is they made the cloud choice anyway. At some point, they decided to go to the cloud.

Yeah.

And the thing I like about their story is that we get to see the full circle. Most companies don’t stick around long enough to give us the full circle. And what’s happening right now is the full circle. “We went to the cloud for these reasons. These reasons are no longer true.” It turns out, there’s a lot of other industries that do this all the time. The gaming industry is notorious for this. You’ve got a new AAA game that comes out; you have no idea what kind of numbers it’s going to do. It’s like a movie. You put a lot into it, you know on the release that you’ve got to be ready for explosion. So the cloud is really good for dynamic capacity, and it’s worth the premium. So you pay AMC to show the movie because you don’t know how many people want to see the movie. So you’ve got to pay all the movie theaters to play the movie, and they get their cut. But when nobody wants to watch the movie no more, it makes no sense to be dealing with the movie theaters, because you don’t need the dynamic capacity that movie theaters offer, at that premium. You need to just pick like a Netflix, and just stream it there, and you’re good. People watch it occasionally.

So I think the other side of this equation is that the gaming industry, when a game is not as popular, it’s like four years – like, if you’re still playing John Madden or NBA 2k21 - like, nobody playing that right now. Now, it might be 50,000 of y’all globally that just really like the mechanics of that version, but everyone’s moving on to the new one.

The diehards.

So if you were hosting the ‘21 edition of that game, you probably just want to move it to a colo, keep it running on three or four servers, and it’s good. There’s no need to pay that premium. And so I think what 37signals have said - they’ve looked at some of their properties… It’s in their blog post, “Hey, look, we’ve got predictable traffic patterns now. We bootstraped this thing.” But the thing that was dope, he said, “Look, we went to Kubernetes.” And it was straight overkill for them. They didn’t need all the stuff Kubernetes does, and when people – and I say when people start cosplaying a cloud provider, that’s when you start getting into trouble. When you start trying to put your database in there… Now we’ve gone too far off the deep end in terms of what Kubernetes is optimized for.

But the thing that was dope about this full circle - he said “Hey, we realize that all the work it takes to get into Kubernetes makes it real easy to leave the cloud.” Once you have a container image, you can just run it on Docker. Once you have a YAML description of what’s supposed to be in Kubernetes, you can almost feed that logic to any orchestrator you want. And so they were just able to build just enough orchestration tools that they needed.

So them going back on-prem, and the fact that he said they’re saving 7.X million dollars? I don’t know about y’all, that’s real money. For a company that’s all about profit… I think they’re getting criticized for being profitable, which is insane.

I don’t get that.

But they’re like “Yo, why do we spend money we don’t need to, when we can optimize?” And so I look at that whole journey - like, number one, I learned a lot from that journey myself… It’s like, look, cloud cannot continue to be more expensive than it was before, because a lot of people that get cloud that it should optimize itself over time. So so far, I think we’ve been on the new feature train. Cloud allows you to do even more stuff, new database types, ML, etc. etc. But I think some people are looking for cost reduction now. “I need the technology to get cheaper.” And I think smartphones are in the same trajectory. We have accepted price increases for a long time… Now we’re all like “Hey, hold on, man… This phone is kind of like last year’s phone.”

“I’m gonna hold on to my old one.”

“I’m gonna hold on to the old one, or the next time I might just go back to the Nokia flip.” You know what I mean? Like, “I don’t really need all this stuff.” So I think we’re at that stage now in cloud where we’re going to have some careful evaluations. If all you’re doing is hosting a web app - and you can do that very cheaply…

[58:03] There’s another part of this equation though that I still for the life of me don’t understand why the colo providers haven’t upped their game. I watched VMware do it. People were like “VM is gonna go out of business.” I don’t know if y’all – remember when Kubernetes came out, and cloud-native people were like “Alright, VMware is now done-done.” We thought OpenStack was gonna do it; it didn’t do it. We thought cloud was gonna do it; it didn’t do it. But when Kubernetes came out, we’re like “Oh, people are just now moving away from VMs, on-prem and cloud? Alright, that’s the end of VMware.”

So what did VMware do? VMware was like “Yo, how about we go get Heptio, we get in the game?” And if you look at their offering right now, they can give you Kubernetes. It turns out, when you zoom out and you look at how a lot of people are using the cloud these days, there’s a lot of container usage of things like ECS and Kubernetes. And guess what? Your existing vendor has kind of met you where you are, even in the cloud. So if you are really going to the cloud for Kube, and you look at what’s on-prem now, you can get a pretty good Kubernetes offering on-prem.

At this point, what is the baseline for colo? You just can’t give me ping, pipe and power and say “Good luck. Go order some stuff from Dell.” Come on.

Right…

You’ve gotta be more like Packet. There should be an API to get some bare metal. And if they can just get to where cloud is with IaaS, “Give me a machine, give me some storage, give me a load balancer”, they don’t have to do all the other stuff. I think a colo would actually be a viable choice, given the fact that we have APIs like Kubernetes and Prometheus on-prem.

Didn’t they build their own server, though? They built like a 192-thread Dell R7625S, I don’t know what that is…

Well, I mean, they bought a big server…

Okay. I mean, that’s their own stack there. That’s not metal as an API. That’s like, they ordered a server, and sliced it up with KVM.

Old school.

Yeah.

Yeah, but they’ve got the skill to do it.

No, no, I’m not criticizing. I’m just saying they didn’t even go to a Packet/Equinix, which is what Packet was acquired by them. They went to a whole different route.

Well, because the premium is still a bit too high, right? I think the premium even on those services is a little bit too high.

Yeah.

But again, remember what their objective was; their objective probably was to save the most cash possible. If they left the cloud to save 10%, I don’t think that would have been enough. But if you think about them leaving the cloud to save – I don’t know what percentage of their budget this was… That’s significant. And I think the only way you get to those numbers though - you may have to order your own server from Dell, and you may have to consider custom tooling to take full advantage. If we throw VMware on top of that, there might be a little bit too much overhead for what you’re trying to do. Those licensing fees may add up a little bit…

So this iteration - because we’ve also got to think about this. This is just iteration, one of what they’re doing. So iteration one says “Hey, get the biggest server you can. We can carve that thing up with basic –” KVM is really good these days, if people haven’t checked it out. KVM is actually pretty solid in terms of its APIs for creating a managed VM. And so if you take that, plus the fact that they had container images already, they have a good description of what they want to do, they’ve already done the heavylifting to make sure that their app can work well in an environment like this… So it’s no surprise that in their iteration one - I think they did what most good engineers should do… Start with the minimal thing you need. Go buy a big server from Dell. And what’s the minimum we need to do to that Dell server to get what we need? And I bet you someone probably went through a few prototypes and was like “Yo, KVM is freaking good right now. You can just create a VM with a two liner. Alright, so we take that plus Docker. What do we get?” And I think that’s what they showed to us.

There’s one more layer, MRSK. What is this MRSK in this world they built out? I know there was a tweet, and you might be talking to DHH soon about this, or…

I mean, I haven’t played with it, but the way they described it was - it reminds me a little bit of HashiCorp’s Nomad… What does it take to pull a container image, run it and keep it running, and maybe manage some of the dependencies, like “Is it running? What’s the IP address of it?”, maybe a little bit of troubleshooting.

[01:02:01.10] So if you asked yourself, “How would you build that from scratch right now today?” Like, if you were to start from scratch, assuming Docker was on the server already, what would you build? So basically, you would have a little tool that read some type of config, or has some flags that said “I want this server to run these three or four images.”

And so when you look at the tool that they built, I think they’re starting from those kind of base, fundamental principles. Like, do they really need this auto-scaling, multi-cluster API? I don’t think they need that. And so for them, they look at Kubernetes and say, “Hey, if we just have 5% of what Kubernetes does, we’re good.” And so I think that’s what they built.

And remember, my mindset still is this is v1. They took v1, and they got to a point where they can now do a lot of things with just like 5% of the logic. And then the real test would be what they add next. So give it six months, and watch that change log. And that change log will be them in real time showing us what was missing. “Okay, we needed something for secrets. We needed something for volume management, because doing that out of band was a little hard.” And what I think will happen is in about a year or two, they’re gonna get all those features to make it generally usable by other people, and that’s when you’re going to get to a point where you say, “Okay, that is a minimum viable product for working that way. Like, Ruby on Rails - that’s the minimal viable product for working that way. If you want to do something different, then you may need to go to like Vercel, because you might want React or something for your frontend.” Maybe Rails is not good enough for that type of activity.

So I think that’s just where we are… I like it… See, here’s the thing - even though I’m in this Kubernetes space, and I spent a lot of time in the serverless space, anytime I learn a new technology, the first thing I do is I break it down to the lowest elements, and try to ask myself “What would it take to rebuild this on the fundamentals alone?” And I think that’s what that represents, so I’m gonna pay attention.

Good.

For the old school Ruby devs out there, they describe it as “Capistrano for containers.” So if you know what Capistrano is…

Oh, gosh…

It’s been around as long as I have. And I don’t know, I guess they still must think fondly of Capistrano. I certainly use it to deploy some apps.

I’ve got a feeling though, from an industry standard perspective - just like Linux… Linux is big, and most people don’t need the full Linux distro. It’s big. Most people will never use the tens of thousands of packages available in the upstream repositories… But a lot of people will never start from scratch either, because that’s equally as hard, and requires a lot of maintenance. So I think Kubernetes and Nomad and those tools will continue to be mainstream.

If you have a large team, you’ve done lots of acquisitions, it’s hard to get everyone on the same page… So I think those things will still be the bulk of what people do. But I do think there’s going to be a lot of these smaller utilities that’s going to make a resurgence for people who are like “Yo, can’t we just peel off the lightweight version of that thing?” Because a lot of tech we have these days is the lightweight version of the big thing, and then the lightweight thing proves it’s just enough. In many ways, Golang is the lightweight version of a lot of these bigger systems that we used to have… And Go is like “You know what? We’re going to focus on this one little niche, we’re going to remove a lot of the language features and constructs that you see in things like Java, or things that you would find in Rust… What is the minimum viable language we need in Golang to build the things we want to build?” And look what happened: Docker, TerraForm, all of the HashiCorp tools, all the CoreOS tools, databases… CockroachDB, all of these things are built in Golang. So this thing emerges as a lightweight version - some people would say lightweight version of C++, but I think it’s its own thing, that takes a lot of experience from using all these beautiful languages and saying, “But what do you need for the type of systems we want to build?” And so I think we should always have room to carve off the lightweight version to see what we do next.

[01:05:47.24] Yeah. It happens a lot, and I think it’s a good way to make progress, is you have like a thing, and then you add to that thing, you build upon it… And then eventually you’re like “You know what - what was good about this thing? Let’s start a new thing that has the good stuff. Let’s leave all the cruft, and then maybe it has its own ideas, and we can burgeon from there.” We see it a lot of times; even products - look at like Google Docs… It’s like, Microsoft Word became to the point where it’s like nobody knows all the features that are in Microsoft Word… And Google Docs is like “What if we just took the core of an editor, or a Word Doc, whatever it is, and just gave it one new thing, which is like web collaboration?” which turns out to be killer. “And like nothing else.”

Yeah, we have a tendency in this industry to keep throwing stuff at software until it’s so bloated that it either breaks or gets abandoned.

Yeah.

And whenever that happens, I think a community splits off and says “Hey, same thing without the baggage.” Because it’s really hard to delete stuff that gets used by real people… So the best you can do is get these spin-offs that just take the best parts and start anew.

It reminds me of that guy - what was his name, Adam? …from Hog Bay Software. We had him on the show… He builds like little Mac apps. Jesse…

Jesse Grosjean.

And he talks about his love for his 1.0 products. So he builds indie Mac apps, and he talks about his 1.0s. He always hates beyond that, because he’s just like “It was what I wanted it to be, and now here I am, adding features and like feature requests, and I have customers, and I have to keep working on it. But the 1.0 was what I wanted to build.”

Two decades as a solo indie Mac. That was Jesse Grosjean… Kelsey, if you’re gonna go back and listen, it was episode 492… But that was a good one, because he talked about the beauty of the 1.0. And we asked him “Do you want to keep adding to this stuff?” He was like “God, no. I kind of like solved that, and I leave it as a 1.0, and if I want to make something new, it’s a whole new product.”

Yeah.

And you keep that blissful, perfect world of the 1.0 there. “And if I’ve got new ideas, it’s going into a whole new thing.” That’s a TL;DR of that episode, but I would encourage you to listen to it.

Yeah. Author of the TaskPaper app. TaskPaper… And then he came along and he’s like “I’ve got an outliner now.” It’s kind of like, this looks kind of like TaskPaper, but it’s different. He’s like “Yeah, it’s a different idea, but it’s still me.”

Same core. And he’s built on the same ideas and stuff, you know…

IT sounds like that Unix philosophy, you know what I mean? Like, do one thing well, and once you start breaking the bounds, you start adding too many flags, it might be time for a new tool.

You’ve got top, you’ve got htop, and you’ve got btop.

That’s right.

I just installed the btop yesterday. I’m like “Man, I didn’t even know about this tool.”

[laughs]

I saw – I think it’s Fatih, from the Go world… I saw something he was talking about and I’m like “I’ve gotta install btop.” So I installed btop yesterday, I’m like “This is amazing.” But I still use htop, too.

You know what’s funny - imagine if music worked that way… People have their favorite albums; you know all 12 of the songs on that album, and that’s it. But imagine if that album kept adding songs. They just don’t sound like the same album anymore. Like, once you get past track 13 into track 72, it’s a whole different vibe.”

Right.

That’s the cool thing about albums too, is when you listen to them, you know what’s coming next, and you groove, the whole time. Like, I just listen from track one to track ten… I’m on a lot of stuff. Right now I’m on my Led Zeppelin Greatest Hits. I’ve got the four-album greatest hits from Led Zeppelin, and I’m just like listening to that nonstop, while I’m driving now… For that reason, because I know what’s coming next, you know?

Well, guys tell the kids what an album is, so that everybody knows what us old folk are talking about… [laughs]

Oh, yeah, you know what? There are a lot of people – it’s funny, my daughter was listening only to singles. She only knew playlists, and mixing and matching.

Oh, yeah.

And I think I remember just like driving and just playing an album from an artist; like, one artist. Some of them have intros. Like, before the music starts they might say a few words, or they’ll give you like a little bit of experimentation, a little kind of sound they’re going to play with, a little trailer… And they just go straight through. And the good artists, I think, try to produce albums where you ain’t gotta hit the skip button.

Right.

Yup.

That’s hard to pull off. Like, to have a classic album where you don’t skip nothing.

It is hard, yeah.

[01:09:53.04] But that’s the album, it’s like a movie.

Alright, so you did System Initiative, we did Kelsey’s take on 37signals moved to the cloud… I’m curious your thoughts on Dagger. Solomon Hykes –

Move away from the cloud.

Yes. Sorry. Their move away from the cloud. Solomon Hyke’s New Project, CI/CD as code, that runs anywhere. It’s a project, it’s a startup… It looks a lot like other startup/open source things that exist nowadays, which is new in the last decade… Have you taken a look at Dagger? Do you know what they’re doing? What do you think about it?

Yeah, I’ve seen Dagger a couple of times. A couple years ago Solomon reached out and he gave me a little demo of what it was… And I had this tweet years ago, and I said “Pretty much all CI/CD systems are the same. It’s a series of Bash scripts running in a certain order.” You can make it as fancy as you want, you could put a great UI on it, it’s up to you. At the end of the day, you’re basically chained together a bunch of jobs to either produce an artifact, or to deploy something, and make sure that it’s running. That’s the core of it. And so when I think about that process, Jenkins knocked it out of the park, because they embraced that philosophy. Jenkins was like “Listen, you can have as many of these steps as you want”, and they had that little box; you could type in the script, you could paste the script, or you could just give it the whole script file, and then Jenkins would just run it in order, and then tell you what happened in between the steps. Gold.

And now we have more declarative pipelines, GitHub Actions, ArgoCD… We have all of these kinds of declarative ones, but the reason why those work is because the system in which they target can be declared. And so it’s a different take on this. So you saw the Jenkins project kind of pivot. “Hey, we’ve got to add support.” So Jenkins X. “We’ve got to add support for this new declarative model, because there’s a lot more that you can do. And then if we turn it from just Bash scripts to something a little bit more portable, language-independent, like container images that could host Bash scripts, or some other language, or whatever control loop you want, and let people chain those together.” So now, to your question, when I saw Dagger, I’m like “What’s this?” I also saw TypeScript again. [laughter]

Multi-language at this point… I know they’ve gone beyond, but…

That’s why I said the prediction for System Initiative is they’re going to have to go beyond, because I just don’t think you can do that in 2023. People have their favorite languages…

Yeah, exactly.

…and they all have their pros and cons, and it’s hard to tell people that they have to break from what they’re doing to just one tool. So when I saw it - and he actually gave a presentation at VMworld. I gave a five-minute keynote at VMworld, and I saw Solomon presenting Dagger. And I saw that probably about three or four months ago maybe; maybe a little longer. And I saw it and I said, “Solomon, I think I know what it is, or how I would explain it to people.” CI/CD is kind of like when you go get Jenkins or any of the kind of pre-packaged solutions that are out there - it’s almost like buying or using someone else’s factory. They tell you, “These are the only cars we can build at this factory. We’re fitted for these things.” So if you’re going to build something, it has to be within these parameters, and we can build a million of them for you. But these are the parameters, this is all you get.” Dagger feels like the ability to create custom robotic arms. Like, when you go look at a Tesla factory, you can’t make an F-150 in a Tesla factory. You can only make the cars that Tesla makes in that factory. And so Dagger is kind of like this idea of whatever complex step you have - it could be deploying something to a server… I think what Dagger gives you is the ability to create that custom robot for that one step. Would you use Dagger to run the build command? No. That’s overkill. You could put it in there, but that’s like kind of a solved problem. So just use your standard conveyor belt to get the base of the car to this arm. But that arm is going to be really customized. You want it to work a certain way.

[01:13:48.05] And so I think what Dagger gets people to do is if you look at your whole factory - let’s say you’re using Jenkins. You still could use Dagger as part of the [unintelligible 01:13:54.29] So hey, Jenkins, you do the checkout from GitHub, you do the build, you do the packaging. But when it comes to like deploying to Kubernetes across six regions, and the load balancer updating Cloudflare and busting the cache on something like Akamai, I ain’t trying to articulate that and a bunch of Bash scripts. I also don’t want anyone else trying to do that five times throughout the organization. So what we’re gonna do is we’re gonna use Dagger to build this custom robot, give it an API, and then Dagger as an atomic unit. When I look at my Jenkins pipeline, this fifth step is Dagger. Get rid of the 25 Bash scripts; throw the Dagger robot in for dealing with Kubernetes. Throw the Dagger robot in there for dealing with Amazon Lambda. And so I think if Dagger is going to be successful, just like what they did with Docker images, and like having those kind of be ready to go, the units - I don’t know if they really thought about it this way, but this is how I express what I thought about it. You’ve gotta think about Dagger as these customized units or robots. And if they’re shareable - so if I come to a company and say “Hey, what is the best way to deploy to AWS Lambda, IAM, load balancer, the whole nine, and do rolling updates? “Oh, man… That can be challenging.” So Dagger already kind of gives you some “If then, do this” kind of primitives out of the box. It gives you enough feedback loops so that you can integrate with a Jenkins, like a higher reporting tool, something that has a UI, and can decide when things go on and when things don’t go… Man, if I could just reuse those in the organization to say “Yo, to me Dagger is a last-mile technology, that allows people to do all that customization that Bash is just not equipped for.” That’s the way I think Dagger will add value to the world and tools like it.

Well said. I think that’s interesting and informative to me as a person who knows what Dagger is mostly because Gerhard talks about it all the time… Our friend Gerhard, who works at Dagger, and we have a Dagger pipeline in our system, but it’s because Gerhard wrote it for us…

How far am I with this analogy? Because you’ve used it, you’ve seen it a lot more… I would love to hear what is Dagger in reality, how you actually use Dagger, and then maybe thirdly, how far is my kind of analogy of like trying to put Dagger in this box and fit it into the big picture?

You’ve probably seen it more than I have. I mostly hear about it from Gerhard. I know that it’s replaced a lot of the makefiles in our system, which are basically the Bash scripts of deployment; Gerhard likes make more than Bash, although it’s all mixed together into one glorious ball… But it’s replaced a lot of that; it allows us to do fancy things, like have our versions of dependencies live in a single place in the codebase, and have different disparate parts that need different versions pull them in through custom Go code… So I think the idea of this particular robotic arm that you are customizing, like I said, it fits into the way I’m thinking about it. But there’s certain areas of our codebase, as the main application developer that I’m just kind of like “Cool. Is it working? Is it not working?” It’s our pipeline now, but I don’t know exactly how it works.

We get to prod.

Which is why I ask people, I’m like “Tell me more about Dagger, because I’m using it, but I don’t know how I’m using it.”

But you know what? I mean, I think sometimes that is the end game…

Right, to be removed.

Yeah. I mean, as a developer, I’m like “Yo, you built this software that needs to be shipped. If you wrote a book and you put it in the box, do you really want to know how the post office gets it from point A to point B?”

Nah.

Nah, I’m good on that. And so I think if your team is able to take something like Dagger, and help you be super-successful, you’re like “Yo, we got Dagger. I’m sure it’s doing something. I don’t think about it, but the right things happen” - man, that’s gold stamp for me.

Yeah, I mean, in the old days, I would use Capistrano, or some other tool… I’ve used Puppet, and I would wire it, I would get it working, it’d take hours, I’d get my stuff out there, and then I would just pray that it didn’t break somehow. And now I have somebody who knows this world much better than I do, and that’s what makes for good teams, it’s just like complementary skill sets, working together…

[01:17:56.20] In a lot of ways, he predicts what Jerod wants, and they work really well together. And Jerod goes and does his things in isolation, async, which is great for – you know, Gerhard’s in London. He’s not even in our timezone. He doesn’t even work for us full-time. So basically, a contractor, but very much on our team… They get to work and dance so well together. He doesn’t have to step into Jerod’s world too far, and viceversa for Jerod into Gerhard’s world. It’s a beautiful thing to sit back and watch. Because I get to watch them – I’m more of maybe the PM in this world, I suppose… I don’t touch any code here, really…

I’m not even that curious anymore about this area… So I mean, I’ve been doing full stack web development for 20 plus years now, and solo for a lot of that. So I didn’t understand how to deploy a web app, and I’ve done it just through brute force, and attempts and blah blah blah. And when I got to a point where I had a teammate who like loves this stuff, and is good at it - honestly, I’m just okay just checking out and just being “Awesome. I don’t have to think about this part. I don’t have to SSH in.” I don’t have to do that sometimes, but… You know, if I could not think about it, I don’t have the curiosity that I used to have as a young Linux network administrator… When I first came out of school, I was a Linux network administrator. I loved all of that stuff. And eventually, I just got to a point where I’m just like “No, I’m just done with that part of the stack.” And so I just don’t have the curiosity in that area that I used to have. And that you guys clearly have. I mean, Adam is a homelabber. He likes to tinker with the Linux command line tools and stuff, and I still use them all the time… It’s just, I just lost a little bit of my love of that particular area.

But I think you all explaining what I thought the end game for DevOps would have been. You want people to collaborate, but not need to. Like if we work so well together that you and I said, “Look, I got you.” Your specialty is writing the software. You’re gonna give me feedback about what you need from your deployment infrastructure. And my job as a specialist on that side of the house is just to make sure that it never gets in your way. And if I do a good job, the collaboration can be explained the way you all did. I think some people started to say, “Oh, those are silos.” I don’t know if they’re silos. Like, is my ISP a silo? Or are they really good at internet?

Or are they reliable?

Or are they reliable? And I think that should be the Northstar, where - not that you can’t look at the Dagger code, not that you can’t make suggestions on how to improve, it’s just that you don’t have to, if you don’t need to.

And I could write it as well, and he would be like totally fine with that. He probably would encourage it, you know. So there’s definitely that level of trust, and… It’s just that I don’t really want to, or have a need to… And if I did, I would hop in there. I just haven’t yet.

The silo thing, like with your ISP - you know, when they fail you, and you can’t get a hold of them… Like, now we’re starting to talk about like the old school dev versus ops distinction, right? Like, “Well, I thought this was a team.” No, no, no. They’re just a service provider who has millions of people like me, and they’re trying to not talk to us.

But yeah, I mean, if you actually have that collaboration, that communication, that trust between the individuals… We’re supposed to be able to have our particular skill sets, of course cross those, divides, and be able to go across the fence, so to speak, and work on the other side… But Michael Jordan - you’re not going to find them in the post very often, Like, he can post them if he has to, but you want him coming off the dribble, or coming around the screen, and you want Hakeem Olajuwon – now I’m talking dream team. You want Hakeem down there in the post. Because I can’t remember who the Bulls post player was. Horace Grant…

Horace Grant. Luc Longley. They had a few.

Yeah.

See, I was going a different direction. I was gonna talk Days of Thunder. You can learn a lot about DevOps from the movie Days of Thunder.

Oh, why is that?

Well, it’s NASCAR. Cole Trickle, Tom Cruise’s character, says “You told me to get in the car and drive. I can drive. They tell me to take a turn here, or wedge out there, in terms of the mechanics of the car to operate, for drift, and turns, and how a driver drives… He’s like “You told me to get in the car and drive. I can drive. That’s all I can do. I don’t know how to speak car.” Harry Hogge - ops, essentially. Cole Trickle is dev. Harry Hogge is ops. He’s the mechanic in the car, and taking the turn out here, and the wedge out there… But Harry tells him how to drive to get the car to operate the way he has designed it to, and they work in tandem as a team. But Cole Trickle’s character - he’s the driver. He drives. He pushes that red line; he goes to the edge. He’s got the risk factor. Harry Hogge is back at the pit, on the radio, call here, this, go there, whatever, and they’re working as a team. That’s a beautiful dance together.

That’s dope. Now everybody has to go watch that movie, so they know what you were talking about there.

“I know Cole. He always goes to the outside!”

[01:22:34.08] “He’s going to try and slingshot past you, Russ…”

“Don’t worry, I know Cole. He always goes to the outside!”

That’s my favorite line.

“He’s just rubbing. And rubbing’s racing.” That’s the only line I remember from that.

[01:22:44.10] “No, he didn’t slam into you, he didn’t bump you, he didn’t nudge you. He rubbed you. And rubbing, son, is racing.”

That’s right. Alright, can we do a 45-second unpopular opinion, Kelsey?

Oh, gosh… On the spot like this?

We can.

What is your most unpopular opinion you’ve got right now? Lay it down.

Oh, snap.

Oh man, I don’t know… I’ve got opinions. I don’t know if they’re popular or not, but… The one that I think I’ve been talking about lately is I think DevOps was used as an excuse not to make progress. Or DevOps only describes the problem, not the solution. And so the reason why I kind of hold that opinion - I don’t believe it’s fair or accurate that you can allow DevOps to come in and say “Collaboration, being a professional, learning how to work with others.” That is not something that started with DevOps. That started thousands of years ago. And of course, we have to remind people that they’ve got to do it in the software industry, too… But the people I’ve seen that have been talking about DevOps for a decade, and I say, “Show me your setup”, they’re still struggling with pipelines, they’re still struggling with the fundamental idea of how to put software on a server. And to me, if you’ve been flying kind of this flag of DevOps and you still haven’t made progress… I mean, we’re talking decades. We’re not talking like a few years. After 10 years, you’re saying, “Hey, we’re a championship team, but we still ain’t made the playoffs.” Something’s missing here. And I think what’s missing from a lot of the DevOps stuff is straight up real accountability. Because DevOps doesn’t necessarily say you have to be good. You don’t have to be good. You don’t have to be good at any aspect of it. You don’t have to actually know TerraForm. They say tools don’t matter sometimes… And it’s not everybody. I want to make sure I’m avoiding general statements here. But I think my high-level premise is that it’s been used as an excuse more than a tool to make progress.

How to make progress then? What do we have to do different?

Just like anyone that has to manage a team. You look at the people you have… “Here’s what we got to do to get things done. Number one, do we even understand what done looks like? Do we actually understand what the problem is?” If you get those two right, then the next thing is do you have the actual team members to do it? And I’ll give you a clear example. When I go and do whiteboard sessions with enterprise companies, and the network team is like “Hey, we have no idea how to get involved with the DevOps stuff.” I say, “Hey, look, don’t say DevOps. Now tell me what you want to get involved with.” And some people get confused and paralyzed with the thought process.

So you have a team saying they will like to have some automation around IP allocation, right? They want to use VMware, or something like Kubernetes. What they need from you is not a manual set of IPs that they can assign manually to each server, and you go set up static routes by MAC address. No one wants that. That process takes too long, and look how you’re tracking it with a spreadsheet. This is non-scalable. And look, you’ve been doing that for a decade. There’s got to be more important work. So how do you fix that? “Number one, does anyone on this team know how to code at all?” “Nope, we’re network folks. We don’t code.” That’s no problem. Nothing wrong with that? “Do you want to learn how to code?” If the answer is no, “Alright, now we’ve got an issue. Now I’m coaching. I’ve gotta go get some software development talent on the network team.” So then go within. Anyone that writes software want to learn networking? Come over.

Now look, we’re about to raise the bar for what it means to be a network admin. You’ve gotta learn at least one scripting language. That’s it; not everything. You ain’t gotta go learn Java, but you’ve gotta learn one scripting language. I’m gonna start with the basic, and I’m gonna give you years to learn, but you’ve got to meet me halfway, but be willing to learn. If that’s not you, then the team gonna have to change over time. We have to bring in some new people, we’re gonna have to let some people go, because they don’t want to evolve to what the team needs to execute.

And then the accountability is, what is the current way of giving out IPs? We put it in the spreadsheet, you open a JIRA ticket, and we then do these 28 steps. Nothing wrong with that. But that’s got to be 12 steps, in three months. It has to be 12. If it’s not 12, then we haven’t made any progress.

So in six months, it’s got to be 12. And in the six months after that, it needs to be four. And we’ve got to get it down to the bare minimum of our team’s capabilities. But what we can’t do is still be complaining two years from now that it’s still 35 steps. We can’t have that. And that is something that has to be done at the team level, individual development, I think that’s what our team needs are for, that’s what our managers are for… You can’t just do this by bringing in one VP and making them change the whole company culture. That doesn’t work. So we’ve got to make sure that each of our pockets get what they need, and we’re constantly giving people that feedback and then just holding them accountable. So I think it’s literally that nuanced, that detailed, that we’ve just got to make sure we have the right people, the right attitude, and they have to show progress.

Well said. I couldn’t have said it any better myself.

Yeah. My friend Kelsey, it’s been fun having you here, man. I’m glad to have you here so quickly after you announced your retirement, so we can go into the details… Congrats to you on that two-year journey to sharpen your axe, and chop down that retirement tree for you… And we obviously love having you back. So come back again when you’ve got more to talk about, more things to share. That’d be fun.

Yeah, man.

I will, 100%. I love this format. Thank you all for having me.

Bye, friends.

Bye, y’all.