Kaizen! Three wise men?

Well, Gerhard, did you create a slideshow for us?

I did. Do you want to see it?

I absolutely want to see it.

Alright. Let me screen share it.

This is you kaizening our Kaizen episodes with slideshows.

Of course. You have to have them. It just makes things so much more interesting. I haven’t posted the last one, but I will do this time.

Okay.

Let’s see. Screen, window, boom, boom. There you go. Kaizen 17.

Oh, man…

Geez, dude. It’s a nice font. What’s that font?

I’m not sure. Let me double-check. I think this is the default one that the template has.

I like the one.

San Francisco… It’s inter.

Inter.

I-N-T-E-R. Inter.

Interesting. Okay…

Inter is cool.

I’ve used that font.

IA Presenter. I’ve been loving their stuff. IA Writer… I’ve been using it for seven, eight years. So IA Presenter is –

This is IA Presenter. Okay.

Yeah. I love like how they write about it, and the whole idea behind it… So it’s nice and simple, and I can knock these out so quickly.

Love it. Love it. Well, take us on a ride, Gerhard. Your magic carpet ride of Kaizen.

Cool. Okay. Well, the big thing, or the main thing should be the main thing. So I was thinking we should start with the main thing. If Adam is ready for it, I’m ready for it.

Always be ready.

We’re ready for it, Gerhard.

Okay.

What’s the main thing?

Well, you tell us, Adam. I’m looking at you.

Oh, you’re talking about CPU.fm and the change we’re making.

Oh, yes.

This is good stuff. So there’s a very famous person… His name is Newton, at least.

Sir Isaac?

Isaac Newton. Sir Isaac Newton. And he said - to paraphrase him and to paraphrase a very awesome movie, “Humans, the only way they’ve found to move forward in life or to get somewhere is to leave something behind.” And so as part of that idea and mantra, we have decided to change things in 2025 to focus solely on making this podcast you’re listening to right now. The single best developer podcast experience. News on Monday, interviews on Wednesday, Fridays on the weekend, or on Friday, mostly… And that’s going to be some fun stuff. So that’s what we’re doing.

Some of our favorite shows are going away, transitioning, moving on, spinoffing, continuations… But that’s what we’re doing. What do you think? Was this a shock to you?

It makes sense. I was surprised when we talked about it. Honestly, I could see it coming. I mean, it just makes sense.

It resonates with how I like to approach things, and it just fits with that mentality. Double down on the thing that you enjoy doing the most, and the thing that, I think, makes you special.

Yeah.

Because this is what makes you special, I think.

On that note though, we’ve really enjoyed producing Go Time, JS Party, Shit It and Practical AI. We loved the people, the shows… Jerod will tell you, because he hasn’t spoken about it yet, at least in this podcast… We really deliberated over this for a while. I would say so long so that it’s probably at least a year or more of considering how to change to get to where we’re trying to go. And it was a struggle, because it’s not easy to retire something, move on from something, to give someone or an entire world of audience bad news… And that’s not something that you do quickly or lightly. You do it with intention, and precision… And even then, even when you do it precisely, and with precision and intentionality, it’s still hard to get it right. And so there’s no really good version… Some will be upset, and that’s how it goes, I guess. But our intention is to love well, and to love the hosts and panelists we work with for many years very well… And to ease this process of them either spinning off something to do their own thing, which - almost every show has some version of continuation. Actually, every one of them does.

JS Party has a new show called Dysfunctional.fm. Nick, KBall, Amy… Go Time has its own spinoff called FallThrough.FM. Kris, Ian, and more… Extended friends from our existing podcast family of people who listen. Ship It has its own spinoff called fafo.fm. Folk Around and Find Out. Cool. Love that. Justin and Autumn. And then Practical AI, they’re keeping their name, because Chris and Daniel were unique. We’ve been working with them for many years… Probably the longest-running show, and independent. They had no other panelists.

It was just those two, yeah.

[07:48] Just those two, for many years. And it’s just a labor of love to produce these shows. It’s a treadmill in the media world, and we’ve been doing this for 15 years… It’s not like we’ve been doing this for a day. We’ve got some reps under our belt. We’re swole, so to speak, in the podcast world… [laughs] It’s funny to say it out loud… But we’ve been doing this for a while. And I think just having a chance to sort of pause for a moment and think about - okay, to produce a really good podcast, that has a video-first production workflow…. To do that even for this show, that produces three shows a week - we have no more bandwidth to do it. We would only be able to do that if we scale the team, or if we just didn’t do it. And we haven’t done it for many years.

You go on YouTube and you have clips only, which is great… But people have been asking us for years, “Can I get the full show on YouTube, the full video show?” And I think there’s opportunity cost there, not doing it. We’ve missed out on audience growth, connection, more in-depth content etc. And the only way to get there really is to make a major change. That’s what we did.

But I think cpu.fm is a big vision, burgeoning new idea, that doesn’t have the full fruition yet, but it has a big vision. So I think what we’ll do to support these shows and to support this Changelog Podcast Universe - that’s why it’s called cpu.fm - I think it has big opportunity. And so I think if people continue to trust us, join cpu.fm, let us support them… But they produce their own shows. We’re no longer part of the media machine they are, where we’re helping them ship shows daily, weekly. That’s a tough one. It’s arduous.

Yeah, I think that the end result of this - there’s a very real possibility that this change, while it effectively takes away shows in the short term, I think it actually might result in more, better developer pods down the road… Because we had reached our capacity, and for many years we’ve turned down ideas and opportunities because we were just maxed out. I mean, how many times has a listener requested a podcast from us about Rust? Probably a hundred, maybe slightly less… But many, many, many times. And I hate that – I don’t hate it, but I have an internal anxiety about that request, because it’s like, I just knew I couldn’t do it. We couldn’t do it under our current structure.

Now, there was another option… Like, we could grow our business, we could grow our team, we could build out that way. And I think a large part of life is knowing what you want… And I’ve lived long enough to know that I didn’t want to do that. I don’t think Adam wanted to do that. And so that wasn’t an option for us, just because we just don’t want that to be our life.

And so the other option is either stay at capacity, five shows a week… Technically it’s like seven shows a week, because The Changelog is three episodes a week. And live that life, and make those shows; and we did that for a long time, and that was a totally legit route. Or double down on the main thing, let a few of the things that we love go, and see if their love returns to us tenfold. Now, that’s just the corny saying about if you love something, let it go, right? Encourage the people who have been making those shows with us to make same, or similar, or new shows that we could support them. But independently, as their own thing, so they could have full ownership. They could have equity. And we can all podcast together, and work together, and collaborate.

[11:40] And so I’m very excited about where it’s going to go. Obviously, in the short term, it’s on the bitter end of bittersweet, especially for me. I mean, speaking very personally, JS Party is like one of my favorite things. I’ve grown very fond of that podcast, and those people, and what we do together. And Go Time very similar. I’m not on Go Time on a regular basis. I do show up from time to time, but I’m a regular on JS Party. And so just emotionally, it’s just a very hard thing to stop.

I’m very excited that Nick and Kball and Amy are going to continue podcasting together… And then I have a standing offer to join their show whenever I want and hang out. Because I just love the shows that we made together, and the times that we spent. So it’s been a hard decision… It’s been a long time coming for us. Obviously, we don’t talk about our indecisions publicly.

We talk about our decisions. But we’ve been toiling over this change, like Adam said, probably for a year. We almost did it a year ago, honestly, but we didn’t. And now we’re doing it. Sometimes you’ve just got to pull the band-aid off, you know? And it’s hard, but it feels right… And I’m excited about January, because I think it’s going to breathe some new life into our show, into these other shows… And yeah, those are my initial thoughts on it.

Embracing change. If we could use the title again, I think we should.

Yeah…

Because it fits it really, really well. Change happens, whether we like it or not. This way we are in control. Gaining something, losing something… It’s just all part of the game. You have to go back to the roots, doing the things that you love. First of all, knowing what that thing is; it’s really hard, because the world is a busy, noisy place, and there’s always the imposter syndrome… We know how well that works. And we also know there’s always the grass is greener on the other side.

Yeah, the FOMO.

It’s not exactly. But the Joy of Missing Out, right? The JOMO.

There you go.

I really like that take.

The JOMO. [laughs]

The JOMO, exactly. This could be it. Another baby title idea. The Joy of Missing Out.

That’s the first time I’ve heard that.

Oh, really?

That’s brand new to me, right to this moment. Yeah.

I think I’ve heard it first from DHH…

Oh, really?

Yeah. The Joy of Missing Out, in the context of remote work.

Of course he would make that up.

I’m sure he did. I don’t know. The point is, it’s about leaving slack in the system, focusing on the things that you know you enjoy and you’re good at, sharpening that axe, and doing the best work that you can, leaving room to do that. And that’s really hard, and really challenging, but it’s worth it. Because at the end of it, you look back, 10 years from now, 20 years from now, and you realize how rich your experience was because you made the choice. It all starts with a choice.

And it’s not like you’re stopping everything, right? You’re finding another home, there’s a whole new twist, a new perspective… It’s the next evolution in the Changelog universe. And I like that.

Yeah. You know, just to laser focus on one specific thing that I’ve personally had angst with is that we’ve had a great opportunity to help many brands reach developers over the years. Like, obviously, we’re sponsored. Most podcasts are. Any podcast that’s sustainable or being sustained is usually sponsored in some way, shape or form. You find any podcast out there. The biggest ones out there. Even Joe Rogan; he’s sponsored. You know, we’ve had this ceiling of ability to help folks because we have limited shows. I really believe that Jerod and I are pretty good tastemakers, and I think the idea with CPU is pretty cool… And we want to help more of those brands reach more developers. And we had a ceiling. I would often tell folks - because I’m a big, big helper when I work with these different brands… And I know that I’m like “I can only help you this well with the shows I command under our belt.” And I think that Jerod and I are two people, we have limited bandwidth, we add folks onto our team as necessary over the years to support us in producing podcasts… But at some point we had a limit. And now we have so much more opportunity to help more brands reach more developers through CPU.

[15:53] So I think that’s the coolest thing for me. I think being able to expand that to 10 or 15 podcasts with an index, with a single subscribe point for many really awesome developer podcasts that join this, I would say somewhat of a movement in a way… World-class developer podcasts… That’s a cool thing, in my opinion. And we have this big vision that is literally at the spark of the moment at cpu.fm. And I have, and we have a big vision for it… And I think it’ll be good for us, good for, like Jerod said, the awesome shows that will come from it… But I think more importantly, helping developer brands reach developers is really, really hard. The ones who have a good story - they’re just so new. They need great awareness, but they’re just so brand new in that story… They have almost nowhere to easily go to execute, to get the word out of who they are, what they do. And that self-serving way to me is one cool way that I see a much more bigger opportunity, for them and for us, and for the shows that get supported from it.

One other point I’d like to make on this, and then I’d love to get into your work, Gerhard, is that we have built out this portfolio of shows that we love, and listeners love, and hosts love… And it was all good. There was no real struggle, or there’s no – of course, there’s the work of scheduling, and rescheduling, and sponsors, and this happens, and that happens, and we’ve got to get the thing out… That’s all just work of producing podcasts. But I got to a point where we’re doing these seven shows a week - and this is relevant to Kaizen - that I didn’t have any bandwidth to actually experiment. And I was writing – I almost said this the other day, Adam… We had Chris Coyier and Dave Rupert on the show last week, on Friends last week…

Yeah.

And we were talking about my desire to stay in the trenches, and be actively developing. I just have not been writing enough software lately. I want to build stuff more. And I just didn’t have any room to breathe… And so I literally would just do Kaizen-driven development, which is every two and a half months… Like, “Please don’t look at my commit messages”, you probably already did, “between the last Kaizen and now.” It’s not very much. I just don’t have the bandwidth. And that’s not good for our show, that’s not good for me personally, that’s not what I want to be doing… I want to build stuff more. And I’m so excited as we do laser-focus on the Changelog, as we do take these productions off of our weekly calendar, even though the shows will exist in spirit as other people’s productions… I’ve got room to breathe, I’ve got room to code… I can block an entire day and just be like “I’m going to build something today.” And that, I think, is a huge benefit to this particular change.

Which brings us to the work you’ve been doing, because I haven’t done jack squat, Gerhard. Hopefully you’ve been kaizening, because we’ve just been producing podcasts.

Oh, yeah.

We don’t have a show if not.

So I really get that at a very deep level, because that was at the root of The Ship It Show, me putting it on hold… That was exactly it. Room to breathe, room to experiment, room to do other things, room to do things differently. That’s how it started. And it’s coming up to a year, and it still feels right.

And you’re right, it takes a certain amount of adulthood and strength of character to know that that’s what’s happening. And I’m really glad that you’re in this point. It’s an amazing point, a very important one; essential for what’s to come next. It’s a catalyst. So it’s the beginning of something amazing. And I’m so excited to be a friend of this amazing journey. Very, very excited.

You are a friend.

You’re part of it. You’ve been a part of it for a long time…

Thank you very much.

…and we’re excited to keep you as a part of this as we grow and change.

We didn’t highlight that much though in our post… There’s nothing changing about Kaizen.

Oh, no.

[20:05] Kaizen is now embedded into the Changelog podcast itself. Just to be super, super-clear, it was implicit. It will be more explicit here. There’s nothing changing about Kaizen.

Right.

In fact, I think – it won’t get more frequent. I think it might get better, because I think, Jerod, to your point, I think you and I might have more time to do more development, to make us less like “Gerhard, what did you make?” so that we can have a podcast and some iteration and some change.

Oh, yeah.

So to everyone listening, I want you to know that this conversation has been thought out weeks in advance. You’re at the baseline… It only gets better from here.

Yeah.

So stick with us. This was the big announcement. This was the important stuff. And now we’re going to have some fun. Okay?

[laughs] Alright…

Even more fun.

And trust me, the last thing – I’ve been trying to keep it a secret for a while…

Uh-oh…

And I think I’ve succeeded. And it’s going to be amazing.

Oh, man…

So sit down. You’ll want to sit down for this one.

I’m more anxious now than I was.

Is it early Christmas?

It is.

Early Christmas. I love it.

It is, actually. Yes. So things are coming together.

Gerhard has some presents for us, I think. Okay, let’s –

One.

Oh, just one. [laughter]

But it’s great, I hope.

Gerhard, you know the philosophy. Have two of everything. Come on, man.

Well, there is another one… But anyway. [laughter] See, I can’t hide this from you. You know me too well.

Oh, gosh. He’s too giddy.

Can we just skip to the end? Can we just not do this middle part?

This is why we need to have a video podcast, too. Because you listening to audio all these years, have not seen the extreme laughter we’ve had on this podcast in particular.

That’s true.

As a video. You may have seen it in clips, but you haven’t seen the full length –

And you can hear Gerhard’s joy in his voice, but you can see it in his face even more.

Gosh… Yes.

Oh, yeah.

Alright. Take us on the ride, Gerhard. I’m so excited for whatever it is.

By the way, this little part was for Jason, the editor.

Oh.

I met him for the first time when we recorded the last Ship It episode.

Oh, nice.

And that was a very nice – so this laughter was for you, Jason.

Okay. There you go.

So this bit you may edit, but I knew this was going to be fun, and I thought about you as we were going through the show.

Oh, nice.

So hopefully it won’t be just my laughter, it’ll be everyone’s laughter, because we’ll go a bit crazy. But in a good, tasteful way.

[laughs] This is now telling Jason what to do… “Make sure it’s good and tasteful, Jason.”

Exactly. No wet wipes, nothing like that. [laughs] Okay, so let’s start.

Alright.

So in the last Kaizen, you were thinking, Jerod, of getting a brand new Mac…

That’s true.

…to try out the newly introduced Just Contribute.

That’s true.

And you were saying that you’ve been waiting for a reason to upgrade.

Right.

It was Black Friday, Christmas is coming…

True.

…and I hear that the M4 is all the rage these days.

I’ve heard so much good.

It’s so good, yeah. I’ve resisted hardcore, but yes. Continue.

I almost went out yesterday to the applestore.com, whatever the website is, and priced one out… My trepidation is we really maxed out these Macs that we’re currently using. Like, my current MacBook Pro, which is a 2021 M1 Max…

M1.

Yeah, it was the first M, but it was the maxed CPU… First of all, it’s still very good. And so that makes it hard to buy something brand new. It is the 2021, but it’s 64 gigs of RAM, it’s got a massive multi-terabyte hard drive… It was basically like go to the configurator and hit the max on everything, besides the size. It is a 14-inch, not the 16-inch. So aside from the screen size, it’s just maxed out. And so I hate to go buy one new, that has less specs than this one… But when I go max out the new one again, I’m like “Can I justify this? Because my one’s working pretty well…”

Or I could go downstream and like “Do I really need all that storage? Do I really need all the RAM?” And so that’s where I just like close tab and move on for a little while. So no, I do not have a new Mac… But gosh, I want one.

[24:09] Did you try to just contribute? Because that was the point. [laughter]

I’m already a contributor, Gerhard, so I didn’t really have… I’ve used just, but I have not tried just contribute. So no, I failed you in that way.

Adam?

No, sir. Sorry.

Oh, man… [laughter] Now, in all honesty, I know that a few listeners have tried… I forget who exactly it was. There was someone that mentioned that a lot of work has gone into it. Do you remember that, Jerod? It was in one of the comments, maybe…

Mm-hm.

I forget his name. That’s all I know. It was – I want to say it was Adam, but no, it wasn’t Adam. It was someone else. Anyway, it was somewhere… I’m not going to look for it now, but it’s there. People can try it, and we’ll be building on top of that.

Also in the last Kaizen - and by the way, there’s going to be video. This will work well. I was mentioning that I’ll be talking at Taloscon about how I took my home lab into production… And so that happened.

Nice.

Something special about that was that it was a recorded talk by myself. I did the editing. I had multiple cameras, and it’s out on YouTube. So you can check it out.

Very cool.

The one thing that was one of my favorite moments about this talk is me showing off the actual home lab. So the home lab was a LattePanda Sigma, and the size, for comparison - it’s exactly as big as an iPhone Pro Max.

Okay.

So you have an iPhone-sized home lab, which is insanely powerful. You can run Kubernetes on it, 16 cores, DDR5, multi-terabyte NVMe drives, and it can serve 300 billion requests per minute. Sorry, per month. Sorry. That would be crazy. No. 300 billion…

I was like, “Wait a second…” [laughter]

No, no, no. Per month. Sorry. Requests per month.

Okay.

So many, many billions requests per month. Anyway, we may link to the talk for you to go and check out.

Oh, absolutely.

So the one thing that was really interesting about – or that’s really interesting about this specific device is that it has an Intel CPU. And I know that Intel now has not been doing that great this year, but they do have video sync, which means they can do video transcoding really, really well. So they have GPUs that can do video transcoding. This one has an Intel Iris Xe Graphics, which means they can do video transcoding amazingly well.

Okay…

So let’s take the nerdness level plus one. Okay?

Always.

And for this, both of you, you’ll need to take out your browsers, and you will need to go to jellyfin.makeitwork.tv.

Are you going to share with us your home theater setup, or what?

Um, no…

[laughs] Okay. I’m there.

You’re there. So enter your GitHub username, and for the password - by the way, if you’re listening, this will change. The password is kaizen17.

I’m in.

Okay. What do you see? Tell us.

My Media… I’m in the Jellyfin homepage. I see Drafts, and recently added in drafts I see some vertical video thumbnails and a horizontal video thumbnail.

Excellent. Click on one… Click on the drafts.

Okay.

And pick the 4K one, 2160.

Full 2160p.

Mm-hm.

I see a loading spinner…

Great. It’s running on my shelf, by the way. There is a CDN, but it’s running on my shelf. The home lab. And the video is being served from that home lab instance. So this is home lab taken further.

What is this video? This video is the last conversation that we had with James and Matt about building out the Pipe Dream CDN. This was August.

Oh.

[28:04] It took me a while to edit the video… Did it load, by the way?

Yes, it loaded. I just had to pause it so I could listen to you talk.

It’s running smoothly, too.

Excellent. So this video is running off that LattePanda Sigma. It’s being fronted by a CDN.

Mm-hm.

So it’s turtles. Turtles and turtles.

Mm-hm.

And this is the content which I always imagined I would produce. Content that has the conversation part, and content that then we do screen-sharing and we go deep. The content that you’re looking at is about an hour, the whole thing, the whole hour, and it was edited down for about three hours. That was a long conversation.

Yeah.

And it’s still a draft, so it hasn’t been published yet, but it’s coming. So taking the home lab further, doing the CDN; in a way, it’s the adventure. It’s a CDN adventure. I think that’s what’s happening. And kaizening our home labs, and kaizening the devices that we all love… Whether it’s an M4 or an Intel-based home lab.

So you’re part of something special, just like cpu.fm is special, and it’s coming… It’s not there yet, but there will be a little more things coming, maybe just in time for Christmas. A series of videos that are – I’m thinking of them as movies for infrastructure nerds. And makeitwork.tv will be the place for this when it’s done. There’s also makeitwork.fm, but makeitwork.fm is just for the conversations, just like the audio part. So that’s how this works.

Nice.

Love it.

So Changelog is very embedded in this, because a lot of the experimentation that I do happens in the context of Changelog. And then on top of that, take it further. Make the time to create the content that uses that… But obviously, there’s so much more that happens. The editing - you have no idea how much it takes me to do the actual editing. That is my biggest issue. So I have an appreciation for video, 4K video done well, and sound and everything. For it to sound right, to look right, the color grading… It just takes a while. DaVinci Resolve - I think I’ve been using that app more than my terminal in the last, I don’t know, six months. I’ve been learning it… It’s amazing.

But anyway, back to Kaizen. Back to the Kaizen 17. So we talked about the cpu.fm, we talked about the home lab, and we finally did it. The thing that we talked about for a while, Jerod. Enable team members to replace Changelog dev with a prod dbdump. That’s pull request 533. Did you try that?

Yes, sir.

Tell us about it.

It works.

Great!

That’s what I care about… It was fast, it was seamless… It just worked. I think I did have to configure something the first time. I think you were making some changes to nvrc files, or there’s just been some environment variable things that I had to do the first time… But I can tell how you – the last time around, by using this just… Is it a toolkit, is it a library?

I would call it a CLI utility. It’s like a make… It just improves on make, basically.

Right.

If you’ve used make, that’s what it is. A command runner, some would say…

Right. So we’ll just call it this CLI tool that you made the change easy, and now you’re making the easy change… Callback to the previous episode, callback to, of course, Kent Beck’s “First make the change easy. Warning, this may be hard. Then make the easy change.” Classic quote from Kent Beck. Because this was like a pretty easy change for you, it looked like, in terms of this particular one. Now, you had some Neon things to do, but you can tell us about the details of how it worked… The PR itself seemed pretty straightforward and small, and then it worked, so that was my experience.

[32:06] Very nice. I’m very glad that you experienced it that way. I’m curious, when Adam tries it out, how well this works for him.

The idea is that we wrap a bunch of commands that you would need to run locally. For example, installing the Neon CLI, so that you can control Neon. By the way, there’s two. You can do it via npm, or you can go and download the binary… The binary – the version, it’s a slightly different one; so there’s some inconsistencies in the CLI binary from Neon. But this basically handles all of that.

And what I thought would be easy – I thought this change would be easy, but actually there were quite a few rabbit holes that I had to go down on. One, for example, was how to download it in a format which is compressed, because you don’t want to download many gigabytes locally. So how do you compress it automatically? And then how do you handle extensions that exist on Neon? They’re installed on Neon, but you don’t have locally in your Postgres database. So there was that as well. So I had to uninstall an extension which was giving us metrics that you can do via SQL queries.

So there were a few things like that that I had to go through. But still, what it means is that this command, just restore devdb from prod - that’s exactly how we call it, and you can look at the pull request. There’s even a video which just shows how it works. It wraps all that complexity. All that know-how. You have to run this command, and you have to pass this value… And the other thing is the credentials - they’re pulled just in time from the 1Password vault. You don’t have to remember them. And it orchestrates all of that.

So the integration - I was really happy with it, how it worked… And it feels like an easy thing to wrap your head around. There’s nothing complex about what’s – obviously, there are a couple of niggles to sort out, but you don’t see them when you use it. And you can even see the commands before they run, you can copy-paste the commands… So everything runs locally. And I think this was the hard part before, because when we were using Dagger for this, that was running in containers, but you don’t use containers… So we had to make that big change so that you would have this nice local experience. Everything runs – again, it doesn’t need Docker, it doesn’t need anything like that. It doesn’t need Dagger. It’s just commands. Installing binaries, things like that. So I’m glad that you tried it. I’m glad that it’s working. Do you see yourself using this?

Yes.

Immediately yes.

It’s just so straightforward. I mean, all the things that you just said right now are things that I love. And so I’m way more likely to pick up this simple tool, especially following your example, than I was – honestly, even with the old makefiles, which because of your expertise in makefiles, I was perpetually lost… I’ve looked at the just files, and they’re just easier for me to grok. The fact that there’s no containers, there’s no Dagger, there’s no things that I generally put in the black box of Gerhard land… For me, as a regular app developer, it’s more approachable.

And so I script stuff all the time… This is basically taking a script of mine that I would do and just have on my local machine to do all these steps, and it’s formalizing it into a shared repository… I’m way more likely to follow that lead and create additional just commands. And now I have time, too. So I’m totally into this, Gerhard.

Very nice. Oh, I’m so happy. I’m so happy. That’s a huge score.

Break: [35:39]

The next one was enabling team members. The next pull request, pull request 534, was basically building on top of this, and it just enables team members to run dev with a Neon DB branch.

So this was a rework of what we had before. That’s why it removed more lines than it added, but basically built on top of the same just approach. Did you try this command, Jerod?

No.

Okay. Will you ever have an interest to try this command?

So this creates a new branch on Neon.

Not just that, it also configures your app. It starts your app with that Neon branch. So there’s no more manual – it’s like one command, and it will run everything. It will create the branch, and then it will configure your app to use the branch. There’s no more manual digging around. And you can believe this.

Yeah, potentially. I do like to develop against production as a branch. I prefer to have it pulled down locally.

Alright.

So I think I would probably opt for the other one that you just created, which is why this one wasn’t as exciting to me, and I didn’t even try it.

I see.

But would this also do syncing between those two branches? Because that’s my bugaboo, is you do a branch off of prod. Then you develop against the branch. And then you want fresh data. And so I already have a command that gets me the fresh data locally. But if I didn’t, then I’d go to the Neon deal, and I’d go find the place in the UI where I hit Sync to Main, or whatever… And I don’t like that step. So if this could do that – I mean, maybe it would be a second command. I’m sure it’s available via the CLI somehow.

Yeah. I haven’t looked into that. It does make sense. It does make sense to add it, by the way. And even if the CLI doesn’t support it, maybe there’s a curl request that we can do to the API to get this synced. But that makes sense.

Yeah. Because that’s really what I want, is I want the freshens every time… And so it would be nice to have something like this.

Right. So to get the freshen once, which is a great idea, by the way - the first command will take longer, because it has to pull down the entire database. And that takes a while. And then it has to load the entire database. It basically removes what you have locally, and it just re-imports everything.

Oh, totally.

So that can take up to five minutes, depending on internet connection… A bunch of things.

Yeah, I experienced that. And so this would be faster.

This would be faster, yes.

So generally, when it’s time for me to develop, I issue that command, I go get a fresh cup of coffee, I come back and I’m ready to rock.

Got it.

So if you’re going to do it once a week maybe, it doesn’t bug me to have the five minutes. If I was doing more experimenting and changing and stuff, I think having these immediate branches, without any sort – I would still have a little trepidation that maybe I’m pointing it at the wrong thing… That’s why the command now handles all of that, so that you don’t have to worry “Did I set the correct environment variables? Is everything correct?” And because everything happens inside of the command, once it gets to a point – we had, for example, Adam’s case, where sometimes it fails, and then you don’t know “What state am I in?” This command tends to be self-contained. And what that means is that once it gets to a certain point, you’re sure it will continue, and it will finish, and it will do the right thing, and it’s all embedded in the actual command.

Now, this command, when you’re connected to a remote database, it can be slightly slower. So for me, for example, I preferred downloading all the data locally and having all of that locally, because it felt snappier. And even though the query planner - we warm up the query planner, so that things are cached, we do a couple of things like that, it still feels slower. It still is slower, because it has to do all those round trips, and they’re all remote. They’re all remote calls to this remote database.

[42:12] So I think the choice is between paying the penalty once, five minutes, three minutes, depending on your internet connection to load all the data, and then you know you have the latest, or pay a little bit of penalty every time you load the page, because it may take, I don’t know, a second slower sometimes, depending on how many queries you’re running. So both options are there. Knowing Adam, I think he would prefer the second option, so that he’s connected against a remote database, and I think you would prefer the first option. So we have a mix of both.

I think that’s probably accurate.

Yeah. But we’ll make sure that this works for Adam as well. Not in this context, but as a follow-up, for sure. Cool.

Well, the next thing which improved for us - and this was great to see - was the all-in Zulip approach. Zulip - how do you pronounce it? Zulip?

Zulip.

Pretty much. That’s how we pronounce it.

Great.

Adam pronounces it Zuli.

Zuli? Like Hooli?

That’s right.

He thinks they should drop the p. Like Hooli, exactly.

See? Kindred spirits.

That’s Adam’s idea. Drop the P. Call it Zuli.

That’s right.

So I think at this point, everyone basically migrated from our Slack to Zuli. Or Zulip.

Yeah.

Everyone’s there, the conversations are there… I won’t call it Zuli. Zulip. It’s just a joke. So everyone migrated to Zulip… How does it work?

Amazing. It’s awesome. I think it’s easier to track and follow better than Slack. When I go back to Slack now, I feel like I’m in like some sort of archaic way of communicating, which is just like, just throw it at the wall, and if you see it, cool. You can’t compartmentalize conversations… Threading is obviously there, but it’s not the same as Zulip. Threaded conversations for teams is what their mantra is, basically.

The one key thing I think that makes Zulip different in terms of how you interact with it as a user to communicate is that everything is based on a topic. So if you’re starting something new, you’re beginning a new topic, which can be to some degree daunting, because you think “Well, if I want to say something, I must have a place to say it.” And if there’s no place to say it, then you feel like “Oh, I’ve got to create it. So is it that important?” Maybe that’s what stops you from communicating.

I don’t fully disagree with that sentiment. I kind of wish there was a merger of the worlds, where you have like a single place in Zulip that is non-threaded, where it’s just “This is where the everything goes.” But then I can kind of feel the angst against that, because if your principle is communication must correspond with a topic, and that’s your way, I can understand why they’ve sort of harkened into their ways to not do that.

But as a user, I kind of want the Slack world in a way where it’s just like everything goes where it can go, like a main channel, for example, and then also still have the topic world. But topic-based, inside of our Zulip we have the different podcasts, their episodes… You can comment on a Kaizen 17, for example… It’s really compartmentalized, which I like a lot. And those threads are long-lived. Like, we’ve got this WordPress Drama thread that was not started by me, or Jerod, it was started by the community, and it’s still being active today.

Whereas in Slack, that conversation would have just died, and got reborn, and the context of prior conversations isn’t there. So you have this community, keep it together, long-run conversations that can be potentially months, maybe even years… And that’s just not equally as possible in Slack. You can do it, it’s just not present so easily in the UI. That’s what I love about Zulip.

[45:59] Yeah. It did take me a while to get used to the idea that everything is a thread…

Yeah.

But then once you make that switch, you realize, “Actually, this allows me to be more focused.” I can just pick a thread and I’m there. That’s the context. I can see everything that was discussed in that thread. So that’s really cool.

Also, having a thread per episode - I think that’s a great idea. I cannot remember how many times there was a comment on an episode on the Changelog website which I just couldn’t find. I couldn’t find “How do I get to the comments on an episode?” Now - so much easier.

Yeah, I think that’s been really nice, especially for podcasts where they are consumed, not just asynchronously, but like massively asynchronously, where there’s people who are living off of the fire hose, and they listen to the episode when you drop it, and there’s people who are like three months behind perpetually… And then there’s people who are like back catalog dwellers, where they’re like listening back through the catalog… And they may listen to it years later. Well, there was never a place where all those people could easily find “Here’s where that discussion is.”

And so the thing that I’ve heard the most, and which I’ve enjoyed, is people’s ability to hop back into an older episode and either strike that conversation back up, or even just read it, what people had to say, in the time between it shipping and you listening to it. So that’s really cool.

I do have, after living – anytime you live somewhere for a while, you see all the warts. I’m starting to have a little bit of the - not buyer’s remorse, but just like Zulip wart finding adventures, if I might call it that… The mobile app leaves a lot to be desired. I know they are rewriting it right now in Flutter, and so they’re working on that. There’s all kinds of little UI things that just bother me about Zulip… And the URLs - I’m a URL guy. I can’t believe some of the URLs these folks put together. Because it’s basically an SPA, and everything is like – go read the URLs. They’re just not pretty. And when you’re trying to deeplink in this stuff… I don’t know. That matters to me. It offends my sensibilities; when you’re deeplinking into something and you’re like “Look at this URL” I’ve got to go give somebody. Stuff like that. Just minor nitpicks.

But it’s definitely better. It’s better than Slack in many ways. And while we are all in pretty much on using Zulip, we haven’t done any sort of finalization in terms of like the blog post I was going to write… I probably still will… We haven’t closed our Slack… Probably can’t at the moment. There’s still conversations happening there, mostly in private team chats. Some with collaborators who we don’t want to just ask them to move to Zulip, because it’s just kind of odd, and strange to be like “By the way, from now on if you want to communicate with us, switch to Zulip.”

“You must come over here.” Yes.

So we are living in a little bit of a –

Blue/green? I’d call it long blue/green…

Yeah, it’s a blue/green deployment. Yeah, exactly. And there’s lots of green on this side of the grass, but it’s not entirely there yet.

Yeah. Do you see us shutting down Slack at some point?

I sure hope so.

We certainly can, especially for just like the public discussions of like – Slack is cut off at this point in terms of joining. The website is all – you join the community, you get invited to Zulip… You can’t get a Slack invite unless you go inside a Slack and invite somebody. And so it’s like essentially cut off from the world. And there’s no conversations happening there in the public at all. Every once in a while somebody will say something; mostly they’re spammers.

Our new episodes are still posting there. I haven’t quite made that change yet. I figured we’d do some sort of more big announcement first, and like encourage people who are still in Slack one last time to come over to Zulip… But we do have a lot of team chats and private chats that are ongoing and used, that we haven’t quite gotten cut over to. And some of them, like I said, are with folks from partner podcasts and stuff. I don’t know. We haven’t decided if we’re going to actually like close the Slack. But we would like to.

[50:06] Yes. It would be nice to just not have to have it as an option so that you miss conversations or have to track one more place. I think the sad part about our Slack is that it is a free Slack, and so that means after the rolling time schedule they have, conversations just go away. And that’s not cool. I just really hope that Slack goes away for us. I love Slack. My real hope, I suppose, maybe if I rewinded prior to Zulip – I have two hopes. I would love Slack to support communities better, and support non-enterprise, not so that they can… I think there’s just so many people who have Slack embedded into their world. It’s not going away. I’m part of other Slack. So Slack isn’t going to go away for me. It might go away for Changelog, but it’s not going to go away for me as an individual human being.

Yeah. Same. I would love it if Slack supported communities better. And that way, places like Changelog could have had some sort of relationship that didn’t have to be free. We don’t want to be freeloaders to Slack. We would love to pay, but we have 7,000 people in main at one point… Not all of them active, but if we had to pay for everybody in there… OMG. We would go broke, right? We can get it sponsored maybe, but then it’s like “Well, does that really add value to a sponsor, to support that Slack channel?” Maybe. I mean, there might be ways we could do it, but it would be kind of maybe icky to enable that.

So I just would love Slack to revisit the idea of the ubiquity and embeddedness that they have with developers, and the community aspect that just doesn’t get to foster without paying large sums of money. Find a different business model that supports those folks; I think you’d change some things.

Second, I think that Zulip has so much potential. And I say that not in a negative way in the fact they’re not reaching it, but they have so much potential to reach lots of people. I’ve had to say to people, “We don’t use Slack anymore. We use Zulip”, and they’re like “What is that?” That is an absolute shame, that’s what that is… Because Zulip is so cool. It’s open source, there’s an amazing team behind it… They have an iOS app, an Android app, a web app… But for the faults that Jerod mentioned, I think they’re held back by some beliefs they have that have just held them back. But then again, their held back may be perception for me and not them. Their held back is like “No, we’re doing exactly what we want to do. We’re reaching communities…” And they’re supporting us. We fit perfectly in their world in terms of how we as a community use Zulip. I just think they could more thoroughly compete with Slack if they changed a couple of things.

I don’t know how to say that really in this podcast, but there’s opportunity there. There’s hidden potential they can seize. And I hope they do it. My two hopes are Slack support communities better, and Zulip to reach their full potential.

Yeah. By the way, just as you solved your problem with 1Password…

Nice…!

I solved my problem with “Who posted that just contribute worked well.”

Oh, gosh…

That was Matt Johnson. It was in Zulip. It was “Kaizen, just do it.” That was the thread. I’ve found it. And Matt Johnson wrote, “Just uninstall and just install worked pretty cool.” I then ran just contribute and - wow. Yeah, that does a lot. It worked though. So just contribute worked for Matt Johnson, September 2024. So now I know how to find it. It was in Zulip all along. Nice.

Cool. Okay. Well, one thing which I also liked is how when you post an episode, there is a markdown for all the chapters. And that was Jerod’s magic. So even though he didn’t do a lot - which I think he did, by the way; he’s just being modest - this was the one thing that he did. And I love that improvement.

[53:57] Thank you. This was one of these moments where you’re just like “It’s markdown. That’s easy. Markdown has tables.

I wonder if they support tables. Yes, Zulip supports markdown tables. That’s pretty easy. Why not add chapters?” And the cool thing about it was I had a little bit of concern that it would be too long if you do that. However, Zulip will take long messages and collapse them by default when you first look at the thread… And so if you don’t want to look at the chapters, it doesn’t bug you at all, if you want to get straight to the other people’s conversation.

So that was nice to see… And yeah, one of the cool things about having our database, our admin, our backend as the central source of truth for our chapters, versus in the mp3 file or in the RSS feed, is that we can basically emit those in different places that make sense. And this seems like a place that really makes sense, because if you’re just hanging out in Zulip and you’re not sure if you want to listen to an episode because it’s not really your bag of tricks, but maybe we talked about something you’re interested in somewhere in the middle - yeah, you could just scan the chapters real quick and see if anything catches your eye.

I thought about linking up each chapter directly to the start time over on the website, so you can actually click on them and listen from there… And I didn’t quite get that far. I’m not sure if that’s – would you love that?

I would love that. Honestly. Once I’ve seen these chapters, that was the point like “Wow.” This all of a sudden made Zulip 10 times more useful for me than Slack ever was.

Nice.

Because now I can see the episodes, I can comment on the episodes - same interface - and I can see the topics. These chapters. Super-useful. The one thing which I was missing is “Where’s the link to click on the chapters?” So this would be so amazing.

That was like on my to do next list, because it’s just easy for – I’ve already done most of the hard work. It’s just a matter of making those links clickable. And so maybe what I needed was a little encouragement. I’m happy to add that as an easy Kaizen.

I would love that.

I would concur and plus one that, because that would make me click a chapter start time, easily, because it would be clickable, for one… And I want to, now.

“I want to, but I can’t.”

And I cannot do that.

Yeah.

I will say that when we go to a video-first world and we’re bifurcated temporarily while we have interviews – this is sort of somewhat in the weeds. We may roll out one show as video first, and the next second; it might make that integration slightly more harder… Because it might need to link to YouTube, for example, to a timestamp, versus to our site as a timestamp.

But you still need to write the timestamps in YouTube, so you have to generate the chapters ahead of time. [unintelligible 00:56:30.27] They convert them automatically.

Yeah.

Hence the workflow challenges we’ve talked about in the new era for the Changelog Podcast Universe is this –

Right. The question becomes, “Do we have one artifact that is identical in both platforms? Or do you have a slightly different show on YouTube than you do in the audio?” Because of reasons. And so we’re still in the throes of figuring all that out. But I think once we figure all that out, adjusting the stuff for the chapters won’t be too hard, because we already had done all the hard work.

I did try both approaches, and I do find that YouTube is a different medium, and I think to get them to get the best out of it you have to cater to the audience that’s on YouTube… And that audience prefers the content to be a bit punchier. A bit like more to the point. Like, don’t lose the audience, because they have certain expectations. And we’re not even talking shorts. Shorts is completely different.

Sure.

So this is just when you put a video on YouTube - yes. Great. But I think the transitions - they need to be a bit quicker than you would have in a conversation.

Yeah. I mean, even just the way that we do a pre-roll ad in our podcast - we come up with like the intro, the voiceover, and then a pre-roll ad. And it’s like, is a YouTube video with a pre-roll ad at the beginning going to get action over there? I don’t think the people on YouTube want that. Do we just cut straight to the interview? So there’s a lot of those kind of – and then all of a sudden now you have basically two shows you’re doing for the price of one. And so, yeah, we’re still figuring all that out.

[57:59] Yeah. Two cuts. It’s like a director trying to do “Oh, here’s the extended cut, and the theatrical, and the producer’s version of it, all in the same release.” It’s like, no, that doesn’t happen. The extended cut always comes later. It’s usually like “Oh, that was much better.” Or in the case of Ridley Scott, that was much worse, because he likes his original cuts better first…

Wow. [laughs] Sometimes the extended cuts are much worse, because it’s the director being like entirely up their own butt with their love of their story. And it’s like “No, the cut was really good, actually.” Your editors are excellent. Other times the extended cut’s awesome. So it is hit or miss, but…

For sure. For sure.

I am on the 10th conversation, editing the 10th conversation right now. The one that you saw in drafts. And the conclusion which I reached is that audio has to be separate. You focus on a listener, not a watcher. Then you focus on the YouTube audience, that they want something quick, they want something free, they want something entertaining. And then you have to focus on cater to the real nerd. They want to see the whole thing, they want to see like a great cut, but they want to immerse themselves in the story. That is not your YouTuber. I don’t think that’s your listener, because especially if you have some video which is like screen-sharing, that is different. And that’s more engaging… That just basically ups the ante and ups the story and takes it to a whole new level.

So those three audiences are very different. And I end up with three types of content. Same conversation, three types of content, catering to the audience. And again, we’re not even talking shorts, or Instagram, or TikTok, which just requires another approach.

Speaking of improvements, there’s one more that I noticed, and I was so happy to see Jerod do that. Deploy notifications in Zulip. That was so cool.

Yay! I forgot I did that.

How was it building that? Like, how was that building it?

I don’t even remember, to be honest. In fact, when I saw those code deploys, I thought “Did Gerhard do that, or did I do that?” That’s how much it’s been a bit of a whirlwind around here.

Wow.

Easy, I guess… I do remember once you have basically the Zulip API – like, their stuff is so simple. That’s one of the things I like about them. There’s no OAuth, there’s no craziness… It’s just like “Look, go ahead and generate a token, and then throw that token in a header. And all the requests that you have that token in the header, we’re going to let you do what you want to do.” Now, there are some finegrain controls beyond that, but they just start from the basic place. And so that made me getting Zulip abilities inside our app, like, 30 lines of code, for the module that changelog.zulip module, which invites people, and posts stuff… And once you can post stuff, then you’re just basically – you’re halfway there. Now, how does this work? Honestly, I don’t recall. Is this going through GitHub Actions?

It is. Yeah. So from GitHub Actions –

Okay, so this probably isn’t even my code doing this. It’s probably just a GitHub Action I installed.

Yeah. I think –

Alright, how’d I do it, Gerhard? How’d I do it?

[laughs] Let me open it up, because it’s been a while since I looked at that. And there’s no pull request, there’s code, so I have to go look at that.

That’s me. That’s me. No pull requests. Yes.

And it’s spread across a couple of actually commits, so…

That’s also me. Yeah. This is how I roll. This is how I roll.

That’s how you roll.

It is. I can’t not be myself, you know?

PRs actually align with the idea of topics in Zulip. To get code into a repo, your way is PR-driven, topic-driven.

Yeah. The reason why we do it that way, or the reason why I prefer to do it that way is because it provides an interface to a conversation. So if people want to check it out, if people want to – for example, they just want to see the video of how the thing works. Well, you go there. It also allows me to capture a lot more content. Like, how do you put a video in a commit message? I mean, you could put the URL, but then where do you host the video? With a pull request, you can put all this context there to capture why I did it. And it’s useful for me as well.

[01:01:57.28] So all the previous pull requests that we mentioned - the 533, the 534 - there is a section which captures how does that thing work. And that’s me making sure that I run it myself on a different machine, to make sure that the thing works, and how does it work. And sometimes, more often than not, I find issues and I fix them. It’s just a more, I don’t know, wholesome way of approaching something, and I enjoy it. I think it’s a more professional way as well. However, I still do commits. Commits have their place. So I’m not dissing them, all I’m saying is like different approaches and different contexts, and different ways of sharing that information.

I think your way is definitely better. I think the difference between you and I is you want to have a conversation about this stuff, and I just want to get stuff done. And so I just do stuff, and then I’m like “Well, Gerhard, I’ll figure out how to talk about it on Kaizen.”

Yeah, that’s it. Yeah.

So I offshore my conversations… Which reminds me, “How did I do this? How did I accomplish this?” Did you find it? I must’ve just installed a GitHub Action.

You do use a GitHub action. I’m looking at – in our Changelog repository, GitHub workflows, Dagger on namespace, that’s the one that I’m looking at… And it’s line right now at 68. “Announce deploy in Kaizen Zulip.” And you’re using the Zulip GitHub Action, Zulip send message v1. You have an API key, you have the email, organization URL, two type topic content. But I think the content was the interesting one, because you had to trim the git SHA, you had to use like a short SHA… I’ve seen a couple of commits where you were trying to fix that integr–

I was tweaking it.

Exactly. Yeah.

Alright. So yeah, it’s just basically you use an existing GitHub Action, and you tweak it to your liking.

Yeah.

And so it’s even easier than writing your own API client, which I did for all the other integrations. But yeah, this one was so easy I forgot.

Yeah. Well, it looks good, it works well. I was very happy to see this. And it’s in the Kaizen channel. It’s exactly where it needs to be.

Heck yeah…

Code deploys… It’s all there.

I’m kaizening.

So you can go and check it out. That’s very nice. So the one thing which I had to do, I guess a follow-up pull request, 536. We have two of everything. We are running the primary deploys through namespace, they run Dagger for us, and they run the actual, the CI and CD parts as well. But we also run GitHub as a fallback. So what we wanted to do is announce deploys in Zulip as well on the GitHub runners. So all I did was just basically copy what you’ve done, Jerod, and put it there. The fact that I copied it makes me think that this should be refactored. This should be simplified in some way. There’s quite a lot of configuration. So it’s something for my list. However, we discussed in the last Kaizen the namespace runners and how much will they cost us per month.

Right.

So now the numbers are in, and we get to pick. So option a, it costs 50 40 cents. I’m typing this up, 40 cents. So $0.4. That’s option one option. No, no, no… I went too far.

You did.

Hang on. I’m doing this as a live edit, and I don’t want to –

He’s live-editing the slideshow, for folks listening…

Cool. B is $1, and C is $2. How much do you think it costs us per month?

40 cents.

A. Correct. It was exactly 54 cents.

Wait a second, you just said 40 cents, and then you said correct. And then you said it was 54 cents.

Well, out of these options, option A is closest to the reality.

[laughs]

I didn’t give you the right number. I didn’t want to make it 50.

Okay… It’d be too obvious, I guess.

Too obvious, exactly. So maybe I could have done like 0.5.

That would have made more sense. There you go.

0.5. So I fixed it.

So half a buck…

Half a buck, yes.

…for a month of namespace.so.

Yeah.

Concurrent runners. That’s what they’re offering, right?

[01:05:58.17] It’s a faster GitHub Actions runner, with caching built in, and a bunch of features, like for example a nice UI which shows you how long your builds are taking, how much CPU they’re using, memory using… So you just get more insights into what’s happening when the builds run.

Right. Not a sponsor, but they certainly should be.

I think so.

They’re on my list.

I really think so.

Put that on your list, Adam. Okay, cool.

2025 named out, so I’m coming for you.

So thanks for spotting us, Gerhard. This is on your credit card, right?

It is. Yes. 0.5 dollars.

So just invoice us.

For the first month. We’ll see how the next month goes. The pay as you go thing is a basic, but yeah.

Yeah.

Pull request 535, this was also a follow-up - we improved on the Zulip auth integration.

This is my fault, too.

Yeah. You remembered like the old way, where we just basically configure environment variables like secrets in our fly.io app… And we don’t want to do that. The reason why we don’t want to do that is because we have the 1Password CLI integration, which means that when the app boots, just in time, it loads all the secrets. So that’s what the 535 is.

Yeah. I just forgot about that, so I just did it the old-fashioned way.

That’s okay.

And so thanks for fixing it.

That’s why I have two have everything. You want a backup. You want someone to back you up.

Yeah. Two developers… [laughs]

Yeah. Cool.

Break: [01:07:16.06]

Alright, we have a bit more time, so now we’re going to go into one of my favorite topics, that has become one of my favorite topics, and that’s the pipe dream.

Yes…!

What is a pipe dream? Tell us, Jerod.

The pipe dream is a world, a future world…

“In a world…”

In which – ooh, Adam should tell us. He has the actual trailer voice… In which you can just run your own little CDN with a Varnish config deployed around the world on fly.io machines, and you don’t need to have a CDN anymore, because you’ve built your own CDN, and it makes sense, and you can just open source it and share it with the world, and everything’s simple, and you’ve got 20 lines of Varnish… Maybe 50, maybe 100 lines, maybe 200. You’ll have to update us on the lines of Varnish.

Still 60. We’re still on 60. That has not changed.

That’s pipe dream.

That’s pipe dream. Single-purpose, single-tenant CDN for Changelog.com, that runs Varnish Cache, the open source one, on fly.io.

So we’ve been working towards this. You’ve been building it. Last time around I remember saying “I can has test suite”, or something like that…

Yeah, pretty much.

And I know I looked at a test suite pull request, so I think I know where this is going…

Yeah. So the issue that Jerod opened in the true open source spirit is wire up a test harness.

That’s right.

Because I said “Create an issue, open source for the win” and you did it.

That’s right. You know, I opened that issue after listening to Kaizen 16, and hearing it back, and you telling me to open an issue, and I was like “Oh, I never did.” So there I went and did. I even quoted myself, and I quoted you in the issue.

Yeah, very nice. You did. That was amazing. So that was issue two.

Issue two.

That was a pull request, pull request three, which adds tests.

Yes.

So what is interesting about this? Well, we’re using just. Same as before, one tool. We seem to be standardizing on that.

Yeah.

It runs just test, and when we run it, it creates a report. Why does it create a report? It uses this amazing tool - many have heard of it - Hurk. Not sure if it’s the best name, but anyway; we didn’t pick it. Hurl.dev. And Hurl.dev is orange open source. It’s a CLI that – actually, it’s more than a CLI, but you interact with it through a CLI, which allows you to write tests and assertions about HTTP requests. So the focus is on testing HTTP endpoints. And it’s written in Rust, which means it’s really fast… It has a very simple DSL. You put it in .hurl files. And what does it look like? Well, let’s have a look at the files changed. So we’re going to look at files changed… There’s the workflow which runs it, and it’s just tests. That’s it. And when it comes to the hurl file - I’ve put it in test, by the way; there’s the just file, we’ll scroll past that… Let’s look at this one. Testadmin.hurl. We get the host admin., we repeat it twice, so that we can confirm the caching behavior… We expect the HTTP status code to be 302, and we say we want HTTP/2. You can specify this in the file, when you do your configuration… And then you can write a bunch of assertions.

[01:13:57.29] In this case, we make sure that the response comes back in a second, by 1,000 milliseconds in this case. We check that the header, the location header sends us back to home page, because we’re not authenticated…

Right.

We also check that there’s an ex-Varnish header, because that means this request has been served by Varnish. We are looking at the age, the age header, which should be zero, because they should never be stored from cache… We look at cache status, just to double-check, and the miss; the cache status header should contain, in this case, hits zero, and it should also always contain miss. All this very nicely laid in a way that I think is easy to understand. And the fact that we can do repeats, too - that’s all we have to do to make sure the request gets run twice, which I think is really nice.

Yeah. So this is like a Hurl-specific DSL, which is text-based, and as I said in your pull request, seems super-simple and easy to use. So I’m excited about that.

Yeah. So this is pull request three on the pipe dream. And again, there’s a video… How does this work? So I’m just going to play through it. There’s no sound… But the thing which I wanted to go to is this report, which is really cool. So after you run the test, you can have a look at the output, which shows you exactly the requests, how they happened, what headers were sent, how did this behave… Which I think is really cool. Did you try running this locally, Jerod?

I watched your video. I didn’t try running it.

Excellent. So the video works.

Yeah.

And you get a nice waterfall to see how much the different parts of the request take… I think it’s super-useful.

So my question that I had after this, which I didn’t ask, yet - I was saving it - was obviously you run the thing, you get the report… But I assume the thing also has some sort of automated one or zero at the end of it, whether or not your tests passed without the report, right? The report’s an additional thing. It’s not like the output.

That is correct, yes. That is correct. So the report is separate. So the report is separate from whether the test was successful or not. And right now, this commit has failed. So this test on main has failed. So we see the failure. And the failure is that string edge grace hit stale should not contain string stale. And it does. And also, the age, like how long this has been cached for - we expect it to be less than 60, but it’s been 67. So the system doesn’t seem to behave the way we thought it would.

So this is testing not the Varnish config, specifically. This is testing the actual running nodes.

Yes.

This is the thing in production that it’s testing, right?

Yes, that’s it.

Okay. So it’s almost like an integration test.

It is an integration test, yes.

How would you use it for development then?

Right. So this is the big thing which currently is missing. It’s not testing the configuration that is local. It’s testing the deployed configuration.

I see. That’s what I was just asking about. Yeah, because what I want to do is TDD some changes.

Exactly. So for that, we need to put more stuff, which spins up Varnish locally. And then the question is, “Should the local Varnish hit Changelog the origin? Or should we also spin up an origin?”

Are you asking me that, or are you saying…?

Yeah. We are going through this to see – because here’s the questions that we need to –

[laughs] I couldn’t tell if that was a rhetorical question or not.

No, no. Here’s the questions that we need to answer so that we know how do we want to continue developing this… Because based on what we choose, there’s almost like different trade-offs, and different levels of how hard this is going to be. So do we, for example, want to run the entire Changelog app locally? And if we do, then we need the database as well, which we can do; it’s not a problem… And then we put Varnish in front, which is the one that we develop, just in time, loaded with the Varnish config… And then we run the test. So we need almost like four things to be running locally, to be able to test everything, how the entire system fits together.

[01:18:09.17] My desire would be to have my changes and additions to the pipe dream config, aka Varnish, tested, to assure that what I’m changing actually affects its way upstream or downstream, whichever way you want to look at it.

Right.

I do not care in this context whether the upstream or the downstream actually work correctly. In fact, I would like to be able to mock them in different ways, like “What if the app doesn’t respond the way we expect to?” I would like to mock that response from the app. Because the app’s interactions and stuff is all tested elsewhere. And then our other upstreams is like Cloudflare R2…

That’s it.

And so that’s outside of our control. So we don’t want to test if that thing’s working as it should. So I think we just want to keep it isolated to pipe dream, and not spin up an entire working system with nodes around the world, and stuff like that.

Okay.

Does that answer your question?

It does answer my question, yes. That’s exactly what I was thinking. I just want to double-check if you’re thinking about it the same way… And it seems that you are. We only care about the configuration and pipe dream itself, and how it interacts with origins, different origins in this case.

Right.

To begin with, we can just point it to those origins, so that won’t change… But what will change is that we are testing the thing that is being developed, the local thing. We’re not testing the thing which is being deployed. Because the reason why I wanted to obviously test the local thing is - well, is my change going to work before I push this out?

Absolutely. And we could even take those origins responses and do something like a VCR, and have those be played back.

Yes, yes.

And then we avoid production altogether.

VCR.

Remember VCR?

Yeah, exactly. For people that don’t know Ruby, are not coming from that world, the moment you said that - of course, VCR and the cassettes, and damn it, I had to recreate them so many times…

Right, right.

But yeah, I remember that. Yeah. It’s a Ruby gem which basically allows HTTP requests to be recorded and replayed, so that you don’t make the real requests.

That’s right. And for those people who are even less old, originally, VCR was a tape-based medium in which you could record and play back television.

The OG VCR.

Yeah.

I hated those movies, because the quality was so bad… And the more you watched it, the worse it would get. The worse it would get because - yeah, the media would degrade. So yeah, luckily we don’t have that problem anymore.

Plus my original Star Wars VCR, which was recorded off of television. It got recorded over halfway through for like a basketball game, or something. It was like “Goodness gracious, man. I’m trying to watch Star Wars here.”

You know what’s funny about VCR? It stands for Video Cassette Recorder. But you would use a VCR to play…

That’s true.

…primarily. As a user.

You could do both.

I know you can, but generally, most people assimilate or think of the VCR as you put a cassette in and you play it, is all I’m trying to say.

Right.

The general usage is not so much the R part of it. It’s the VCP, Video Cassette Player.

I think what we learned here is separation of concerns is a good thing. Don’t make the recorder and the player in the same exact medium. You’re going to record over my Star Wars.

But do you remember the safety mechanism that cassettes had? There was a latch. If the latch was on, you couldn’t record over it.

Oh, yes. The tab, yes. And you could just tape over it. Remember?

That’s right. A physical little thing there.

Oh, gosh… The days. The days, man.

The bad old days.

[01:21:41.15] Okay, so this all makes sense. I think for this what I would do - and again, that’s why we’re talking about this - I would introduce Dagger for this. The reason why I would do that is because I need to create containers quickly, programmatically. I need to create services, connect multiple containers together, and check that everything works in a programmatic way. And if we don’t have that, we would need to have some sort of a container runtime to run all these things. So that’s what I’m thinking here. Sounds good?

Sounds good.

SGTM. Cool.

Make it so.

So we’re almost at the end… This is what we have been building up. Let’s see how this lands. So we talked a few name ideas in the last Kaizen for pipe dream.

Correct.

What do you remember as a name that stuck with us all?

Pipely?

Pipely…

Pipely… [laughter]

So we threw around a couple of domain names…

Correct.

So one that we all liked… [unintelligible 01:22:41.28]

Oh, gosh…

[unintelligible 01:22:46.13] Do you remember the domain which we wanted?

No. [laughs]

Okay… So there was pipe.li. I think that was already registered as far as – no, it’s unavailable, actually. We can’t even register it.

Oh…! For shame?

Yeah. I mean, it says it’s already registered. There’s no info… Yeah, .li domains are difficult, for various reasons. So what was the other TLD that we wanted?

Tech.

Pipely.tech. Let’s see if Pipely.tech is available. No. Someone registered it. When did they register it? Let’s see. What happens if you go to Pipely.tech?

Oh, gosh…

Oh, my…!

Somebody else has the idea.

“A new CDN is born.”

What do we see?

It looks like –

What?! What are we looking at?

Three…

This is us? Get out of here…

[laughs]

These are –

The three Magi.

Oh, goodness.

It’s Christmas. New things get born. [laughs] So a CDN is born…

Oh, the stars in the sky… The CDN is risen.

I don’t want to burst your bubble, Gerhard, but it wasn’t three Magi.

Three wise men?

Nah.

What was it?

Well, it was a group of wise men. There was three gifts given. And so people always think there was just three of them. But people don’t read the account very closely. But this is cool-looking. I actually thought these were Jedi at first. So I thought you were going – because that looks like a futuristic city out there.

Yup.

Certainly, that’s not Jerusalem, or Bethlehem, or anything in the story.

No. It looks like Dubai.

It does look like Dubai. [laughs] So maybe a modern take. But hilarious. “A new CDN is born. Pipely.” Coming – what? Coming on the 25th? Or what are you trying to do here?

Maybe. I don’t know. So… Pipely.tech. So if you go to Pipely.tech, it’s a domain and it tells the whole story. So should we build a CDN? I can click on that.

Oh, cool.

That’s us. Always three. That’s interesting. So three seems to be the theme here.

Always three… [laughs]

Not two, three.

Now there’s three wise men. Is that what you’re trying to say? I like it.

I think so. I think so. Comparing CDNs… That as well, with nice screenshots. So the whole story. The whole Pipely story, which I really I like. I like the name Pipely. I’m sold on it.

You’re sold on Pipely… Well, you bought it. Pipely.tech.

Yeah. It’s a thing now.

That’s the one that I remember… I like it.

I’m so down. Like, beyond so down.

Pipely’s coined.

Can I share some behind the scene nuggets?

Of course.

This is fresh. This is on a pod coming to you soon. Actually, next Wednesday. So on Wednesday of this week, which is the week of, I guess the fourth… No. What was Monday? The second?

Monday was second. Yeah. Yeah. Second.

December 2nd.

[01:25:53.16] It’s December 6th. So on December 4th, I had a conversation with Kurt Mackey. And I think you know his name because he’s one of the founders and CEO of fly.io, which we know and love here. Obviously. I love Fly so much. It’s so cool. And during that conversation - because we talked about Tigris, and the Rebel Alliance. He said it out loud on the pod. So I thought it was secret. We talked deeply about the Rebel Alliance, and all the things that he had envisioned for it… And he’s not – I won’t spoil it, let’s just say. That’s not the point I’m trying to make. I said, “Go to this URL and tell me what you think about this.” And I mentioned this pipe dream idea. And I had forgotten about Pipely, until this moment, in terms of a name. I laughed so hard the last time we said it… And I’m the one that said it. [unintelligible 01:26:40.24] mentioned in the podcast. And so that conversation was focused around the pipe dream idea. And he looked at it, he’s like “I love this. This is so cool. I can’t believe you guys are doing this.” So I’ll just say that, and the fact that Kurt is excited about this. And there is this idea of a Rebel Alliance. Just saying. So we have support. And a blessing.

[laughs]

And a fan.

And a name.

And a name.

And a story.

And a domain.

And a domain, exactly.

Gosh, it really does begin with a name and a domain. Because you can have a good name – I was actually bummed. I was like “Holy crap, somebody took Pipely.tech? Who had this idea?” [laughter] That is so cool. I mean –

You did. [laughter] I just had the execution. I’m like “That’s cool. Let’s go for it.”

I love the creativity, though. I love the idea that this might be Jedi, honestly. And this is a futuristic city, with this star out there.

The Rebel Alliance is born.

Yeah. I think this is just really – it plays in well. I like it a lot. I’m beyond excited. I don’t know how much to share in this podcast, but I’m like beyond excited.

Well, I’m glad. This is a great way to begin something new. And I didn’t know that you’ll make room for this, but I think you just did, without knowing that Pipely would be a thing. Think about how we began, and think where we’re now.

To rewind the conversation back to the beginning… Jerod said something to me. Jerod and I don’t see each other face-to-face too frequently. A couple times a year. And it actually – I’ll say this on the air, Jerod. I think I said this face-to-face, but… He said to me, he’s like “Adam, when you tell me new ideas, I almost immediately cancel them, because we have no time to do anything.” And I’m paraphrasing what you said, but the sentiment is roughly there. So correct me, Jerod, if you want to. But it kind of bummed me out, because I’m generally - not so much the idea guy, but like someone sort of like generating vision in some way, shape, or form… And like when he said that, I also thought in my brain in the moment, like “Gosh, I’ve kind of stopped like casting any sort of vision in my own brain, because I’m kind of tapped, too.”

So anytime I have a new idea, I don’t have any room to explore it, because we’re doing all the ideas we can, essentially.

And I don’t know where Jerod’s at with this, but I would so make room for – I think this certainly makes room for Pipely. And it’s certainly the main thing, because we need an amazing CDN. And I shared in that podcast with Kurt some of the challenges, and he said, “Well, you know, Adam, that Fastly and Cloudflare and every CDN out there is not – it’s not in their best interests to cache all of your content on all their POPs across the globe forever. That’s not in their best interest.

There’s cache misses, not because they can’t cache it, because they don’t want to.” And so the CDN we build, that we want to build, will always cash everything you need to across the globe, and it will only expire when you say “This is expired.” There will never be a cache miss in this world. And that to me is what a CDN is.

And so if we can build that world, I think other people will like that world. And I think people don’t need - and this is even Kurt concurring on this. I don’t think everybody needs what these larger CDNs can offer, or do offer, because they just offer so much more than you need. So I think there’s a lot of opportunity here, honestly.

[01:30:14.29] I like it. I’m behind it.

I could tell. I love it.

As much time or as little time as I have, little by little, step by step… I mean, this is honestly - like mornings, and evenings, and weekends, to the point doesn’t even matter. As much time as I have, I enjoy doing this. And on top of all the other things. And it’s fun. And I can see the need, I can see me wanting to use this, me wanting to build this, and just – I honestly see this improving a bunch of things. And if it doesn’t work out - it may not work out - the story is amazing. And at the end of the day, that’s what people remember. We tried. We showed up every day, and… Let’s see what happens.

We came, we saw, we dreamt. We pipe dreamt.

We pipe dreamt.

And we pipelied.

Pipely.tech.

The Pied Piper. The three of us, we’re the Pied Pipers. Look at me…!

So in all honesty, I think the reason why these Magi, they are faceless, is because it can be many other people. I know that we had Matt Johnson help us. We had James A. Rosen help us. And how many people out there are doing something similar? Maybe they don’t have time, maybe they want to, they’re thinking about it… Crazy groups of people, that have an idea, and just go for it. It doesn’t matter where it goes or how far it goes, as long as you have fun with it. And don’t take it too seriously, I suppose. It’s not about the profit, it’s not about like this VC funding - or at least that’s how I think of it. It’s an idea that we should keep investing in, because it’s the right thing to do, and it’s a great story to tell. Looking back 10 years from now, 20 years from now, we’re like “Wow. We were part of that.”

On, I guess, a different note, how close are we to this pipe dream not being a pipe dream? How real and how quickly can this be truly real, if it’s not real?

So honestly, it just depends on how much time me personally can dedicate to this over the next coming months. That’s what this comes down to.

I don’t mean Pipely the company, or anything like that. I mean like our usage of pipe dream.

Our usage, okay.

The thing becoming real for us as the first, in quotes, customers, if that’s the thing. True usage. Does it make sense? Can it scale for us? Is it truly DevEx usable, etc? And then I think everything else after that is just like comes natural if it makes sense.

So let’s talk through the roadmap. We have it right in front of us, we’ve added tests, we know there’s more that we need to do here… But the same number of VCR lines, so that’s still there. The functionality hasn’t changed. The feed’s backend - the only reason why this is not done is because I wanted to do Pipely.tech. I thought that was a cool idea. That’s the only reason. I did part of it, by the way. This now works. It didn’t used to work, the feed XML. Now this loads, but we only had HTTPS, so I’ve taken small steps towards it, but it’s not complete.

The reason why we need HTTP and not HTTPS is because we’re using the open source Varnish, and that does not terminate TLS. So we can only connect to HTTP backends. That’s something that we discovered the hard way. So this in terms of configuration, we’re talking a few hours to get it all done. Lift it from our existing VCL, and maybe do a few changes. Because this already exists as VCL in our Fastly config.

Then sending logs to Honeycomb. This again, a day, roughly. It’s just like, I’m just basically making stuff up, because I don’t – only once you pick it up you realize how much… Like, all the rabbit holes.

[01:34:08.03] Well, we still have to determine if the way we send logs, the Fastly way is the way of the future. We know that we like a lot about what that does, real time etc. I’m assuming that’s what you mean by that.

So it’s more around making sure that we send logs to Honeycomb, so that we can understand what’s happening in the system. We want to keep the same structure as Fastly, so that all the queries will work. There’ll be no interruption of service or no big changes. So that’s why I mentioned this. We just need to have some way of sending all the request logs to Honeycomb. That’s what I mean by this.

We also need to send logs to S3. This is something that Jerod mentioned last time, when we talked about pipe dream and the roadmap. So for stats, we need to set – just to keep the compatibility, right?

What about swapping out Tigers for that, or MinIO, if we wanted to not S3 it? Is there any reason to not do that?

It doesn’t matter where we send the logs, as long as we send logs to an S3-compatible API.

Gotcha.

Which - I tried to switch over to R2, by the way, but I couldn’t, because of the way R2 implemented streaming, versus the way S3 does, and the way Fastly actually pushes over to that…

Interesting.

And so we could have been on R2 entirely, but we’re actually in both now, because our logs still go to S3 and everything else goes to R2. Now, we wouldn’t have that problem inside pipe dream necessarily. So it could be send logs to R2. But we want to keep the exact same format, so that our analytics stuff doesn’t get rewritten.

Exactly, yeah. So that’s why we do like the first part, which seems a bit easier, to just send the requests. It doesn’t matter which way you do them honestly, but this will ensure that the service behaves the way it should. And I think this is more valuable from a functionality perspective. Then, fairly hard to implement purge across all app instances. I say fairly hard… Oban, I can see me and Jerod getting together here, because we need to figure out how to purge these correctly, and how to do this from the app. So the application itself will know which are the pipe dream or Pipely instances, and will send these Purge requests. And we’re looking at the Fly.io machines, we’re running the same network, we can discover them, we have DNS… So a lot of the building blocks are there, but this is where the application needs to come together with the actual CDN runtime, in this case Fly.io, to implement this purge via Oban, and how we do things. So we don’t introduce a third or an extra service.

And then adding the Edge Redirects - this is really easy, because it’s just basically adding more VCL config. Most of it is copy-paste, maybe a few adjustments. So it’s very feasible. And we’re talking - I would say maybe weeks of work in total. But weeks of work spread over a long period of time. That’s what’s hard. The time on my part; have the time to actually go through all these things… But I can make this my focus. That is a possibility.

Where should we end this pod? Should we end it right here? The possibility of these Jedi Wise Dudes bringing to fruition the Pipely dream?

[01:37:14.03] Yeah, I think Pipely is a good idea. A new CDN is born. I mean, we can see the journey that we’ve been on since January. That’s when he started talking about it, it was like the first Kaizen…

This whole year. Yeah.

So we have been taking all these small steps towards it. We were uncertain for a long time… “Is this real? Should we really do this? Is it a good idea?” So we weren’t like all in to begin with. And I think that we are getting close to being all in, as in “Let’s go for this. Let’s implement it. Let’s see how it would work in practice.” Like, most of the problem space, I think we have discovered it. We know what are the big items; not the actual implementation… But this seems a lot more feasible and a lot more realistic than it was in January, for example, when it was just a question. And I think now it’s not like we’re building it. It’s like, we are, I would say, maybe a third way through, maybe a quarter through; something along those lines. And there are still challenges around, for example, the TLS termination. That is a big one. And that means that we can only proxy or forward-request. The origins, they have to be HTTP. The backends, as [unintelligible 01:38:22.20] calls them, have to be HTTP. I think that’s not a problem for us. It’s not a problem in the context of Fly, because everything is running - like, we have a private network, so all the endpoints are HTTP and they’re private. No one can connect to them. So I think that makes sense. R2 can also be public. It’s not a problem. And when it comes to pushing logs, that’s the one thing which I don’t know. But honestly, I don’t see Varnish being the thing which will push the logs anyways.

What I would use, that I’ve been using for years, is vector.dev. Vector.dev is a great tool for sending logs and metrics or anything like that anywhere. Datadog acquired them since I’ve been using Vector… But it’s all very simple and straightforward. And even to this day, I use it in the context - and we use it in the context - of the Dagger infrastructure. This is a very important component. And other teams are using it in their own infrastructures that we are collaborating with. So Vector seems to be the piece to pick and the tool to use in this context. Again, written in Rust, super-performant, nice CLI… It has a lot of things. So I think the building blocks are getting clear. Just a matter of going for it. New beginning, new year… I think that could be the focus.

Let’s go for it, man.

Let’s do it, right?

Let’s do it.

So excited.

Good stuff, Gerhard. Always a pleasure, man. So fun.

I’m impressed… It is always a pleasure.

We make a great team. We really do.

We do make a great team, we do great things together… And we’ve stood the test of – you’ve been here, Gerhard, since the proverbial beginning; you know, the new beginning, the latest era… Not including this new one we’re about to go on to. Yeah, 2016, like –

2015?

And that really brings my heart a lot of joy, honestly. I like relationships that stand the test of time, really… Obviously, that’s a good thing for relationships, but… Yeah, it brings me a lot of joy. That’s all I’ll say, I guess. I’m excited.

Likewise.

Pipely.tech…!

We have it.

OMG.

We have it. It’s real.

A new CDN is born… Let’s do it.

Kaizen.

Kaizen.

Kaizen!