Kaizen! NOT a pipe dream

We are kaizening once again. Kaizen 15, and Gerhard, you will be pleasantly surprised… This is Friends #50. This will be our fiftieth episode of Changelog & Friends.

Wow… Look at that. No way.

I know you like round numbers and milestones…

I do, I do.

Yeah. So you’re gonna love this. No random integer here… Nice, round 50. So here we are, Kaizen #15, Friends #50, Adam’s here, Jerod’s here, Gerhard’s here.

So good to be back. I was really looking forward to this one. We have a couple of things in store… And I’m not bigging it up… You will be impressed. [laughter] By degree.

Alright, the gauntlet has been thrown. We’ll see. Let’s see if you impress us this time around. Maybe I’ll impress you guys.

I know you will.

Okay. That’s easy then.

Yeah. You’ve already impressed me, Jerod.

Oh, good.

I have been paying attention to that repository. You have impressed me. [laughter]

Thank you.

So good to see you, Adam, as well.

I’m glad to be here, of course. I just love the ride. I love the ride.

Cool. By the way, just a heads up, we will be talking about bidets towards the end. So heads up. It’ll be very short, but there will be a short bidet conversation.

Oh, you have thoughts on bidets as well.

Oh, bidets. I thought you were talking about birthdays; like, you shortened it to B-days.

No, no, no.

Oh, is that how you pronounce, bidets?

It’s just how I pronounce it.

Alright.

I’m glad that you know what it means.

Okay. So you listened to our friend’s episode with Kelsey Hightower, I assume…

Yeah, I really enjoyed it.

Alright. So I’m ready to learn, because I’ve never used one, I’ve never even seen one in the flesh… So let’s save that to the end, because we have more important stuff to talk about. Alright, take us on a ride, Gerhard.

We will start somewhere, which is I think – I think it’s related to Jerod, I think. But maybe it’s Adam as well. So as you pointed out, Jerod, I do like round numbers. I do like dates. I pay attention to those things. So one of you will know what happened on June 21st, two years ago. Something happened this day, two years ago, and it’s related to Ship It show.

Was it your last episode, episode 90?

No.

No, that was in January.

You’re stumping me.

I don’t know. Related to Ship It show in June… Was it the original ship of Ship It?

So anyone with a terminal can verify this.

Oh, gosh…

You can do whois shipit.show.

Oh, we acquired the domain maybe.

We created it. 21st of June, 2022. That’s when ShipIt got –

I don’t know how you know that. How do you know that?

Just random. I was paying attention to things.

Do you have like a list of dates that you put in your diary, and you’re like “June 21st is now important to me, because shipit.show…”

No, I just stumbled across it. I just stumbled across it. That’s how it happened. But I thought it was very interesting that on this day, two years ago, shipit.show became a thing. So the actual domain.

Gotcha.

So that was nice.

That is nice.

Cool. So Kaizen 15.

Tell everybody, so we’re all on the same page, for those who haven’t listened to Kaizens 1 through 14, what it is that we’re doing here with Kaizen, why we’re doing it, and then we’ll get into it.

Okay. So Kaizen stands for continuous improvement, and that’s an abbreviation that many are familiar with, but change for the better. I think that’s what the translation is closest to. So the idea of kaizen started three years ago, almost three years ago, by the way; and we will get to that. Three years ago, I think it was summer. The first Kaizen ever, Kaizen #1. Ship It show episode 10. Shipit.show/10 and you’ll see it. And it started with Fastly, it was summer, and it’s when half the internet went down. That’s what happened three years ago.

[00:05:58.06] So the idea behind Kaizens was to improve things about the Changelog setup, on a regular basis, and then talk about it. So the pace was every 10 Ship It episodes, and it just happened to be every two and a half months. And for the past three years, every two and a half months that’s what we have been doing. And it’s been a lot of fun.

I did a little digging into this extended version of the meaning of this. It says, as we know, Kaizen is a Japanese term; it means improvement, or as you’d mentioned, Gerhard, change for the better. And it says “In the context of business, particularly manufacturing management, it refers to a philosophy or practice that focuses on continuous improvement of processes, products and services.” And here’s the part I like. It says “The concept of kaizen involves all employees, from top to bottom, all of management, to the assembly line workers, and it encourages a culture of sustained, incremental improvements.” I think that encapsulates a bit more, because it involves everyone. It’s not like you, or Jerod, or just me… It’s a collective, and it’s this culture, which I think is interesting, because Kaizen has – you know, most of the pillars, Jerod, of our business are built by you and I, Jerod. And then here comes Gerhard, with Ship It, and all the things he’s been with us for the past - I don’t even know how long, but pretty much forever… And he brings in this new pillar called Kaizen. And I think this is very much Gerhard-born, but adopted by all… And I’m really thankful, because Kaizen has become part of my own personal DNA as well. And I like that.

That makes me so happy. I’m really glad that three years after we started, we just add another break. You’re right, it does include everyone. We’ve always done this as all three of us. We’ve never done it like separately. So from the beginning, this has been the secret formula, and there’s been a lot of navel gazing. [laughter]

That’s a fact… Which is something that I generally am allergic to, and I try to avoid.

But actually, do you know what it means? Do you know what navel gazing means?

Well, it’s a stare at your own bellybutton.

It’s the contemplation of one’s navel as an aid to meditation. So you’re contemplating the basic principles of the cosmos and the human nature. Check Wikipedia. Omphaloskepsis. It’s Greek. That’s where it comes from. Navel gazing.

[00:08:28.29]

Jerod here, in the editing room. You know, I thought omphaloskepsis sounded familiar when Gerhard said that, but I couldn’t quite put my finger on it. Now that I’ve had more time to think about it, omphaloskepsis was our word for round three in the #define game show we played on episode 25. Maybe you recall it, “Game theory, dude.” I’ll link to that chapter from this chapter in case you want to jump over there and listen to it before returning to our navel gazing. Okay, back to the gazing.

And there’s lots of statues of Greek dudes, very muscular dudes, great-looking, gazing at their navels.

Yeah. Similar to us.

Similar to us, exactly. With more hair. They have more hair.

They do have more hair.

Yeah. So I think Kaizen is a better name. I think – yeah.

I like Kaizen.

I quite like that.

Yes, I do as well.

Let’s put a new spin on navel gazing though for me, because it’s generally been like a frowned-upon term…

Well, it’s something that you do for yourself, by yourself. And so navel gazing I think in public is where it becomes frowned upon. You can stare at your own navel all you want, and if it helps you with your thought process and your self judgment and all that kind of stuff, go for it. But navel gazing is a private activity. And when we do it in public on the airwaves, at a certain point you’re like “Let’s stop staring at our own navels.” But insofar as it’s continuously improving our navels…

Better mine than someone else’s… That would be weird. [laughter]

I would have to agree that yes, navel gazing outward is worse than inward. So at least we aren’t going that far. But I do agree that Kaizen is a much better way of framing it, so it feels okay.

[00:10:08.00] Here’s maybe a better twist on it. If you’re gonna navel gaze, turn it into a product. Like, this is a product. Kaizen is a product. We produce something as a result of navel gazing, and we’re doing that, but we’re doing it in the manner, in the culture of Kaizen, continuous improvement, improvement for change, change for the better… But at the same time, we’ve turned this sometimes habit or circumstance of just wanting to popularize the things you’re doing, and not be overly self-promotional. And we’ve always been a little guarded with being overly self-promotional. But I think that there’s been so much value in all the Kaizens, in all this navel-gazing, Kaizen, change for the better stuff that I think I now see a positive style and a positive light to what is typically determined as navel gazing.

Yeah. No, I do think that we’ve been sharing and promoting a lot of the work that the community has been doing from the open source, we’ve been promoting projects, we’ve been helping some of our partners improve themselves… Whether they wanted to or not; we had some of those as well.

That’s right.

The idea is we’ve always been outward-looking, and we try to make everyone that integrates with us better in some shape, or form. And that makes me really happy.

Well, while we’re navel-gazing navel gazing - we’ve gone up a level, or down a level, depending on which perspective you have - I want to talk about continuous improvement versus continuous change. Because while we bikeshed and talk about the details, I think there is a culture of change for change’s sake, which we could fall into, and perhaps have, and sometimes you don’t know what is improvement and what’s actually steps in the wrong direction… So I think we should ask ourselves, as we move along, are we just continuously changing for change’s sake? Or are we actually improving? and that’s part of the iteration process, it’s also part of the judgment, the retrospective, how is it going, was this a change for the better…?

Similar distinction I make between productivity and effectiveness. I think that you can produce a bunch, but really, what you want is to be effective in whatever you’re doing. And sometimes you’re just producing. And it’s like, well, cars produce as well, carbon dioxide, or whatever… So not always the best thing. So instead of trying to be productive, we should try to be effective. And I think instead of trying to continuously change, we should make sure that we are continuously improving… Which I readily admit you can’t know that until you make the change, and then you can judge it. But something maybe as a Northstar as we move forward is like “Is this change actually for the better, or just for the sake of trying something?” Alright, let’s stop navel-gazing the navel gazing. Let’s start getting to the standard navel gazing now.

Yeah. So I have a couple of items on my list, and there’s like three buckets. So what can you expect? You’ve just joined, or maybe you’ve skipped forward through the navel gazing part…

Yeah, chapter break…

And then you just came and landed here… So what can you expect in this episode? The first thing, I’d like to give kudos to some people in the open source community that did some great work. I would like to share some fly.io love. There’s a lot of fly.io love to go around. We’ll get to that in a minute. And also, I want to talk about turning pipe dreams into reality. And that’s an important one.

So, rules for today… We’ll share the background story. We have lots of stories to share between three of us. We will share those stories.

Okay.

No peeing in pools.

Gosh.

Remember the last one? No peein in pools, okay? [laughter]

Alright…

And most importantly, we want to celebrate the wins. We had quite a few. Navel gazing is allowed, by the way; you can do it, it’s okay. And you can’t see mine, so it’s all good.

That’s right.

So the first thing, the first person that I would like to congratulate is you, Jerod.

Me?

Yes.

Well, let me have it. Why? What did I do?

You made the first PR in nine months.

[laughs]

I noticed that.

[00:14:10.26] I did. I did that kind of for you, I did that kind of for me… I was like “It’s been a while…” I normally just push straight up on the master branch. But I thought “This is a noteworthy change. I should open a PR, and Gerhard’s gonna love this.”

I did. Oh, man. I loved it so much.

Kinda pandering…

So 510. Pull request 510. If you go and open up GitHub, our GitHub, changelog.com, “News redesign.” Oh, yes, baby. That was amazing. By the way - “baby”, I’m only using it as an endearment. The fact that it took nine months has nothing to do with it, okay? This is not your baby. It’s a labor of love. It’s a labor of love. It’s passion, commitment, but it’s not a baby. So anyone with this thought, I want to discourage it, okay?

Okay, good. Yes.

So I would love us to talk a little bit about it, and I would like to start – anything you want to share, any of the background stories, any of the things that you want to share about the news redesign, Jerod?

Yeah, absolutely. So the background story is that we had a newsletter for many years called Changelog Weekly, which you would subscribe to on Changelog.com/weekly. We also have a nightly newsletter, which is still going, which is an automated scraper of the most starred repos on GitHub, which is increasingly becoming simply AI, crypto, and malware. So it’s getting to be less useful than it used to be… But that one goes out weekly, it went out weekly, and that’s why he named it Weekly. Right, Adam? We had a hard time coming up with a better name for that, because well, we already had Nightly, and Weekly, and… And it was what it was. And then we started the Changelog News podcast as a flavor of the Changelog… Which was quite a success, a hit. People liked it, we like to do it, so we kept doing it, and we decided to streamline the newsletter, and the podcast, send them out at the exact same time… They were already complimentary; let’s just make them more efficiently complimentary. And so Changelog Weekly became Changelog News, which was just both a podcast and a newsletter. People who are listening to this pretty much already know that.

The technical side of that was that the podcast became its own podcast in our system, and had a podcast page just like all the other ones, just like Ship It’s page, just like JS Party’s page, etc. But it had this newsletter component that wasn’t very well featured, basically.

So when you’d go to changelog.com/news, you would find the Changelog News podcast, and there’d be a subscribe button with newslettery things… But we weren’t featuring the newsletter at all. And so this redesign was an effort to have a different page that lives on changelog.com/news, that serves both the podcast and the newsletter better, in terms of people understanding what we’re doing there, being able to quickly read the most recent issue… You know, give it a better splash page. Because lots of people land on that page, and we want them to know what it is. And it’s more than just a podcast. So that’s the backstory.

A couple of shout-outs. Of course, Cody Peterson, who worked with me on the design and the frontend implementation of the new splash page. Shout-out to me for coding it up and opening up an epic PR, which I don’t normally do… And to a handful of folks who decided to send us kind words to use on the page. We have a lot of people who enjoy the podcast, enjoy the newsletter, and they tell us this privately via email and stuff, and so I reached back out to everybody who reads it and said “Hey, if you would give us a quote, we’d like to put some of those on the homepage.” And so shout-outs to Mary, to Justin, to Chris, to Maros, and to [unintelligible 00:17:57.09] whose quotes were awesome. And we got more than those, so thank you to everybody who sent one in. Those were the five that we selected to feature on the homepage. And that’s pretty cool.

[00:18:15.23] Being able to hear from and see some of the people who take value in the newsletter I think is powerful when you’re trying to decide – like, two things you want to know. A, do people like this? And then B, what does it look like? Show me a recent issue. And I think with this redesign that we’ve accomplished that. So that’s that.

Very nice.

Coded it up, rolled it out… It’s there, it’s ready.

So I really love the emails. It’s one of the few newsletters which I read. Whenever you send the newsletter, Jerod, I really enjoy it. I have clicked on the View in Browser button, and I noticed – I think until about 90, like issue 90, it had previous design. That was [unintelligible 00:18:58.29] side hustle expose. That was the last issue, which had the new design. And then, 91, it was the new one. So all you have to do is literally drop the email part, and you’ll see the previous design. So news/94, /89, so on and so forth.

Right.

And the reason why I said 89 is because maybe you’ve done it by design, maybe you knew what you were doing in this case…

What did I do?

…the previous issues have also improved. So when you ship the new redesign, all the previous ones look amazing.

Oh, thank you. We did it on purpose. We did them all.

You’ve done it on purpose. [unintelligible 00:19:36.15]

Yeah, we did it all.

If you haven’t seen that link in the email, click it and check it out. Otherwise, changelog.com/news/90.

Yup.

So… I have a question, and I’ll go first. Okay?

Okay. That always gives us time to think.

Exactly. What do you like most about the new redesign, about the News redesign? What do you like the most? I will go first, and you’ll understand why in a minute. So my favorite feature of the new redesign is the 27 more reasons to subscribe.

Oh, you like that part?

I love that. In particular reason 15. [laughter]

Okay, which says…?

Who else takes the time to list 27 reasons to subscribe?

Right?!

That’s right.

I thought somebody in the world would get a laugh at that, and I’m happy to hear that was you.

I did. That was so good.

That shows you read all 15, or all 27, because you got to 15. Yeah.

Did you make it all the way through?

I did.

Okay.

And I improved it. And I’d like to show you how. [laughter]

Okay.

28.

So are there 28 reasons now?

Well, let me share my screen… I will obviously narrate this, and you’ll understand why. Okay, so what I’m sharing right now is Changelog.com/news/99. So if you go in a browser, you can see this. “27 more reasons to subscribe”, love it. Reason 15, amazing. So how would I improve it?

I don’t know. Oh, 29 more reasons to subscribe… Okay. I like this.

29 is important to me, right? It’s only my birth date.

It’s a very important number. Now, did you add two reasons, or…?

I did.

Okay.

Oh, my gosh. Scroll… Scroll…

Yes! He noticed it! [laughs] So what I did – hang on, let me just make this a little bit bigger… What is different about this list?

Well, it’s got zeros in front of it. Or did it have zeros already?

No, it did.

Okay. My bad. Well, it’s a –

It’s a pyramid scheme. It’s using a pyramid scheme.

I like that.

I reordered them.

I like that, because I’ve actually done that in the newsletter before. Have you noticed?

I know you did. That’s why I paid attention to that.

Okay, I love that. Yes, it gets longer and longer as you go down. So you’ve reordered them.

I did. So I’m going to submit a pull request, and you let me know if you like it, okay?

I will accept this.

[00:22:05.19] Cool. Okay. And for everyone else, there’ll be a new pull request. I don’t know the number, but by the time this comes out, I’m sure we’ll have a link.

So funny story about this particular feature… This was almost entirely Cody’s idea, because I’m like a) I want social proof; I want people’s quotes on there. And then b) we do a few things differently, and they’re all minor; it’s like minor reasons. But I want to have somewhere a list of the stuff that we do, like we don’t track your clicks, blah, blah, blah. And he’s like “Well, what if you just did way too many of them, and just made some up?” And he actually stubbed it out with 36 reasons… And I took it as a personal challenge to come up with 36 reasons why you should subscribe. And in the mid ‘20s I’m like “Cody, this is too hard, dude. I cannot come up with – he’s like “I did not expect you to come up with 36 reasons. I just made up a number.” And so I finally made it to 27, and left it there. But I love that you’ve improved it, you’ve added two. I will go on record to say we are accepting pull requests on reasons to subscribe. So if you have more reasons, I would love to get that number as high as humanly possible.

99. 99 reasons.

Yeah, I mean… You have to subscribe if you read 99 reasons, don’t you?

Yeah.

Excellent job here. I will merge this.

Thank you. Thank you.

Break: [00:23:21.09]

Now, I want to show you a really cool feature. So I’ve used the previous improvement that we shipped last in the last Kaizen, which is I’m connecting to the Neon from my local environment. Look at this. So I’ve done this, I’ve created a branch, I think it was a few days ago. I can’t remember. And obviously, Master has moved since, right? So if I’m going to close this, I’m going to go to Home, localhost 4000, and if we go to the podcast “Securing GitHub” 2024.06.19, right?

Right.

I’m sure there’s a new one… Let’s just go back to the main one.

Yeah, there is.

Let’s see what it is. What’s the last one? So “Polypandemonium.”

Yes. JS Party episode #327.

Cool. So Neon - I have the console open. I’m going to click on my branch, and I’m going to say “Reset from parent.” This is me syncing the data. Yes, reset.

This is your dev branch. Yes.

Okay? Done.

That was fast.

Two seconds. I come back…

So fast.

…I refresh…

Polypane.

Boom.

Bam. That’s cool.

Cool. So that was it. Alright. That made me very, very happy. Very happy. The whole thing, really cool. So over to you. What do you like the most? I’m going to stop screen-sharing. What do you like the most about the news redesign?

Well, you kind of stole mine, Gerhard. I was gonna say the 27 reasons…

It can be the same. It’s okay. Now they’re 29. That’s good.

That was definitely my personal favorite part, because it’s just kind of zany, and something you don’t see all the time… And probably that most people wouldn’t even notice, but it’s there. And so yeah, I’ll pile on… I’ll say the pyramidical structure of 29 reasons to subscribe is definitely my favorite part. Adam, yourself?

I think it’s just as simple as having a much better landing page after literally just so long. I mean, so long I can’t even tell you how many years to have a better place to send somebody to subscribe, that showcases the content, and the reasons, and the ability, and the social proof, all in one place. This divided design was like – I know when we first talked about our newsletter, early on, before it got into code, or dev, or really like some fleshed out design, we were talking through like “Okay, the newsletter could be on the right, and the left side can be static, and you can easily scroll on the right.” And it matches the brand, because the left side can be dark, and the right side could be the content… And it’s obviously massaged and morphed since then, but the basic idea was really good. But then having a place now to just call home for Changelog News, which is Changelog.com/news… I really wish we had .news, but whatever. We won’t get upset about that.

That’s my – I mean, continuous improvement… I’m still holding out.

Let’s kaizen that at some point.

I mean, Gerhard knows the exact date that we bought shipit.show, right?

Yeah…

When we get Changelog.news, and I can stop saying .com/news… I mean, that’s gonna be a huge win. And someone out there who holds that domain and needs negotiating power better not listen to this. Because when I talk to them, I’m totally gonna play it cool, like “Hey, do you care about this domain?” I mean, just [unintelligible 00:31:35.01] Happy to pay some money for it, just don’t want to get raked over the coals for it… But man, we’ve got to have changelog.news. It’s just so much cooler.

Yeah. And really too, I think my other favorite thing about this is that it didn’t – the existing newsletter design didn’t have to change really at all, if any, to fit into this new design. So the original email design got pulled into this, and so I think that’s kind of nice that we can literally incrementally improve Kaizen, this newsletter, to be - beyond newsletter, a decent design, and good design, and great content - to a full-on landing page that doesn’t have to change a lot; it just sort of added to it. It was additive, versus total morphed. So I like literally how iterative we got to this point to have this kind of page.

[00:32:26.15] And one more note on the newsletter redesign… So for those who haven’t seen it, as Adam described, on the left hand side it’s dark, and it’s really representing what you’re looking at, and has the reasons to subscribe, the buttons and stuff… And on the right hand side it’s the most recent issue, so you’re literally reading what it looks like. And in light mode, the right hand side has a white background. Now, in dark mode they’re both kind of dark, and there’s a border in between… But the idea behind this, kind of the metagame is really – the podcast/newsletter is a hybrid. It’s two things in one. It’s the only podcast that is a newsletter, and it’s the only newsletter that’s also a podcast that we know of, in this space… And so you kind of have this two-faced thing going on, where there’s a duality to the content, and there’s a duality to the design… Which I think is kind of cool. Probably that gets lost on most people, but for those of us who put it together, we just thought that was kind of neat.

Now, I don’t want to directly reference Two-Face, because I realize he’s a villain, and it’s probably not the best association… But you know, it’s got that duality thing going on, which is pretty cool.

It took me a while to realize that there is a button in the top right corner, you can press Play.

That’s the one thing that we probably should continue to fix, and make better is like you want to be able to read it and you also wanna be able to listen to it. And yes, there’s a Play button in the right hand corner, and as you scroll the newsletter - like, that’s sticky; it’s not gonna scroll with it, so it’s always right there. But I’m guessing - and I don’t look at stats very often, but I’m guessing less people are listening to it on the web than it used to be… It used to be our standard episode player, and now it’s kind of a unique thing.

You also can’t skip chapters, and stuff. I think when I’m wanting to scan an episode and see if I want to subscribe, I kind of want to click through it and maybe like scrub it, and none of those tools are there, because we kept it very stark… But definitely ways we could make it better, and make it more obvious that yes, you can also listen to this right now, while you’re staring at it.

And you almost want like a player of sorts, but then that’s too much. Like, does it really require that? Because the listenership isn’t there on the web to like put that dev behind it. And I think design-wise it could stand out a little bit more. Like, the Play button could be more Play-buttony, besides like literally a Play button… I don’t know, it’s great design. It’s just – there could be some tweaks and turns to let it stand out a little bit.

Some sort of affordance that draws attention to that Play button. Maybe like has some sort of a sheen to it, or like something that just draws your eye over there and it makes it seem – maybe even the word Play under, so you can just say “Oh, I can play this”, because we all know what a Play button looks like, but they kind of blend into the background when we’re reading… So yeah, we could definitely make it better.

I really like how we are improving Jerod’s pool. This is amazing.

[unintelligible 00:35:02.08]

I would like us to keep going… No, we are improving it, for sure.

I appreciate it. I appreciate it.

I guess a question for you, Gerhard, is you showed off Neon there, and you showed off how you can reset to - what was it, parent branch? Is that what the –

Yeah.

Okay.

So I have a branch, and I was just basically synchronizing it with all the data that’s in the parent branch.

Well, why did you show that off?

Because I was using it locally, and obviously the two diverge, so I have my own branch, literally my own branch from the database, that I can always synchronize, basically catch up with Master –

Dev_gerhard, or something like that, right? Doesn’t that give you like a default –

It doesn’t have my name yet, but it has a date.

It doesn’t? Okay, gotcha.

Yeah.

So what happens if you make local changes? Maybe you go in there and delete some episodes while you’re doing development.

It’s on my branch. It will not affect the production.

Right. But then when you hit sync, it’s not going to merge changes, it’s going to just –

it’s recess. It’s one way.

It gets you back to where you were.

It’s one way, Yeah.

That’s what you want.

Exactly.

[00:36:00.01] I’m still a little tentative of those things, because I have – I guess anxiety is the word. You know, I played basketball early in the mornings, and one time I got halfway to the gym and realized I didn’t have my socks… Which pretty much ruins it, right? And it’s a 20-minute drive. So I’m either coming home, or I’m missing that day… And ever since then, I’ve had like “Do you have your socks?” anxiety. And to the point where I put them in my bag, I put my bag next to me, I’ll start to drive my car, and I’ll stop – I won’t stop, but I’ll think to myself “Do you have your socks?” And I know intellectually that I have my socks, because I literally –

Oh, man…

But I’ll still just check. I’m like “Well, I’m just gonna check… Oh yeah, there they are.” And I have that a little bit when you’re like just having this production branch, and then your dev branch, and you’re doing stuff and you’re like “Am I sure I’m not connected to the production branch?” You know?

Yeah.

It’s a little anxiety-ridden for me still. And probably because I’m just not used to it. Or if I get bit by it one time, I’ll probably never want to use it again… But the value is there. So it’s interesting to me, but I’m still kind of just a little bit like – I don’t want to accidentally screw something up.

Yeah. Well, there are backups. The whole point is like if we did make a mistake, is the system resilient enough that we can recover from it? And how quickly can we recover from it?

Sure.

Right now, let’s say we delete something in production… Well, do we have a backup we can go back to? Yes, we do. How much are we going to miss? Not that much.

Yeah, I guess the things that make me nervous are accidentally sending emails to people, or deleting mp3s off of a CDN, or… There’s stuff that is recoverable, but a bunch of work, and embarrassing, and stuff. So that’s just standard – I think all developers have at least concerns in the back of their head as they’re doing stuff, and I’m just one of them.

That’s why a well-designed system really helps. For example CDN, you only upload. You don’t allow the keys to – the keys that you used to upload are not able to delete, for example. You have a separate set of keys. For this branch –

You’re just telling me what we should do, right?

Well, yes, exactly. [laughs] Very subtly. Very gently, sliding it into the conversation.

I’m taking notes… Yes, that’s really smart. [laughs]

Yeah, that’s interesting though that you have keys that unlock, but not keys to lock. That’s kinda like the concept. I have a key in my hand. If the door is unlocked, I can lock it. But I can’t unlock it with the same key.

And actually, we can’t delete mp3. We could overwrite them with other mp3s… But it is write-only in that sense, not delete. So that was just an example of things I think about.

The reason why I asked you about why you show that off, because I think the last - either last Kaizen or the Kaizen before, Jerod, you shared apprehension of Neon in the dev branches space… Because we contemplated latency, we contemplated the costs… And there’s a lot of things that I’ve learned since then. I had a great conversation, literally yesterday, with a fella named Bryan Clark, who is the VP of product at Neon. And so we were talking at length, and this will come to an ad spot near you, of course, but we like to do really cool ad spots for our content. But I learned a lot about this, that the database spins up in 500 milliseconds or less, so literally half a second… That if you walk away from your machine, you’re in development mode on your local branch, doing development on a branch, that the database spins down to zero after five minutes of inactivity. So there’s no long-term polling. So it spins down to zero.

The cost is basically nothing, because you’re in dev mode; unless you’re creating data… Then and only then would you pay more for space. So it’s really just time on the CPU, the diffing there, and then obviously the whole thing that Gerhard just showed off, which is why I asked this question, was this reset to parent, so that you can suck in more data, so that you can actually have prod-like or literally prod data in dev mode with just the slight cost of a little bit of latency. Because it’s literally not local, so there’s not going to be – it cannot compare to local.

Right.

[00:39:55.25] But then Brian did also divulge some potential future stuff that I’m gonna mention here, because it’s not an ad spot, and it’s cool. They’re going to work on the ability to go offline with your database, and still be Neon-like. Neon, locally dev branch, all the fun things you want, with Neon local dev branches, but go offline. Go on a plane, go on a train, whatever it might be. That’s future coming. I’m kind of like future-spelling for them, so don’t get too upset, Neon, please… But this is cool tech, they’re working on it, and they have a plan to continuously improve this developer workflow with a database… Which has not been there really, at large, ever… And it’s on our favorite database, Postgres. So that’s a beautiful thing.

Well, that leads me to what I asked Gerhard for last time. Didn’t I ask you for something like that as a gift?

Yes, you did. Fresh data… Something about booty… No, no, no. [laughter]

Yes, I do recall something about that…

So yeah. Okay. So yes, you did. I have something else for you, though…

Okay. It’s not ready. [unintelligible 00:40:53.16]

Is it called more booty, or double booty, or [unintelligible 00:40:55.10]

It’s called something – it’s called Pipe Dream.

Oh, it’s called a pipe dream.

Cool.

Let’s hear it.

So we’re not ready for that just yet. We still need to stay in the news redesign camp for a little bit longer…

Oh, we do. Okay.

…because I was going to share a couple of things that I think would make it better.

Okay.

So first of all, there’s obviously that player. We talked about that a fair bit. The second thing which I wish was there, and I don’t know where you stand on this, but just a very faint background on the sponsored segments. Right now they all like blur… And I think it would be very helpful to see what is sponsored and what isn’t without – basically, just like by skimming. Because it just puts me in the right mode, and it feels like it’s more honest that way. So sponsored sections, just like a fainter gray, or a faint whatever… Like, it has to be tiny. It’s not an ad, but it’s slightly different.

Also the shoutouts. I think if the shoutout section would be slightly different, it would make it easier to digest. Because now it’s not like one long newsletter. It has parts to it. I would quite like that.

What do you mean by the shoutouts? Are you talking about when I give Changelog++ shoutouts?

Quote of the week.

Oh, quote of the week.

Yeah. I think also like a different background color… Because those I think [unintelligible 00:42:12.27] And for example, I’ve seen quotes, I wanted to go back to them, and I didn’t quite remember where they were, so I had to basically start reading to get to “Oh, here’s the quote that I was looking for.”

Right.

So… Tiny things. Tiny things.

Yeah. I mean, I’m not against any of those things. Some of this is my desire to write the newsletter on my own terms, which is basically in raw Markdown, without any sort of additional tooling. And so the only like special formatting that we do is on the sponsored posts. That’s an h4, I think, when we give a thanks to 1Password for sponsoring Changelog News… And we just style that differently, literally because it’s like just the only h4 or h5, I can’t remember which one, that I actually use. And everything else is H2, and H3, and blockquotes… And it’s like literally just raw Markdown, which helps me actually author in a timely and enjoyable fashion. And so there’s no special – there’s no special anything. So there’s no way that – like, we don’t know semantically that a certain thing is sponsored, except for that it has that little h 5 or h4 under it. Now, we could probably do some fancy footwork and detect that, and like just based on the way I write it…

So I could get more formal on certain things, but I don’t do any classes, any tagging, anything. The quote of the week is literally me making an h3 that says “Quote of the week”, and then a blockquote. And then that’s it. Like, that’s what the quote of the week is. So in our system, there’s no fancy formatting, because it’s so free-flowing. And that allows me to be more, I think, creative and experimental with what I’m doing. Quote of the week is not always there, which is I think is funny. There’s not always a quote of the week, there’s not always a video of the week… So it’s not like I’m saying “I need a quote for this week”, and I plug it into the quote of the week block. So it’s more a result of how I write, versus an editorial decision.

[00:44:14.02] And we definitely used to have – the sponsored section I think was more called out than it is currently, and I think we could do some coding and just follow the same syntax that says “When you find a section that has this particular element in it, then change the style”, or something. So I’m not against it. It was just like we wanted it to be as free-flowing as possible, so that I could continually put them out every week.

Yeah.

Your perspective though, Gerhard… I understand that too, but do you feel – I mean, this is important. Do you feel duped? What is your feeling whenever you feel like these things blend? Because it’s not meant to be dubious, by any means… It’s just meant to blend a bit more. So like Jerod had said, it’s not something special, and then adding “Hey, thanks to 1Password for sponsoring Changelog News.” That was last week’s sponsor, I’m just reading that. And obviously in the audio it’s called out, right? It’s pretty clear in the audio… But it’s less – it blends more so with other content in the newsletter.

So, I would like to have the option of quickly skipping over a section if I so choose. It’s almost like “Okay, this is different. I know what this is. I can skip it, I can come back to it if I want to.” But this is something different from the news items. And this is a sponsored news item, and this will talk about most likely the benefits of that tool, or how it can help me, or things like that. And I would like, again, to have that option of skimming over it quicker, if I so choose.

Now, sometimes I’m in the mood of reading everything, and that’s okay, but other days I’m just skimming for something; I remember seeing something and I want to go back, and I have to do quite a bit of reading to get there. The headings help, but it all like blends together. And also the quote of the week, I think they’re good to shout out, and seeing that slightly different, because it just puts me like in another mindset. This is something that some person said, and maybe I’m interested in like digging deeper for this quote. Maybe there’s an episode, and there most likely is an episode, so it pulls me in that direction. And being honest about that almost like interaction, I would find it more pleasing. Maybe it is just a personal preference, honestly.

Yes, it’s good feedback. No, I mean, I definitely think that there’s times where you’re reading a newsletter, and there’s times when you’re scanning the newsletter. And I think that while it’s not difficult to see the things for sponsoring Changelog News with the – we’ve put the cash bag emoji. I mean, most of our actual sections and breaks is like horizontal rules and emoji, right?

Yeah.

The quote of the week always has the exact same emoji to start it out. So I try to be consistent with those things, because I know they are visual cues for people. I think that we could probably make the sponsored posts slightly more prominent without having to rework my entire publishing workflow.

Yeah, maybe always just putting the – for example, to read the headline, “Strong, unique passwords for you and your team.” That was the 1Password one from last week…

Mm-hm.

…maybe move the money bag up to be the emoji on the thing, versus the strong-arm thing, which I totally think is contextually correct. Maybe it’s always the money bag emoji before it, that way it’s like a visual cue. I don’t know. Gerhard, would that be enough for you to keep it simple?

Yeah, if the moneybag emoji is the sponsored news cue, that’s one less emoji for me to select as well. I’ve gotta come up with which emoji to pick.

That’s right.

I like it.

Because then it’s always leading with the money bag, and you can kind of spot the money bag, and the money bag, obviously… “Hey, this is sponsored” - and by the way, we do love 1Password - and that’s cool. I think that that’d be a good –

[00:47:44.03] Well, some people put like sponsored in parentheses in the title. We’ve considered doing that kind of stuff as well, and it just gets to be a little bit too noisy… But I think of it just like, the money bag – because contextually, the money bag doesn’t make sense until you know that “Oh, it’s sponsored.” And it’s always just the money bag.

I like that idea.

And you can just be like “Skip the money bag if I want to.”

That could work. Or look for the money bag… If there’s some money for me, let me check it out. [laughs]

There you go. Well, it’s funny, because – who was I speaking to years ago, that kind of changed my perspective a little bit on Google ads? …which - my perspective was skip the sponsored ones immediately. I do a Google search; I’m gonna skip right past the sponsored to the organic. Habitually, intuitively, easily. Give me the organic searches.

And somebody who was older and wiser than me, but maybe a little bit less technically savvy, but definitely more business savvy, was like “I go straight to the sponsored ones.” And I said, “Why?” And he goes “Because those people are serious about what they’re doing, and they want to reach me.” And I’m like “Well, that’s interesting.” So they’re legitimate, because they’ve got money to spend on this exact thing I’m looking for.

I do agree with that context, too.

And I was like “That’s actually interesting. Maybe I should click on those more often.” Because at first I was like “Yeah, they’re just trying to spend money to get here.” It’s like “Yeah, because they have a serious business that provides the exact thing that I’m looking for… So don’t skip the sponsor, look at him.” So maybe sometimes you look for the money bag, you know?

I think - and I’m biased - that if a company is sponsoring Changelog News, they are a company that is worth paying attention to. I think so. And so look for the moneybag. It’s a good point, Gerhard.

I had this exact challenge, I suppose, on a call this week. I was trying to showcase how we showcase people in the newsletter. And I’m scrolling it, trying to find the sponsored ones, and I couldn’t find, because it blended in so well. Then I was like “Hey, don’t think we’re trying to blend them in too much because I can’t find it, but they just blend in. It’s just natural.” But I think the money bag being in front of it could be the easy icon…

Totally.

…because I would have been looking for it. So it’s a win/win.

We still put the thank you underneath it, you know?

Yeah, exactly. Keep the same subheading.

We’re not trying to hide anything. Like, that’s not the point at all. It never was. These are sponsored. That’s what they’re there for. We just wanted to do it in a way that’s a) tasteful, and then b) allows me to write in a flowy manner, which is like throw in an h2 and an h4 and move on. But yeah, the moneybag for scanning – I mean, I do like the emoji for scanning; I use them a lot myself, so I think that that would be a positive improvement.

Quote of the week - not quite so sold on making that standout, because it’s honestly just a thing I made up and do sometimes. It’s not like a formal part. There are no real formal parts… Besides, there’s going to be five or six main stories, and then the rest I just make up each week. Like, is there three smaller stories? Is there a list at the end that’s shaped like a pyramid? You know, I just kind of roll with it, and I like that. But definitely good feedback.

Break: [00:50:45.16]

So I’d like to give a shoutout to Brandon Smith now. He’s not involved with Elixir. Pull requests 513. He is in the list of contributors. It’s a one-character change. He’s my type of people. His comment is way, way longer than the actual change, and I love it, because it has all the context.

Shout to – is it Brandon?

Brandon Smith, yes.

Brandon.

Brandon Smith.

Yeah, good job, Brandon. He basically was just removing a hash sign - or as some people call it, an octothorpe - from our show titles, or something. Help me understand.

Yup, it had to do with GitHub interpreting the hash sign and a number as a pull request for an issue. And obviously, we don’t want that in the context of GitHub. So what he did, he basically fixed that. But if you check out pull request 513, there is a lot of detail. And again, I’m shouting that thing out specifically because that was a very nice contribution. In terms of why he made the change, not necessarily the change, which is also great, so thank you for that, but it’s the detail around it. So more like that, please. I love them.

Yeah. Small change, but he made the Merge button very easy for us to push, because all I had to do is read all the context, and I was like “Looks good to me. I like it. Merge button.” Whereas if he had just opened a PR that just removed the octothorpe from our transcript titles or whatever that is, I’d be like “What’s the point of this?” I’d think he was just trying to get a Hacktoberfest T-shirt, or something.

In June.

Yeah. Well, it’s like Christmas in July. You know, Hacktoberfest in June.

Yup.

So I was launching a show on Wednesday, as we do, every Wednesday, and I logged into the admin, Jerod, and you know what I saw, right? I saw Changelog++, I saw all memberships, I saw all feeds in there… I think feeds has been in there for a bit, but Plus Plus is new… And we’re excited, because we’ve been wanting to bring from Supercast over into changelog.com proper our Plus Plus stuff… Because it is better, it’s not going anywhere, and we want to continue to Kaizen that as well. What’s going on there?

Right. Yes, we want to make Plus Plus even better, and one of the ways that we want to do that is by allowing our Plus Plus members to create custom feeds. So the number one request to anybody who signs up for Changelog.com/++ is “Why do I have to listen to all your episodes?”, which - you don’t have to listen to them all. But basically, the way that Supercast works, which is our platform that we’ve used since the initiation of this program to actually manage the entire process, is that we publish a single feed to them, and then they manage the individual people feeds that you subscribe to as a Plus Plus member. So we only get one to them. And so to them, we send over everything… Plus Plus versions, of course, which is the bonus content, extended episodes, ad-free. We send them those versions. And so as you subscribe, maybe you only listen to JS Party and you subscribed to changelog.com/++, and you’re like “Wait, now I’m gonna get all your shows.” And we just say “Yes, we’re sorry about that. You can just delete the ones that you don’t want.” Maybe your app has a custom filter, where you can just filter out, just so you’re getting JS Party… But like that’s literally – our hands are tied, because we don’t control that platform. This is how we do it.

[01:00:06.23] And we’ve wanted to fix that problem for a very long time, and at first I thought we had to just completely ditch SuperCast in order to do that, and then I started thinking how much work that would be, and how we want to eventually do that, and just have our own autonomy… But in the meantime, can we get halfway there as a stepping stone and give people what they want? And I determined “Sure we can.” All you need to know is who’s actually a Plus Plus subscriber and has an active account on our side of the fence. And then you allow them to have a feature and nobody else to have the feature.

So I started off with building the feature called Custom Fees, where you can set up, exactly what it’s called; log in, you can have more than one, you can pick which podcast is in it, you can pick your own album art, you can say how the titles are going to work, where it starts, etc. Just a few different things. And then you can subscribe to that. And I’ve been subscribed to one myself for the last six months, just to make sure it works every time… And it’s working. So that’s great.

We’re getting to where we’re ready to roll that out to some folks… And so most recently, the next step is “Well, how do we know who has a Plus Plus membership?” And so that became a Supercast integration, which was actually a Stripe integration, because the cool thing about Supercast, which was one of the reasons why we went with them in the first place, is they say “We do not hold your customer base hostage. It’s simply a Stripe integration. You own all your relationships; it’s in Stripe.” And so they’re managing a Stripe account of ours, and now we’re also not managing it, but we’re syncing a Stripe account of ours. So that allows us to then suck in all the membership information into our database, and so now we know natively inside of Changelog.com who is a Plus Plus member and who is not… Which means now all we have to do is build and enable the feed building form, the UI for building and managing your feeds, and Plus Plus people will be able to build their own custom feeds. And so you could have a Go Time feed that’s just Plus Plus for Go Time, you could have a JS Party feed, you could have “Pick my three favorite podcasts” of yours, or you can just say “Give me all of them.” You can do all the things. And it’s pretty cool. It works well. I’m excited to roll it out. We haven’t rolled it out yet because I just got the membership information into the system… This week, Adam? Was it this week?

This week, yeah.

Yeah, this week. So that’s been a lot of work behind the scenes.

When I first went there it was empty, so… Now it’s full. First time there it was empty. And I was like “Oh, this is in motion.” And then later that day you pinged me, you were like “Hey, I added some stuff.” I’m like “I saw that”, when I was shipping a show. That’s cool.

Yeah, there’s two parts to it. I mean, one is just a scheduled job that goes out, I think, every three hours, and just sucks in all subscription information from Stripe, and updates everything accordingly. And then the second thing is the WebHook system. So if you just sign up, we don’t want to have to wait three hours, or even an hour… I mean, it doesn’t take long., I could probably run it more often. It takes like 90 seconds to run, something like that. But I just set it up every three hours. And so the WebHook system - as soon as you subscribe, we know about it, and you can log in and actually manage that.

One thing that’s bugged me for years is you can sign into changelog.com as a Plus Plus subscriber and it still says “Sign up for Changelog Plus Plus”, because we just didn’t have any connection back. And so I’ve already changed that; it doesn’t say that to you if you already are subscribed. And that’s just bugged me for so long. So that’s gone.

But yeah, next up, and hopefully by next Kaizen, we will have custom feeds, the ability to create and manage them, and actually roll them out to our Plus Plus people… Even though you’ll still sign up and do everything through SuperCast, basically, for now.

And eventually we’ll have a page like SuperCast, that will be a tier. We’re still not sure; we’ve thought about several things. I think we spit-balled some ideas when we were in Seattle, about ways to do that…

[01:04:05.06] Yeah, basically like a fully autonomous Plus Plus, without Supercast, would be the end goal. And the fact that we can now integrate with Stripe directly… We’re halfway. We have a Stripe integration, it’s just not the full loop. And so eventually we’ll be able to close the loop. It requires us to build a bunch of pages, and emails, and… You know, many of you all listening are web devs, and so you know all the stuff you have to build… And so we have to build all that in order to completely jettison ourselves from SuperCast.

The emails… There’s a lot of different things that happen, like “Oh, your car didn’t bill.” The things that are all involved in managing subscriptions, essentially. We become a SaaS, but not really full-on SaaS. A lot of the mechanics that a SaaS –

It’s a PaaS. It’s podcast as a service. [laughs]

I like that. That is a great one.

We should write that down, so we can use it in our marketing copy.

Is it the PaaS out there already, or is it a different version?

No, that’s the one. Yeah.

I get it. I get it.

So yeah, custom feeds are there, and they’ll be coming to you, Plus Plus people, very soon, with a trademark next to the Very Soon.

Yeah. I’m excited about this specific feature myself, because whenever I – I do have a Changelog Plus Plus subscription, and whenever I want to listen to a specific episode because I’ve seen it on the website, I have to wait for Overcast to download the others before I can go to the one that I want. And usually, maybe the one – because there’s the last recent, I think, one, or last ten, maybe the one that I want to listen is further down below. So I was actually going to disable the automatic download. But if you’re saying that this is coming, this would be amazing. I can test it with Overcast specifically, to see if I can only see the episodes from my favorite shows. So that’ll be very nice. Amazing.

Yes. That’s exactly how it works. Especially – I mean, so many people sign up and they say “I just listen to Go Time”, or “I just listen to the Changelog. Why do I get all these?” And we just don’t have – now we finally can say “Yeah, go create yourself a custom feed, and you can get exactly what you want, and nothing else.”

That’s so cool. Super-cool. Speaking of things that we’ve talked about for ages, but haven’t done… I have one to share. I’m going to screen share now, and I’m going to walk through as if you are here… So we’re looking at pull request 516, Trace GitHub Actions Workflows.

Okay.

So what this does - whenever a workflow completes, the entire workflow trace gets sent to Honeycomb.

Okay.

So whenever GitHub Actions triggers, whatever happens in GitHub Actions, we can see that.

You can see step by step each thing.

So let me scroll down to see the picture… This is what it looks like. The image. All we see is like the duration; we have the Ship It workflow, and we can see whether it succeeded… And if we go a bit further down, we can see the various pipelines, because we have a bunch of pipelines within Ship It. [unintelligible 01:06:56.02] and some don’t always run. But then what’s even more exciting is that you can click on one run, and you can see details about how long different pipelines took, and what went in.

So this specific one we have, we’re looking at Dagger on Fly run. And by the way, these are the screenshots on the pull request; if you can open it up, you will see it. And it’s the blue one that’s highlighted. And we can see how long it took to set up the job, how long for example it took to connect to Fly, setting up the WireGuard tunnel… And then connecting to the Dagger engine, spinning it up, all of that. And we can compare these runs amongst themselves.

Okay.

So what I would like to do is go to the UI now, and click a little bit around. So we’re going to Boards… We are in Honeycomb and we have three boards configured. GitHub Actions is the last one. We have Fastly content stats, and Fastly service stats. We won’t be looking at those two, we’ll just be looking at the GitHub Actions. So we click on that, we see all the workflows that ran in the last seven days by default. We can change this to say “Show me the workflows for the last 14 days.” What do we see? What do we see, something that stands out just in these two graphs?

[01:08:15.02] Duration got higher on the 19th-ish…

Yes. 11 minutes. Exactly. Why did this workflow take 11 minutes? That’s exactly it. Things like “Why did this one, take only 2.4 minutes, versus this one which took 11?” What was going on here? So click on that. Boom, you get there. Boom, you click on that, you click view trace… And within a few clicks, you go to the actual workflow.

So we can see that there were some errors. We can see that this one took 11 minutes. Dagger on Kubernetes - that ran in 196 seconds. So that was good. But Dagger on Fly failed. And Dagger on Fly failed when it was actually running. 139 seconds; it was running for 139 seconds, then it failed. And then remember, always have two have everything. So then it fell back to GitHub, and that’s the one that succeeded in 438 seconds, and eventually, the deploy went out.

So this was – we can see it was Jerod, the one that committed, so it was his commit. And if we scroll down here on the right hand side, we will be able to see a link, and we have to click on the actual Ship It. Yeah, that’s the one. The actual run. GitHub Actions run. Go to, actually; that’s what we want. We want to go to, we open it up, Dagger on Fly, boom, we click on this… We see the build test setup. What happened? Gen server call, await tar ball, timeout. So this was the timeout pulling down a package on this run. And because we tried it again, in a different workflow, it succeeded. What do you think?

I think that’s very cool. I think that’s great for actually knowing why things are happening in GitHub Actions… Because I always wonder, “Why? Why is it not working?”

Cool. So while I was looking around the whole Fly world, I came across this; I came across the blog “World Page Speed Test”, by none other than Chris McCord, the creator of the Phoenix framework, which is what Changelog.com runs on. And again, he is our type of person, for sure. He tinkered for one afternoon, he took Elixir, Flame and Fly and introduced worldpagespeed.fly.dev. May 8th. So not that long ago. Very recent. And it’s a great read. But we’ll skip over that, because we’re not interested right now, in this context, in the details. We want to see the finished result. Worldpagespeed.fly.dev. What is the first website that we’re going to test?

Changelog.com.

Of course. That’s exactly what I did. That was the first website which I’ve tested. So what it does behind the scenes - it connects to a couple of data centers, Fly data centers, and it loads the website, the changelog.com website in a Chrome headless browser. So this is a full like DOM interactive sort of situation, and we are looking at Frankfurt, looking at Hong Kong, we’re looking at Chicago, Sydney, Tokyo, Sao Paulo, Santiago, and Mumbai. So it’s as if you would be running speed tests for a website from all these places. This is really cool.

So what do we see? Germany’s a bit slow…

Mm-hm. Hong Kong’s a little slow…

Just to, again, add more detail… Changelog.com, when it loads on this headless browser, that is running in Frankfurt, in a data center, a Fly data center, took 1,400 milliseconds to load. That’s a bit slow. So I’m wondering if we reset it…

Not for me…

So click Go again. Second time, it’s a bit quicker. And this shows you the caching.

[01:11:58.26] Oh, that’s why mine was slower then. Because I pulled up my own page and did the same thing, and mine was 494 milliseconds for Frankfurt. And yours was 1,400 and something the first time. Okay, that makes sense.

The first time – so now you can test caching. So this is more realistic. 575 milliseconds once the cache gets primed. Which cache? The Fastly cache, because this serves it from Fastly, so we can see that… So this is pretty cool, but I’m wondering, how does it compare when you go directly to Fly? So this should be really quick, because – well, kind of quick, because we are going to the Fly origin, which is running in the Fly data centers… But from these remote locations, they have to traverse the Fly network, and they have to end up in Virginia, where the Changelog application is running. So let’s try that. Let’s do fly.dev. So if we’re loading the website directly from the origin, this is what we can see. So it’s not that much slower. 648 milliseconds. Somewhat slower, but not that much slower. I thought this was interesting.

I like how you – this is like a subtle jab, this segment here. We won’t say why, but let’s just say it’s a subtle jab again.

Yup. So let’s keep going. Let’s keep going. So how does News Y Combinator compare? It’s slower, right? 850 milliseconds.

It’s way smaller, too.

It is. Is Changelog faster than Y Combinator?

It looks like it.

It looks like it is. So that’s good.

And they’re sending probably a [unintelligible 01:13:29.22]

[unintelligible 01:13:30.25]

I mean, that 52-kilobyte page… It’s tiny.

I know, right? So what’s going on there?

They’re still taking a while.

Yeah. I’m just going to go back and I’m going to reload it again.

Theirs is all text, so it [unintelligible 01:13:39.27]

I know, right? So yeah, 850 –

They must have a slow CDN, if there is one at all. Or maybe they just exist in one server somewhere.

Yeah, it looks like that.

Fastest in Chicago, Illinois. 315 milliseconds. So they probably have a US-based server. It’s probably not at all distributed around the world.

How does google.com compare? I have to check myself, right?

I mean, Google’s slow.

Google.com – it’s like, I don’t know what’s happening here, but yeah…

Maybe this is Chris’s tools having problems.

I don’t know. But –

Although – similar to Hyperping. I mean, that’s what I thought of when I first saw it, I was like “This looks like Gerhard’s Hyperping service, basically.”

Yeah.

But right there in the browser.

Yeah. So Sydney is especially slowing for Google, 2000 milliseconds.

I doubt that that’s true. Can they continue to make money with that being true?

I don’t know.

2.262 – that’s two full seconds To load google.com in Sydney, Australia? I mean, they’re surely losing money. So from a Fly node near Sydney. Right? Is that how it works?

Yeah. It runs like in a data center around Sydney, yeah.

That’s non SSL though. Does that matter? HTTPS vs HTTP…

It is. I’m saying –

It’s probably upgrading.

I’m saying HTTP – I mean, yeah, I can try HTTPS explicitly…

Well, I just noticed that it doesn’t do that when you just type in changelog.com, the domain. It’s not defaulting to secure.

Yeah. I know that we have a redirect, and…

Yeah, we do. They probably do as well.

It’d be cool to have this tool as traces back into Honeycomb, like you did. It’d be cool to pick a service like this, as opposed to pick the places that matter most to you. Not so much matter most, but ones you want to make sure “Okay, what are ups and downs in Australia? How frequently are we slower than we want to be?” And track that, similar to the way you do with GitHub Actions, and traces, and that whole flow, which - I didn’t chime in too much there, but I thought was just magical, how you can like stack-trace, essentially, where in the bottleneck of GitHub Actions a deploy fails. That to me is just like super-magical, to have that kind of visibility. But the same kind of visibility into PageSpeed, especially if we care so much about it, because I think we do.

Yeah.

Go back to that Google one, because there’s some interesting results on that Google one.

Yeah. Going back to the Google one.

So in Frankfurt, Germany, it took 800 milliseconds. That page was 1.4 megabytes. In Chicago, Illinois, it took 775 milliseconds, which is faster. The page is twice the size.

[01:16:13.12] Yeah, I don’t know what’s going on here.

What are they serving to us United States folks? Do they know we like it supersized?

Maybe… Tokyo, again, it’s 900. Tokyo is like the smallest one.

And then Brazil - it’s large again, but it’s faster. And then in Chile, it’s large and fast… They have different payloads based on where you are, to a large degree, in terms of data transfer.

I’m wondering if they’re like serving the cookie notification… You know that? Like when you load Google for the first time…

Oh, maybe.

…you have to accept to some cookies; maybe that has something to do with it. I don’t know.

I don’t know if that would double the size of the page, but yeah.

Yeah. But I’m wondering how does bunny.changelog fair.

Is this still a thing? You still have that?

It’s still a thing. Yeah, it’s still running.

Okay.

You can check it out.

A little faster.

Yeah. It’s 300 milliseconds, 400 milliseconds… Pretty good, I think. This is has been the fastest one so far. Cool. So Chris, we want to know more…

Why do you have to open up old wounds, Gerhard?

Well, you will understand in a second. We’re almost there.

Oh, my gosh…

Oh, gosh…

[unintelligible 01:17:13.24] Jerod.

It’s happening.

What’s happening…?

So I’m going to quote Jerod. And Jerod was saying, on the last Kaizen, “I like the idea of having this 20-lined varnish config, that we deploy around the world, and it’s like “Look at our CDN, guys.”

Pipe dream.

Yes, pipe dream.

You got there? Did you get there?

It’s so simple, and it can do exactly what we want it to do, and nothing more. But I understand that it’s a pipe dream. I’m still quoting Jerod, “I understand it’s a pipe dream… Because that Varnish config will be slightly longer than 20 lines, and we’d run into all sorts of issues that we end up sinking all kinds of time into.”

So by the way, I think I have a title for this show… “Not a pipe dream.

“Not a pipe dream.” I like that title.

So if you introduce pipedream.com, it’s not that. So if you make this mistake, that’s an honest mistake. Pipedream.com is a thing. There you go, pipedream.com is a thing. I didn’t even know.

Oh, wow. It’s actually like a developer service.

Yeah. Connect APIs, AI… There you go. I thought this was going to be the first AI of the show.

Well, they have to put that in the hero. This thing has almost 8,500 stars on GitHub. This is like a legit deal.

8,500. So yeah. It has a video, and all…

So this is not what you built.

No. This is not it. This is not it.

Alright. Well, then get off this page.

Okay.

Gosh…

So you just need to do pipedream.changelog.com.

Okay.

You can load it. You can try it. And I’m going to share something else while you load it, and while you look at it.

Okay.

Okay.

Very excited.

I’m glad. I kept the best for last. That’s what’s happening here. This is the grand finale.

That’s a good title, too. “The last for best.”

Cool.

Okay.

Can you see my terminal?

Yes.

Okay. Can you see that? What did I type?

ls.

Perfect. That’s exactly what I typed. Cool.

This is a weird quiz. [laughter]

A weird choice. So here’s a run script. It looks fancy. It’s an executable, and it can do a bunch of things. So what we’re going to do today - we’re going to run demo 2024.06.2021. That’s what we’re doing right now. Cool. And by the way, this is coming in the pull request. So by the time this is out, you should be able to see it.

So let’s keep going. And you may be wondering “Hey, this is not the first demo.” There was another one.

Back in January, it looks like.

Back in January. Yes.

When you guys ran into problems.

Yes, exactly. With James.

This is you and James Rosen?

[01:19:51.29] Yes. The video is out. “Let’s build a CDN.” You can check it out, the exact problem, what we ran into. So that was the first demo. Now we’re doing a better one. Cool. We hope. So let’s see. So we’re looking… We ran httpstat, which - I love it. It’s a tool built by Dave Cheney.

Yeah, Dave Cheney from the Go community.

That’s the one, yeah. Httpstat, the Go version, and it gives you a breakdown of the request to an HTTP endpoint. So we can see all the headers, and we can see how long the DNS lookup took, two milliseconds, the TCP connection took four milliseconds, the TLS handshake took eight milliseconds… Server processing - 573 milliseconds.t That took a while.

Okay.

The content transfer, 727. Total time for the pipe dream to load, 1,300. That’s a bit slow. So let’s keep going. So what was happening is Pipe Dream was only running in Sydney. And now we’re going to use the magic of Fly to scale it around the world. It will run in 16 regions.

Yes.

Yes. So you approve. I like this. I like where this is going.

I approve. I like this…

So let’s scale it –

To narrate, there was a prompt that asked him Yes/No, and he said Yes.

I said Yes. I’m ready. Bring it on.

And that’s what’s happening. Boom.

Executing. Done.

The machines are up. They’re running. And we have a question to answer. How many dollars do you think this is going to cost per month? There’s emojis, there’s everything. [laughter]

That’s gonna be a lot of dollars, I think… Do the big cash bag emoji.

Can you see that?

30 bucks.

30 bucks.

Not too bad. So this is just for the instances themselves, just to be clear.

Not for the bandwidth.

Yeah. Not for the bandwidth. We’re only using the memory, 256 megabytes of memory. But one instance per month cost $1.93. 16 around the world. $31 per month. Plus bandwidth, maybe taxes. I don’t know.

Right, right. Plus or minus 10 bucks.

So let’s do that again, and let’s see what happened. 369. And what do you see there? [unintelligible 01:22:09.03] miss. So this actually was served very close to where I am, but because it was a miss, it had to go – so this was a cache miss, and it had to go and fill it. We can see the TTL; this will be considered fresh for 60 seconds, and it’s going to be considered stale for 24 hours. That’s how this is configured. Cool. We keep going to the next one…

So what I’m using now is a tool - I love it - a CLI. I used Oha. It’s from the Cargo community. And it will run a latency test on endpoints. Right now we have it configured to send 30 requests for one client, spaced one second at a time. To see the latency distribution for this HTTP endpoint, to see how many succeed, how many fail. [unintelligible 01:22:57.18] it’s beautiful, there’s bars, there’s green, there’s yellow… You have to try it out. So what do we see? We see that 26 out of the 30 requests took 38 milliseconds.

Lovely.

Isn’t that nice?

The slowest one was 300 milliseconds.

300 milliseconds, yes. You will always have some outliers. And the fastest one’s also eight milliseconds. I think we can ignore them, because you always get one which is super-fast, and you always get one which is super-slow.

Sure.

Maybe there’s something with the tool, but the bulk of the requests, the majority of the requests took 40 milliseconds. And this is pipe dream.

This is not a pipe dream.

It’s not a pipe dream. I don’t think it is.

I’m beyond tickled. I want to see the code for this. I wanna see my 20 lines of Varnish.

We’re getting there. We’re getting there.

Okay.

[01:23:48.25] So the next thing, we have to compare this to Changelog.com. We’re doing the same measurement. We measured Pipe Dream, and we went to pipedream.changelog.com, to see the webpage, how it loads. So now we’re doing the same speed test. Same Oha, 30 requests, one client, one second apart, to Changelog.com, and in a few seconds we will get the result. How is this going so far?

This is going great.

It’s visually impressive.

Amazing. I was like, this brought me so much joy to put together. This was just like the best. 48 milliseconds. So Changelog.com, same test, same tool, same everything. 10 milliseconds slower than the Pipe Dream.

Than the Pipe Dream.

This is what you were asking for, right? It’s almost as if I could tell that you’re going to ask this. How many lines of Varnish config? That’s what we want to know, right?

That’s exactly what I want to know.

So we’re counting, we’re seeing 43… We keep going. 91, 117. However, there’s a lot of comments. A lot of to do’s… It’s a well commented config. 117 in total. How many lines of Varnish config without comments - now you get to guess.

91.

91. Jerod?

47.

46.

Wow, good guess.

46. If there were any prizes, you would get it. 46 lines of code. 46 for this whole thing. Cool.

That’s amazing.

So that was it. That was the end of it.

So that’s Pipe Dream. It’s a Varnish config that’s deployed to 16 Fly machines around the world.

Yup.

When you go to pipedream.changelog.com, it is a proxy for changelog.com.

It’s actually Varnish. It’s a Varnish configuration…

Yeah. It’s just that Varnish configuration, just running around the world.

That’s it.

When do we ship it?

Exactly.

And it got down to - what was the milliseconds? In the sub-100s, right?

38. You know what? If only there was PageSpeed. let’s try it, okay? So I’m going to share another screen. Back to PageSpeed. Let’s see how it loads. How does it compare?

World PageSpeed? Yeah.

So how does it compare? We see [unintelligible 01:26:00.18] here, so we’re going to open another one… Let’s google pipedream.changelog.com. HTTPS. We’re going to HTTPS, and it’ll redirect nothing. Let’s see. 700, 500, 436… This wasn’t cached in some regions, so let’s reset and let’s try it again. Let’s try the pipe dream again. What do we see? 400 milliseconds. 500. 300. 300. 457… If only, if only we could see this over time, right? So let’s see. What do we see here? So we see – again, Hyperping…

Well, you do have it over time.

24 hours, last 24 hours, from a bunch of locations… Average response time, 140 milliseconds.

I love it. Let’s ship it. Let’s ship the pipe dream.

[laughs] So hang on… We keep going. Changelog.com is 312 milliseconds. Same test, same everything. So twice as fast as changelog.com, worldwide, as measured by a tool that was made for this. So before we ship it, before we do that –

Well, there is other challenges, right? Because like there’s still bandwidth costs.

Yeah, there’s other things you have to do.

Oh, yes.

I’m just – I’m being excited.

Yeah. This is amazing.

That’s a very basic Varnish config, right? I’m assuming that’s not handling any of our specific rewrites and needs and all that kind of stuff that we have in Fastly right now.

Correct. Yes. That’s correct. Also, we’re not serving any static content. So that’s one. But maybe we can leave that for later. Maybe we can just serve the website. We’re not sending any logs to Honeycomb, and we do want that configuration… There’s one thing which we still have to figure out… For example, when an application gets redeployed, for some reason the backend, as it’s called in Varnish, remains sick. So this is a Varnish term, which is the backend, which is the application. When there’s a redeploy, or when the application shuts down and restarts, the backend remains sick. I’m not sure why it’s happening, but obviously, we need to fix that. There is no feed proxying… There’s a bunch of things like that. And if you want to purge anything, we still have to have a system that purges across all these Varnish nodes. So how do you – because you can’t just purge one. So the question is, do we want to continue with this pipe dream?

I would like to…

[01:28:12.03] Adam? What do you think?

I’m torn. I don’t know. I think it’s – we have to weigh it against the whole “Shall we buy it or should we build it?” That’s the question, really, here. We’re saying we should build it, because we can customize it to our specific needs, but then we have to maintain it. And then we have to find a way to pay for it, which may or may not be something that we can get given to us, so to speak, as part of a partnership, which is how we’ve done it before… So there’s definitely some – in dreamland, I think, as a pipe dream officially, it does make sense to pursue, because it’s the itch; it’s the scratching our own itch. Back to the really simple CDN idea we came up with four Kaizens ago… I don’t know, that’s kind of what this is, is can there be a version of this that doesn’t have to be - not so much that the Cloudflare, CloudFronts, or Fastlies, or Bunnies, or anybody else out there is a bad or a good thing… But like if you can build it yourself – I think that’s the question; like, if you can build it yourself and maintain it yourself, and fine-tune it, and it’s worth it to you, is there a way you can go your own route? And I think that’s where a really simple CDN comes into play, where that’s really what we want. We don’t want all those bells and whistles. I think the purging thing is probably the most pertinent thing, because we do purge frequently enough when we have misses or issues or whatever. That’s a frequent feature. Otherwise, it’s what nodes and regions matter to us. Can we monitor speed and fastness, and can we fine-tune the system we’ve built enough to eke out every millisecond possible? And I think the answer is yes, but the question for me is how do we pay for it? If we can figure out how we pay for it, and make it not so much profitable, but make it something that bakes into our core story, like we have done in the past, to me I think that’s a win/win. And I prefer triple wins. Win/win/win.

[01:30:04.19]

We win/win/win. We all win.

The answer, when it comes to paying for it, is that this would be within the Fly sponsorship. We wouldn’t exceed it.

You don’t think so?

If we don’t handle the static content. Because that’s where the bulk of the bandwidth is. Even if we handle the static content, even if we serve the static content through this, even in that case, we would maybe exceed it by a little bit, but we would still be like – we’d be right there at the limit. And this limit has been in place for a couple of years now. So maybe that’s a Fly.io conversation. But it’s not twice as much. We’re not talking twice as much. We’re talking maybe 50% more, maybe.

Yeah… I think you said before we do like 20 terabytes of bandwidth a year. Is that right?

Maybe. I can’t remember. I need to check the numbers. Maybe.

Yeah, thinking here on the fly on the podcast, it’s like podcasts as negotiation; like, we just – we need to revisit the Fly conversation, basically, and see if this is interesting to them. Because I think there’s two sides of this coin from my perspective. I guess three. One, it’s super-cool. BA. This is awesome. That’s my abbreviated version of bad-something-else, to keep our show clean. Then two, I think, should we build it, and should we maintain it - what is the maintenance burden we’re taking on to build and deploy it? And is it literally just enough in maybe a couple iterations to be exactly what we want?

So should we build it, should we maintain it? That’s one question. Then the other question is can we have a partner love what we do enough to sponsor it or to bring it into the fold that we’ve already got? It seems like the first one’s a yes, because it is awesome. And the third one’s a possibility, because it’s within the range… And I think the middle one is probably one to maybe spitball right here, which is - you’ve built most of it. Is this 80% of it? Is there 20 more percent, and is that the hard part? Like, how close to a version of done is this?

[01:32:07.26] To be honest, it feels like we’re 50% there. It took me some number of hours. We’re not even talking days, we’re talking hours of effort so far. So it was nothing massive. It didn’t take me like days and days of like wrestling with this. There are a couple of challenges. I am curious about this myself, because I think it’s really cool to be able to build your own CDN on top of Fly. Kurt blogged about this years ago, and he was using Nginx in that context.

I think we would be fulfilling - not the prophecy, but the story, which is we’re using Fly the way it was meant to be used. We’re using Tigris as well. R2 migration? Maybe. Who knows? I don’t know. I’m just dropping it as an idea. But if we use Tigris for the static content, and if you use this for the dynamic content, we may have a winning proposition. The only unknown is the logs. So that’s the one that – but now we have full control over that. We’re not like “Maybe we can do it.” No, no. We can definitely do it. It’s just a matter of spending some hours to figure it out.

Well, I think that, as I said last Kaizen, there is a halfway there, similar to my halfway off Supercast move, which is we leave cdn.changelog.com alone, which has the bulk of the bandwidth, and has the logging issues, and has all of that stuff, and we just focus in on changelog.com and the feeds, and just a straight up content cache for the dynamic pages, with purging, and we see if we can tackle that. If you can tackle that – because we need purging on feeds, for sure. If you can tackle that, you’re halfway there, and you’re way further than you are now. And once we get this into a place where it’s like it’s a Varnish config that I can author and test and try out, then I think that it’s a lot more something that even I could take the last mile… Because I can figure out some of that stuff.

The purging is definitely an issue. Like, do we have to have some sort of an orchestrator or something that you notify “Hey, it’s time to purge”, and then that thing trickles down to all these little Varnish caches? …versus the app knowing all the Varnish caches. There’s some sort of middle piece of layer in there that we have to figure out, but… The rules around the feeds and all that kind of stuff, if we’d just focus in on that and leave the cdn.changelog.com alone,

then we learn a lot more, we build something that’s continually cool, and in the meantime we can figure out your other questions, Adam.

Do you guys agree those are the three kind of pillars that are hinging upon? Like, one, it’s awesome, two, should we build it, should we maintain it? And three, can it be – what’s that called? Not endorsed, but provided by, courtesy by, kind of thing, whenever a brand gives you something and you do something with it, and it’s not really –

Can’t afford it… Yeah.

The way I think of this is simple. And we come back full circle to where we started. Fastly is a bidet. How do you pronounce it, again? [laughter]

Bidet.

There you go.

A bidet.

It squirts, it remembers, it does the water, the whole thing.

Okay, keep going. I’m tracking this analogy.

Pipe Dream is a wet wipe. [laughter]

Oh, gosh…

It’s simple, it’s practical… Take it anywhere, same experience.

This has gone too far. [laughter]

And there’s one more step.

There’s one more step… He’s gone one step further…

No CDN - singles-ply.

Oh, my goodness…

We’re off the rails now.

That took me all day.

That took you all day. That took him longer than it took him to make Pipe Dream, was to figure out that analogy.

Exactly. It did.

That’s hilarious.

So I’ve got a proposition here then. I think this is an endpoint, to some degree. I love your [unintelligible 01:35:52.20] by the way. This is a good laugh. But I do have one more question that I think might make more sense potentially for Plus Plus. So is this roughly where we want to end? Because if so, then I have a potential five-minute, eight-minute Plus Plus.

I think this was definitely the crescendo.

Okay. Well, let’s pause a second and say that demo that you did was definitely like the Fourth of July here in the United States whenever it’s the finale scenario. The skies were ablazing, and it was thunderous… For those who didn’t see it, we’ll throw some screenshots. Kara, pull some screenshots if you can, of that, so we can throw them in the shownotes… Because I think it was super-cool. It’s worth seeing. And I think that, you know, for the Plus Plus folks who are subscribed - boom, there you go. You’re getting a little bonus here in a minute or two. For those who are missing out, you’re just missing out. Go to changelog.com/++. Or changelog.com/plusplus. Right? Don’t both redirect, Jerod?

They both work. You can spell it out, you can use the characters…

Either way. And get there. It is better. It’s better! 10 bucks a month, 100 bucks a year, and see what we’re gonna do with this CDN, this saga. I mean, it never ends. There’s subtle jabs in the middle of shows, there’s finales at the end, there’s pipe dreams… I mean, what more do you come here to Changelog & Friends Kaizen edition for? I mean, this is what you come for, right? That being said… We done?

Yes. Kaizen!

I had so much fun. Three years in the making… It’s only getting better.

Only getting better.

Yeah, man.

I love it. I love it.

Good stuff.

Bye, friends.