Gerhard joins us for the 12th Kaizen and this time talk about what we DIDN’T do. We were holding S3 wrong, we put some cash back in our pockets, we enabled HTTP/3, Brotli compression, and Fastly websockets, we improved our SLOs, we improved Changelog Nightly, and we’re going to KubeCon 2023 in Chicago.
Gerhard joins us for the 11th Kaizen and this one might contain the most improvements ever. We’re on Fly Apps V2, we’ve moved from S3 to R2 & we have a status page now, just to name a few.
Gerhard is back! Today we continue our Kaizen tradition by getting together (for the 10th time) with one of our oldest friends to talk all about the continuous improvements we’re making to Changelog’s platform and podcasts.
This is our 9th Kaizen with Adam & Jerod. We start today’s conversation with the most important thing: embracing change. For Gerhard, this means putting Ship It on hold after this episode. It also means making more time to experiment, maybe try a few of those small bets that we recently talked about with Daniel. Kaizen will continue, we are thinking on the Changelog. Stick around to hear the rest.
For our last 2022 Kaizen episode, we went all out:
- 💪 @jerod outdid himself in the number of improvements shipped between Kaizens
- 🕺 A few of our listeners contributed → prompted us to create a new contributing guide
- 🗺 We now have a new infrastructure diagram
All of this, and a whole lot more, is captured as GitHub discussion 🐙 changelog.com#433. If you want to see everything that we improved, that is a great companion to this episode.
In today’s Kaizen episode, we talk about shipping Adam’s Christmas present: chapter support for all Changelog episodes that we now publish. This feature was hard because there are many subtle differences in how the ID3 spec is implemented. Of course, once the PR shipped, there were other issues to solve, including an upgrade the world kind of scenario. Since Lars Wikman did all the heavy ID3 lifting, he joins us in this episode.
In our 6th Kaizen, we talk with Jerod about all the things that we cleaned up after migrating changelog.com from a managed Kubernetes to Fly.io. We deleted the K8s cluster and moved wildcard cert management to Fastly & all our vanity domain certs to Fly.io. We migrated the Docker Engine that our GitHub Actions is using - PR #416 has all the details. We did a few other things in preparation for our secrets plan. Thank you Maikel Vlasman, James Harr, Adrian Mester, Omri Gabay & Owen Valentine for kicking it off in our Slack #shipit channel.
Gerhard’s favourite improvement: the new shipit.show domain.
This is our 5th Kaizen where we talk about the next improvement to changelog.com: we are now running on Fly.io and our PostgreSQL is managed. This is a migration that many were curious about, including Simmy de Klerk, the person that requested this episode.
We want to emphasise the type of partner relationships that we seek at Changelog & why they are important to us, as well as to our listeners. Honeycomb & Fly embody the principles that we care about, and Gerhard thinks that we are currently missing a Kubernetes partner.
We finally did it! All our static files are served from AWS S3. This is the most significant improvement to our app’s architecture in years, and now we have unlocked the next level: multi-cloud. We talk about that at length, and how it fits in our 2022 setup. The TL;DR is that changelog.com will fly, both literally and figuratively.
We also address Steve’s comment that he left on our previous Kaizen episode - thanks Steve!
Towards the end, we talk about Gerhard’s new beginnings at Dagger, where he gets to work with a world-class team and build the next-gen CI/CD. That’s right, Gerhard is now walking the Ship It talk all day, every day. If you want to watch him code live, you can do so every Thursday, in our weekly community session.
This is our third Kaizen episode in which Adam, Jerod & Gerhard talk about GitOps the wrong way, ask questions with Honeycomb and realise that they must be holding the CDN wrong, and the effort that has been going into moving all changelog.com static files from regular volumes to an S3-like object store. If you like a good yak shake, listening to this one is a lot more fun than doing it.
Gerhard is most excited about the Ship It Christmas gifts that we have been preparing for you. While GitHub Codespaces is not going to be part of the upcoming Christmas special episode, today’s talk covers why investing in a Codespaces integration is worth it.
Changelog #459 and Backstage #20 are related to this topic.
This is our second Kaizen episode, where Adam, Jerod & Gerhard talk about changelog.com improvements since episode 10. OK, so Gerhard deleted the DNS API token. Not only did he take the time to understand how that happened, so that he could actually learn from his mistake, but now we have a system in place so that we can share learnings from incidents. By the way, these are publicly available in our #incidents Slack channel.
A great & unexpected thing that happened since we recorded this episode, is Jerod fixing 99% of all the errors that were happening in prod. The top error was the broken Twitter auth - sorry Matt - which was a result of us upgrading to OTP 24 a few months back. Episode 3 show notes include a YouTube stream which captures it all.
We wrap up this episode by each of us sharing the improvements that we would like to do until our next Kaizen. You heard it from Adam first: Ship It Driven Development
This week we’re sharing a special episode of our new podcast called Ship It. This episode is our Kaizen-style episode where we point our lens inward to Changelog.com to see what we should improve next. The plan is do this episode style every 10 episodes.
Gerhard, Adam, and Jerod talk about the things that we want to improve in our setup over the next few months. We talk about how the June Fastly outage affected changelog.com, how we responded that day, and what we could do better. We discuss multi-cloud, multi-CDN, and the next sensible and obvious improvements for our app.
Kaizen means “change for the better”, continuous improvement in this context. Failure is essential to learning, but how do we learn as a team? The simplest thing is to regularly dedicate time for taking a step back, talking about what works & what doesn’t, maybe writing some of it down, and eventually deciding what we should improve next. I intend to make every 10th Ship It! episode a Kaizen one.
This is the first one when we talk with Adam and Jerod about the things that we want to improve in our setup over the next few months. We talk about how the June Fastly outage affected changelog.com, how we responded that day, and what we could do better. We discuss multi-cloud, multi-CDN, and the next sensible and obvious improvements for our app. Let us know via Slack or Twitter what learnings are valuable to you so that we can produce the best content for you.