This is our 9th Kaizen with Adam & Jerod. We start today’s conversation with the most important thing: embracing change. For Gerhard, this means putting Ship It on hold after this episode. It also means making more time to experiment, maybe try a few of those small bets that we recently talked about with Daniel. Kaizen will continue, we are thinking on the Changelog. Stick around to hear the rest.
The rough math goes like this: We spent $3.2m on cloud in 2022. Just under a million of that was on storing 8 petabytes of files in S3, fully replicated across several regions. So that leaves ~$2.3m on everything else: app servers, cache servers, database servers, search servers, the works. That’s the part of the budget we intend to bring to zero in 2023. Then we’ll worry about exiting the 8PB from S3 in 2024.
David spoke at length on this decision when he was on Ship It! late last year.
Tim McNamara is known as New Zealand’s Rust guy. He is the author of Rust in Action, and also a Senior Software Engineer at AWS, where he helps other builders with all things Rust.
The main reason why Gerhard is intrigued by Rust is the incredible resource frugality. Fewer CPUs means less energy used, which is good for the planet, and good for the monthly bill. This becomes most noticeable at Amazon’s scale, when S3, Lambda, CloudFront and other services start adding Rust components.
Lars is big on Elixir. Think apps that scale really well, tend to be monolithic, and have one of the most mature deployment models: self-contained releases & built-in hot code reloading. In episode 7, Gerhard talked to Lars about “Why Kubernetes”. There is a follow-up YouTube stream that showed how to automate deploys for an Elixir app using K3s & ArgoCD.
More than a year later, how does Lars think about running applications in production? What does simple & straightforward mean to him? Gerhard’s favourite: what is “human scale deployments”?
Welcome to 2023! A new year is the perfect time to start with a fresh perspective. Given a few bare metal hosts with fast, local storage, how would you run your workloads on them? Would you cluster them for redundancy? What operating system would you choose?
Steve Francis, CEO at Sidero Labs and Andrew Rynhard, CTO at Sidero Labs join us today to talk about running Talos Linux on bare metal.
Nishant Roy, Engineering Manager at Pinterest Ads, joins Johnny & Jon to detail how they’ve managed to continue shipping quality software from startup through hypergrowth all the way to IPO. Prepare to learn a lot about Pinterest’s integration and deployment pipeline, observability stack, Go-based services and more.
I recently discovered an awesome tool called Coolify that calls itself an open-source Netlify or Heroku alternative that can be self-hosted. Gave it a whirl for my own personal projects and loved it, so I thought I’d share.
In your company, who designs the end-to-end developer experience? From design to implementation, what is the developer experience that you actually ship? Even though the average developer wastes almost half of their working hours because of bad DX, many of us don’t even know what that means, or how to improve it.
Kenneth Auchenberg is working at Stripe, building economic infrastructure for the internet. Gerhard found his perspective on Developer Experience Infrastructure (DXI) refreshingly simple, as well as very useful.
This week on The Changelog we’re joined by Adam Wiggins, co-founder and former CTO of Heroku, for an exclusive trip down Heroku memory lane. Adam and Jerod are both tremendous fans of Heroku and believe (to this day) they represent the apex in developer experience for delivering code to production.
We talk through the beginnings of Heroku, the v1 most people have forgotten about, the era of web hosting back in 2008-2010, the serendipity of Silicon Vally in those days, pitching to Y Combinator, the makings of git push heroku, the Heroku style and name, the sale of Heroku to Salesforce, potential regrets — and we tee up part 2 coming next week with Adam going beyond Heroku and the story of Muse.
15 years ago, Gerhard discovered magic in the form of Ruby on Rails. It was intuitive and it just worked. That is the context in which Gerhard fell in love with infrastructure and operations.
Today, for special episode 77, we start at Seven Shipping Principles, and, in the true spirit of Ship It, we’ll see what happens next.
Our guest is David Heinemeier Hansson, creator of Ruby on Rails, co-founder of Basecamp & HEY, and a lot more - check out dhh.dk.
Few genuinely need a multi-cloud setup. There is plenty of advice out there which mostly boils down to don’t do it, you will be worse off. Vex.dev is a startup that provides APIs for video and audio streaming. The hard part is real-time combined with massive scale - think hundreds of thousands of concurrent connections. They achieve this by using a combination of Fly.io, AWS and GCP. Jason Carter, founder of Vex Communications, is joining us today to talk about the multi-cloud setup that vex.dev runs.
This is awesome — Brandon, please write down more lessons learned, even if they’re just something for you to reference later on.
I’ve been doing this “reliability” stuff for a little while now (~5 years), at companies ranging from about 20 developers to over 2,000. I’ve always cared primarily about the software elements I describe as living “outside” the application – like, how does it get its configuration? What kinds of instances does it run on, and are those the best kinds to use? What steps does it take on its path from “code in a repository” to “running in production”? And I’ve always kept track of what I liked – which mechanisms allowed fast iteration and which caused frustration, which led to outages and which prevented them.
… So! With that out of the way – this is how I’d rebuild it all from scratch if I could.
Matias Pan is a Staff Software Engineer at Lemon Cash, a crypto startup based in Argentina. Lemon infrastructure runs digital wallets & physical cards, which technically makes them a bank. How does Matias & his team think about enabling developers get code from their workstations into production? Remember, we are talking about a bank - a bad deploy is a big deal. And when a bad database migration goes out, what happens then?
One of our listeners, Andrew Welker, suggested that we talk about Klustered, so a few hours before David Flanagan was about to do his workshop at Container Days, we recorded this episode. We talked about all the weird and wonderful Kubernetes debugging sessions on Klustered, a YouTube playlist with 43 videos and counting.
We then talked about Rawkode Academy, and we finished with conferences. Good thing we did, because David almost forgot about KubeHuddle, the conference that he is co-organising next week. Gerhard is looking forward to talking at it! No, seriously, check it out at kubehuddle.com.
In today’s Kaizen episode, we talk about shipping Adam’s Christmas present: chapter support for all Changelog episodes that we now publish. This feature was hard because there are many subtle differences in how the ID3 spec is implemented. Of course, once the PR shipped, there were other issues to solve, including an upgrade the world kind of scenario. Since Lars Wikman did all the heavy ID3 lifting, he joins us in this episode.
Maybe it’s the Californian sun. Or perhaps it’s the time spent at Disney Studios, the home of the best stories. One thing is for sure: Taylor Dolezal is one of the happiest cloud native people that Gerhard knows.
As a former Lead SRE for Disney Studios, Taylor has significant hands-on experience running cloud native technologies in a large company. After a few years as a HashiCorp Developer Advocate, Taylor is now Head of End User Ecosystem at CNCF. In his current role, he is helping enable cloud native success for end-users like Boeing, Mercedes Benz & many others.
Most of you already know what it’s like to work in a startup or a small company. A few of you have been asking us for conversations with engineers that work for big companies, the kind that run everything from big title games to banking, and even critical national infrastructure.
In today’s episode, we talk to Ganeshkumar, a Software Engineer in the Azure Kubernetes Service team, who works on Node Lifecycle and Kubernetes Versioning, and Brendan, Kubernetes project co-founder and engineering Corporate Vice President of Microsoft Azure OSS and Cloud-native Compute. We talk about what it’s like to work for Microsoft, how mentoring works in practice, and what Kubernetes, Omega, & Borg have to do with it all.
It is a very good thing for Salesforce to be more focused on Heroku’s future, but the glaring detail shared by Bob Wise today is the era of free on Heroku is over.
Here’s what they announced:
A few weeks ago, Jerod spoke with Liz Rice about the power of eBPF on The Changelog. Today, we have the pleasure of both Liz Rice, Chief Open Source Office at Isovalent & Thomas Graf, CTO & co-founder at Isovalent, the creators of Cilium.
Around 2014, Facebook achieved a 10x performance improvement by replacing their traditional load balancers with eBPF. In 2017, every single packet that went to Facebook was processed by eBPF. Nowadays, every Android phone is using it. Truth be told, if it’s network-related and it matters, eBPF is most likely a part of it.
Gerhard’s transition to a senior engineer started 10 years ago, when he embraced the vim mindset, functional core & imperative shell, and was inspired to seek simplicity in his code & infrastructure. Most of it can be traced back to one person: Gary Bernhardt, the creator of Execute Program, Destroy all Software and the now famous Wat idea.
Few stick around long enough to understand the long-term impact of their decisions on production systems. Even fewer are able to talk about them as well as Gary does.
Today we talk with two lovely folks from Transistor.fm: Jason Pearl, Senior Software Developer & Jon Buda, co-founder. Gerhard was curious to find out about their setup & how did it change with the launch of the new podcast website builder. After all, you have been hearing us talk about our setup for years, so it was high-time to challenge some assumptions and learn how another team is solving similar problems.
TL;DL: keeping it simple is at the root of smooth operations & stable systems.
In our 6th Kaizen, we talk with Jerod about all the things that we cleaned up after migrating changelog.com from a managed Kubernetes to Fly.io. We deleted the K8s cluster and moved wildcard cert management to Fastly & all our vanity domain certs to Fly.io. We migrated the Docker Engine that our GitHub Actions is using - PR #416 has all the details. We did a few other things in preparation for our secrets plan. Thank you Maikel Vlasman, James Harr, Adrian Mester, Omri Gabay & Owen Valentine for kicking it off in our Slack #shipit channel.
Gerhard’s favourite improvement: the new shipit.show domain.
Ben Johnson, the creator of Litestream, joined Fly.io a few weeks after we migrated changelog.com - episode 50 has all the details. That was pure coincidence. What was not a coincidence, is Gerhard jumping at the opportunity to talk to Ben about Postgres vs SQLite with Litestream.
The prospect of running a cluster of our app instances spread across all regions, with local SQLite & Litestream replication, is mind boggling. Let’s find out from Ben what will it take to get there. Thanks Kürt for kicking off this dream.
Rob Barnes (a.k.a. Devops Rob) and Rosemary Wang (author of Infrastructure as Code - Patterns & Practices) are joining us today to talk about infrastructure secrets.
What do Rosemary and Rob think about committing encrypted secrets into a repository? How do they suggest that we improve on storing secrets in LastPass? And if we were to choose HashiCorp Vault, what do we need to know?
Thank you Thomas Eckert for the intro. Thank you Nabeel Sulieman (ep. 46) & Kelsey Hightower (ep. 44) for your gentle nudges towards improving our infra secrets management.
Today we are talking how to optimise sociotechnical systems with Ben Ford, founder & CEO of Mission Control. The correct order is: people, process & technology. The tools are important, and we talk about specific ones in the second half of this episode, but there are rules and principles that govern how people interact, and we need to start there.
