Infrastructure Icon

Infrastructure

27 Stories
All Topics

Ship It! Ship It! #70

Kaizen! Four PRs, one big feature

In today’s Kaizen episode, we talk about shipping Adam’s Christmas present: chapter support for all Changelog episodes that we now publish. This feature was hard because there are many subtle differences in how the ID3 spec is implemented. Of course, once the PR shipped, there were other issues to solve, including an upgrade the world kind of scenario. Since Lars Wikman did all the heavy ID3 lifting, he is here with us too.

Ship It! Ship It! #69

The cloud native ecosystem

Maybe it’s the Californian sun. Or perhaps it’s the time spent at Disney Studios, the home of the best stories. One thing is for sure: Taylor Dolezal is one of the happiest cloud native people that Gerhard knows.

As a former Lead SRE for Disney Studios, Taylor has significant hands-on experience running cloud native technologies in a large company. After a few years as a HashiCorp Developer Advocate, Taylor is now Head of End User Ecosystem at CNCF. In his current role, he is helping enable cloud native success for end-users like Boeing, Mercedes Benz & many others.

Ship It! Ship It! #68

Behind the scenes at Microsoft Azure

Most of you already know what it’s like to work in a startup or a small company. A few of you have been asking us for conversations with engineers that work for big companies, the kind that run everything from big title games to banking, and even critical national infrastructure.

In today’s episode, we talk to Ganeshkumar, a Software Engineer in the Azure Kubernetes Service team, who works on Node Lifecycle and Kubernetes Versioning, and Brendan, Kubernetes project co-founder and engineering Corporate Vice President of Microsoft Azure OSS and Cloud-native Compute. We talk about what it’s like to work for Microsoft, how mentoring works in practice, and what Kubernetes, Omega, & Borg have to do with it all.

Bob Wise blog.heroku.com

Heroku's "next chapter" doesn't include free plans

It is a very good thing for Salesforce to be more focused on Heroku’s future, but the glaring detail shared by Bob Wise today is the era of free on Heroku is over.

Here’s what they announced:

  • They launched an interactive product roadmap on GitHub
  • They are focusing on mission critical and will discontinue free product plans and delete inactive accounts
  • They are starting a program to support students and nonprofits
  • They will continue to contribute to open source projects, notably Cloud Native Buildpacks and offering Heroku credits to select open source projects through Salesforce’s Open Source Program Office (OSPO)

Ship It! Ship It! #67

All your network are belong to eBPF

A few weeks ago, Jerod spoke with Liz Rice about the power of eBPF on The Changelog. Today, we have the pleasure of both Liz Rice, Chief Open Source Office at Isovalent & Thomas Graf, CTO & co-founder at Isovalent, the creators of Cilium.

Around 2014, Facebook achieved a 10x performance improvement by replacing their traditional load balancers with eBPF. In 2017, every single packet that went to Facebook was processed by eBPF. Nowadays, every Android phone is using it. Truth be told, if it’s network-related and it matters, eBPF is most likely a part of it.

Ship It! Ship It! #62

Operational simplicity is a gift to you

Gerhard’s transition to a senior engineer started 10 years ago, when he embraced the vim mindset, functional core & imperative shell, and was inspired to seek simplicity in his code & infrastructure. Most of it can be traced back to one person: Gary Bernhardt, the creator of Execute Program, Destroy all Software and the now famous Wat idea.

Few stick around long enough to understand the long-term impact of their decisions on production systems. Even fewer are able to talk about them as well as Gary does.

Ship It! Ship It! #61

The ops & infra behind Transistor.fm

Today we talk with two lovely folks from Transistor.fm: Jason Pearl, Senior Software Developer & Jon Buda, co-founder. Gerhard was curious to find out about their setup & how did it change with the launch of the new podcast website builder. After all, you have been hearing us talk about our setup for years, so it was high-time to challenge some assumptions and learn how another team is solving similar problems.

TL;DL: keeping it simple is at the root of smooth operations & stable systems.

Ship It! Ship It! #60

Kaizen! Post-migration cleanup

In our 6th Kaizen, we talk with Jerod about all the things that we cleaned up after migrating changelog.com from a managed Kubernetes to Fly.io. We deleted the K8s cluster and moved wildcard cert management to Fastly & all our vanity domain certs to Fly.io. We migrated the Docker Engine that our GitHub Actions is using - PR #416 has all the details. We did a few other things in preparation for our secrets plan. Thank you Maikel Vlasman, James Harr, Adrian Mester, Omri Gabay & Owen Valentine for kicking it off in our Slack #shipit channel.

Gerhard’s favourite improvement: the new shipit.show domain.

Ship It! Ship It! #59

Postgres vs SQLite with Litestream

Ben Johnson, the creator of Litestream, joined Fly.io a few weeks after we migrated changelog.com - episode 50 has all the details. That was pure coincidence. What was not a coincidence, is Gerhard jumping at the opportunity to talk to Ben about Postgres vs SQLite with Litestream.

The prospect of running a cluster of our app instances spread across all regions, with local SQLite & Litestream replication, is mind boggling. Let’s find out from Ben what will it take to get there. Thanks Kürt for kicking off this dream.

Ship It! Ship It! #58

How to keep a secret

Rob Barnes (a.k.a. Devops Rob) and Rosemary Wang (author of Infrastructure as Code - Patterns & Practices) are joining us today to talk about infrastructure secrets.

What do Rosemary and Rob think about committing encrypted secrets into a repository? How do they suggest that we improve on storing secrets in LastPass? And if we were to choose HashiCorp Vault, what do we need to know?

Thank you Thomas Eckert for the intro. Thank you Nabeel Sulieman (ep. 46) & Kelsey Hightower (ep. 44) for your gentle nudges towards improving our infra secrets management.

Ship It! Ship It! #51

From Kubernetes to PaaS - now what?

Today we talk to Mark Ericksen about all the things that we could be doing on the new platform - this is a follow-up to episode 50.

Mark specialises in Elixir, he hosts the Thinking Elixir podcast, and he also helps make Fly.io the best place to run Phoenix apps, such as changelog.com. In the interest of holding our new platform right, we thought that it would be a great idea to talk to someone that does this all day, every day, for many years now.

We touch up on how to run database migrations safely, and how to upgrade our application config to the latest Phoenix version. We also talked about some of the more advanced platform features that we may want to start leveraging, like the multi-region PostgreSQL.

Ship It! Ship It! #50

Kaizen! We are flying ✈️

This is our 5th Kaizen where we talk about the next improvement to changelog.com: we are now running on Fly.io and our PostgreSQL is managed. This is a migration that many were curious about, including Simmy de Klerk, the person that requested this episode.

After migrating all our media files to AWS S3 (check episode 40), we thought that this part was going to be easy. Plan met reality. Pull request 407 has all the details.

We want to emphasise the type of partner relationships that we seek at Changelog & why they are important to us, as well as to our listeners. Honeycomb & Fly embody the principles that we care about, and Gerhard thinks that we are currently missing a Kubernetes partner.

Ship It! Ship It! #36

Keep on-call simple

Gerhard loves simple ideas executed well, which is why he is excited to be speaking today with Ildar Iskhakov & Matvey Kukuy about their startup Amixr, a.k.a. Grafana OnCall.

Ildar & Matvey started with a simple idea and a simple stack - Django, Celery, RabbitMQ & MySQL - all running on Kubernetes. Because they kept their main thing their main thing, and kept improving it every day for a couple of years, now your on-call can be simple too.

This is another Big Tent philosophy story with a Black Swan moment towards the end.

Ship It! Ship It! #35

How I found my lost network packets

Today Gerhard shares the entire story behind his lost packets. He is talking with Drew Marshall, director at Trunk Networks and No One Internet, a Cloud Services Provider & ISP based in Sussex, UK.

Gerhard’s Vodafone ISP gateway was losing packets, and recording some of the previous episodes used to be challenging as his internet connection would cut out up to 10 seconds at a time, multiple times per recording session. He was convinced that his Unifi Dream Machine Pro was not the issue. Drew helped Gerhard realise that it actually was. Not only has Gerhard’s DNS latency improved by 3x, but he can now fail-over between two WAN connections. And because nothing beats a real-world experiment, you can guess what is coming in this episode 😉

You will find latency & packet loss graphs, speed test runs, and a few other interestings in the show notes. We hope that they inspire you to setup a better home network. Most importantly, may you find your humble & brilliant Drew.

Ship It! Ship It! #34

Where is the cloud native App Store?

In our first 2022 episode, Alexis Richardson, co-founder and CEO of Weaveworks, is talking to Gerhard about going fully remote, what a great team looks like, and GitOps. While you may have heard of GitOps, now is a good time to check out opengitops.dev.

The most interesting part of today’s conversation is the missing cloud native App Store. While Apple revolutionised the world with the App Store and the iPhone, we don’t yet have something similar for cloud native apps. You may be thinking “But what about OperatorHub?”, or all the Helm registries out there? The registry fragmentation, operator deprecations and lack of curation are not what people have in mind when they think App Store. But there is more to it, so let’s hear how Alexis thinks about this.

Ship It! Ship It! #20

Kaizen! Five incidents later

This is our second Kaizen episode, where Adam, Jerod & Gerhard talk about changelog.com improvements since episode 10. OK, so Gerhard deleted the DNS API token. Not only did he take the time to understand how that happened, so that he could actually learn from his mistake, but now we have a system in place so that we can share learnings from incidents. By the way, these are publicly available in our #incidents Slack channel.

A great & unexpected thing that happened since we recorded this episode, is Jerod fixing 99% of all the errors that were happening in prod. The top error was the broken Twitter auth - sorry Matt - which was a result of us upgrading to OTP 24 a few months back. Episode 3 show notes include a YouTube stream which captures it all.

We wrap up this episode by each of us sharing the improvements that we would like to do until our next Kaizen. You heard it from Adam first: Ship It Driven Development

Ship It! Ship It! #19

Real-world implications of shipping many times a day

This week Emile Vauge, founder & CEO of Traefik, joins Gerhard to share a story that started as a solution to a 2000 microservices challenge, the real-world implications of shipping many times a day for years, and the difficulties of sustaining an inclusive and healthy open-source community while building a product company.

Working every day on keeping the open-source community in sync with the core team was an important lesson. The second learning was around big changes between major versions.

The journey from Travis CI to Circle CI, then to Semaphore CI and eventually GitHub Actions is an interesting one. The automation tools inspired by the Mymirca ant colony is a fascinating idea, executed well. There is more to discover in the episode.

Ship It! Ship It! #18

Bare metal meets Kubernetes

In this episode, Gerhard talks to David and Marques from Equinix Metal about the importance of bare metal for steady workloads. Terraform, Kubernetes and Tinkerbell come up, as does Crossplane - this conversation is a partial follow-up to episode 15.

David Flanagan, a.k.a. Rawkode, needs no introduction. Some of you may remember Marques Johansson from The new changelog.com setup for 2019. Marques was behind the Linode Terraforming that we used at the time, and our infrastructure was simpler because of it!

This is not just a great conversation about bare metal and Kubernetes, there is also a Rawkode Live following up: Live Debugging Changelog’s Production Kubernetes 🙌🏻

Ship It! Ship It! #15

Assemble all your infrastructure

In this episode, Gerhard follows up on The Changelog #375, which is the last time that he spoke Crossplane with Dan and Jared. Many things changed since then, such as abstractions and compositions, as well as using Crossplane to build platforms, which were mostly ideas.

Fast forward 18 months, 2k changes, as well as a major version, and Crossplane is now an easy choice - some would say the best choice - for platform teams to declare what infrastructure means to them. You can now use Crossplane to define your infrastructure abstractions across multiple vendors, including AWS, GCP & Equinix Metal. The crazy ideas from 2019 are now bold and within reach. Gerhard also has an idea for the changelog.com 2022 setup. Listen to what Jared & Dan think, and then let us know your thoughts too.

The Changelog The Changelog #452

Kaizen! The day half the internet went down

This week we’re sharing a special episode of our new podcast called Ship It. This episode is our Kaizen-style episode where we point our lens inward to Changelog.com to see what we should improve next. The plan is do this episode style every 10 episodes.

Gerhard, Adam, and Jerod talk about the things that we want to improve in our setup over the next few months. We talk about how the June Fastly outage affected changelog.com, how we responded that day, and what we could do better. We discuss multi-cloud, multi-CDN, and the next sensible and obvious improvements for our app.

0:00 / 0:00