Ops Icon

Ops

DevOps, infrastructure, etc.
162 Stories
All Topics

Ship It! Ship It! #62

Operational simplicity is a gift to you

Gerhard’s transition to a senior engineer started 10 years ago, when he embraced the vim mindset, functional core & imperative shell, and was inspired to seek simplicity in his code & infrastructure. Most of it can be traced back to one person: Gary Bernhardt, the creator of Execute Program, Destroy all Software and the now famous Wat idea.

Few stick around long enough to understand the long-term impact of their decisions on production systems. Even fewer are able to talk about them as well as Gary does.

Ship It! Ship It! #61

The ops & infra behind Transistor.fm

Today we talk with two lovely folks from Transistor.fm: Jason Pearl, Senior Software Developer & Jon Buda, co-founder. Gerhard was curious to find out about their setup & how did it change with the launch of the new podcast website builder. After all, you have been hearing us talk about our setup for years, so it was high-time to challenge some assumptions and learn how another team is solving similar problems.

TL;DL: keeping it simple is at the root of smooth operations & stable systems.

Ship It! Ship It! #60

Kaizen! Post-migration cleanup

In our 6th Kaizen, we talk with Jerod about all the things that we cleaned up after migrating changelog.com from a managed Kubernetes to Fly.io. We deleted the K8s cluster and moved wildcard cert management to Fastly & all our vanity domain certs to Fly.io. We migrated the Docker Engine that our GitHub Actions is using - PR #416 has all the details. We did a few other things in preparation for our secrets plan. Thank you Maikel Vlasman, James Harr, Adrian Mester, Omri Gabay & Owen Valentine for kicking it off in our Slack #shipit channel.

Gerhard’s favourite improvement: the new shipit.show domain.

Ship It! Ship It! #59

Postgres vs SQLite with Litestream

Ben Johnson, the creator of Litestream, joined Fly.io a few weeks after we migrated changelog.com - episode 50 has all the details. That was pure coincidence. What was not a coincidence, is Gerhard jumping at the opportunity to talk to Ben about Postgres vs SQLite with Litestream.

The prospect of running a cluster of our app instances spread across all regions, with local SQLite & Litestream replication, is mind boggling. Let’s find out from Ben what will it take to get there. Thanks Kürt for kicking off this dream.

Ship It! Ship It! #58

How to keep a secret

Rob Barnes (a.k.a. Devops Rob) and Rosemary Wang (author of Infrastructure as Code - Patterns & Practices) are joining us today to talk about infrastructure secrets.

What do Rosemary and Rob think about committing encrypted secrets into a repository? How do they suggest that we improve on storing secrets in LastPass? And if we were to choose HashiCorp Vault, what do we need to know?

Thank you Thomas Eckert for the intro. Thank you Nabeel Sulieman (ep. 46) & Kelsey Hightower (ep. 44) for your gentle nudges towards improving our infra secrets management.

Machine Learning github.com

A collection of resources to learn about MLOps

While still in its infancy, MLOps has attracted machine learning engineers and software engineers in general. With every new paradigm comes new challenges and opportunities to learn. In this primer, we highlight a few available resources to upskill and inform yourself on the latest in the world of MLOps.

Good resources, regardless of whether you think MLOps is its own thing or should be rolled into DevOps.

Ship It! Ship It! #57

What do oranges & flame graphs have in common?

Today we are talking with Frederic Branczyk, founder of Polar Signals & Prometheus maintainer. You may remember Frederic from episode 33 when we introduced Parca.dev.

This time, we talk about a database built for observability: FrostDB, formerly known as ArcticDB. eBPF generates a lot of high cardinality data, which requires a new approach to writing, persisting & then reading back this state.

TL;DR FrostDB is sub zero cool & well worthy of its name.

Ship It! Ship It! #56

DevOps teams with shared responsibilities

Today we are talking with Maikel Vlasman, technical lead for a large Dutch machine construction company, and a cloud engineer by heart. We cover self-updating GitLab & ArgoCD, Maikel’s thinking behind dev environment setup and a Kubernetes workshop that he is preparing for his team. The goal is to function as a true DevOps team with shared responsibilities.

This conversation started as a thread in our community Slack - link in the show notes. Thank you Maikel for being a long-time Changelog listener and for reaching out to us - we enjoyed telling this story.

Ship It! Ship It! #53

Securing K8s releases (KubeCon EU 2022)

Today we are at KubeCon CloudNativeCon EU 2022, talking to Adolfo García Veytia about securing Kubernetes releases. Adolfo is a Staff Software Engineer at Chainguard, and one of the technical leads for SIG release, meaning that he helps ship Kubernetes. You most likely know him as Puerco, and have seen first-hand his passion for securing software via SBOMs, cosign and SLSA. Puerco’s love for bikes and Chainguard are a great match 🚴‍♂️

Founders Talk Founders Talk #89

Leading GitLab to IPO

This week Sid Sijbrandij, Co-founder and CEO of GitLab, is back talking with Adam about all the details of their massive IPO last October 2021. To set the stage, this episode was recorded on Feb 1, 2022. During the show Adam mentioned they IPO’d at a $13B market cap, but they actually ended their opening day at approximately $15B. That’s a massive win for open source, GitLab, Sid, and the rest of the team. For loyal listeners you know we’ve had Sid on this show before, so of course we had to get him back on the show post-IPO to get all the details of this new journey.

Ship It! Ship It! #51

From Kubernetes to PaaS - now what?

Today we talk to Mark Ericksen about all the things that we could be doing on the new platform - this is a follow-up to episode 50.

Mark specialises in Elixir, he hosts the Thinking Elixir podcast, and he also helps make Fly.io the best place to run Phoenix apps, such as changelog.com. In the interest of holding our new platform right, we thought that it would be a great idea to talk to someone that does this all day, every day, for many years now.

We touch up on how to run database migrations safely, and how to upgrade our application config to the latest Phoenix version. We also talked about some of the more advanced platform features that we may want to start leveraging, like the multi-region PostgreSQL.

Ship It! Ship It! #50

Kaizen! We are flying ✈️

This is our 5th Kaizen where we talk about the next improvement to changelog.com: we are now running on Fly.io and our PostgreSQL is managed. This is a migration that many were curious about, including Simmy de Klerk, the person that requested this episode.

After migrating all our media files to AWS S3 (check episode 40), we thought that this part was going to be easy. Plan met reality. Pull request 407 has all the details.

We want to emphasise the type of partner relationships that we seek at Changelog & why they are important to us, as well as to our listeners. Honeycomb & Fly embody the principles that we care about, and Gerhard thinks that we are currently missing a Kubernetes partner.

Practical AI Practical AI #176

MLOps is NOT Real

We all hear a lot about MLOps these days, but where does MLOps end and DevOps begin? Our friend Luis from OctoML joins us in this episode to discuss treating AI/ML models as regular software components (once they are trained and ready for deployment). We get into topics including optimization on various kinds of hardware and deployment of models at the edge.

Ship It! Ship It! #49

Improving an eCommerce fulfillment platform

Alex Sims, a Senior Software Engineer at James & James, an eCommerce fulfilment company, reached out to us about the Kaizen story of the third-party logistics (3PL) platform that he has been involved with for several years now.

The system delivered 16 millions of orders in 10 years, and 4.5 million in the last year alone. All the numbers are going up, and there is only so much that a single PHP monolith deployed as VM images can handle. So how do you even start thinking about the architectural improvements, and inspire everyone involved to move towards better?

We encourage you to look at the architectural diagrams in the show notes, especially the 10 year roadmap, and ask Alex for a blog post follow-up. While today’s episode was a good conversation starter, there is a lot that we did not have time to cover.

Gergely Orosz newsletter.pragmaticengineer.com

Inside the longest Atlassian outage of all time

Gergely Orosz did an excellent job detailing the ins & outs of Atlassian’s epic outage:

Hundreds of companies have no access to JIRA, Confluence and OpsGenie. What can engineering teams learn from the poor handling of this outage?

The TL;DR on the cause of the outage is a script that was supposed to “mark for deletion” some records also had “permanently delete” functionality and was run against a wrong list of IDs, improperly deleting 400 of their customers. Oh, and their backup restore process is really good at doing all customers, but not a subset. Ruh roh!

Lots to learn here, and Gergely puts a fine point on the biggest takeaways. A must-read!

Ship It! Ship It! #48

Launching Dagger

In this episode we talk about launching Dagger with all four founders: Andrea, Eric, Sam & Solomon.

While you may remember Sam & Solomon from episode 23, this time we assembled all four superheroes in this story and went deeper, covering nearly three years of refinements, the launch, as well as the world-class team & community that is coming together to solve the next problem of shipping software. Container images and Kubernetes are great steps in the right direction, but now it’s time for the next leap into the future.

You can use Dagger to run your CI/CD pipelines locally, without needing to commit and push. You can also use Dagger as a Makefile alternative, which resonates with Gerhard, but go further and your perspective on documentation & automation may start shifting.

Gerhard believes that this is the Docker moment of CI/CD.

Ship It! Ship It! #47

The Docker Swarm story

This episode was requested by Tyler Smith who feels that he may not need Kubernetes just yet. Tyler has a few questions about Docker & Docker Swarm, so Andrea Luzzardi, former Docker Swarm Lead, joins us today to answer them.

We talk about Docker Swarm beginnings, some of the challenges that it faced, and what Andrea’s recommendation is for Tyler’s journey with Docker Swarm.

After dedicating four years of his professional career to Docker Swarm, Andrea is the best person that Gerhard knows to talk about this subject. And guess what, the same thing happened now as it did at KubeCon 2015: Sam pointed to Andrea. It will all make sense in the first five minutes. This one is going to be fun!

Ops squeaky.ai

Why we don't use a staging environment

The Squeaky team goes from dev straight to prod (same here), but many people advocate for (and use) staging or other “pre-live” environments instead.

While there are obvious benefits to deploying to different environments, at Squeaky we’ve decided to take a different approach. We only have two environments: our laptops, and production. Once we merge into the main branch, it will be immediately deployed to production.

Perhaps that sounds unusual, but so far it’s outweighed the benefits of pre-live environments, and we believe it’s helping us to ship faster, and lower the number of issues on production. So, I thought I’d write this post to share why we think it works, and why you should consider it too.

Ship It! Ship It! #46

A simpler alternative to cert-manager

Nabeel Sulieman, Senior Software Engineer at Vercel, talks about KCert, a simpler alternative to cert-manager that he built. Gerhard tried it out, and he thinks that Nabeel is onto something. If you want to see the video that they recorded, ping us on Twitter or Slack.

We love this story, especially the long-term approach of working on something that one truly believes in, and the only reason is because it’s fun. The world needs more people like Nabeel, and we hope that this episode inspires you to go all out, and do just that.

Lars Wikman underjord.io

Fundamentals & deployment

Lars Wikman reacts to Gerhard’s excellent conversation with Kelsey Hightower on Ship It!

So he essentially said, I’m interpreting here, that when it comes to deploying software to servers the documented manual steps for deploying something need to be the canonical reference. Then whether you build bash scripts, Ansible playbooks, Makefiles, Dockerfiles, Terraform, Kubernetes or something else to encode that procedure into something repeatable and scalable that’s a separate step. Having documented the process required to set it up means that there is an answer to the question: How do I get this running? An answer that doesn’t require you to parse the .yml files or grok Ansible roles and groups.

Lars springs forward from there with many thoughts of his own on the matter.

  0:00 / 0:00