Changelog News

Developer news with ZERO April Fools jokes

Hey there! 👋

The big story right now is the recently uncovered backdoor in liblzma (aka XZ) – a relatively obscure compression library that happens to be a dependency of OpenSSH.

This incident is noteworthy for so many reasons: the exploit itself, how it was deployed, how it was found, what it says about our industry & how the community reacted.

Today’s issue is almost entirely dedicated to this story, looking at it from all those angles. I’ll throw in a list of other interesting links at the end, though, in case you’re not interested in the deep-dive.

Ok, let’s get straight into it. (Audio Edition)

🎧 Good pods for your week

🎙️ Two interviews with us on other people’s podcasts changelog.fm/584
💚 Adolfo Ochagavía is an undercover generalist changelog.com/friends/37
🚀 Justin & Autumn take you with them to SoCal Linux Expo shipit.show/97
🪩 13% of the time, Devin works every time jsparty.fm/317
🤖 AI’s complicated relationship to software developers practicalai.fm/262
⏰ Bill Kennedy & Matt Boyle disagree on debugging gotime.fm/309

🧐 Backdoor in upstream xz/liblzma leading to ssh server compromise

Let’s start our story the same way most folks did, on Friday when Microsoft researcher Andres Freund posted an email to Debian’s oss-security list containing this bombshell:

After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer:
The upstream xz repository and the xz tarballs have been backdoored.
At first I thought this was a compromise of debian’s package, but it turns out to be upstream.

Andres goes on to explain his findings in detail. The mind blowing thing is that he decided to shave this particular yak because he was doing some micro-benchmarking and needed the system to be super low load, which made him realize sshd was using a lot of CPU. Go read all the work he put in to find the backdoor and then consider how specific his situation had to be in order to even notice it… Thankfully, he found the backdoor relatively early in its rollout:

Due to the working of the injected code (see below), it is likely the backdoor can only work on glibc based systems.
Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by linux distributions, and where they have, mostly in pre-release versions.

🤯 “It’s RCE, not auth bypass, and gated/unreplayable”

The exploit itself is super interesting as well. I’m not ashamed to say most of it’s over my head, but recurring Changelog guest, Filippo Valsorda, does a great job explaining the nitty gritty details. Follow the link if you’re interested in all of the particulars, but this statement sums it up well:

This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library.
Looks like this got caught by chance. Wonder how long it would have taken otherwise.

So the attacker is competent, malicious & has authorized write access to a widely used library. How’d that happen?!

❓ XZ Utils backdoor

This page is maintained by Lasse Collin, the solo maintainer of XZ. Can you guess where all this is headed? Lasse says:

XZ Utils 5.6.0 and 5.6.1 release tarballs contain a backdoor. These tarballs were created and signed by Jia Tan.
Tarballs created by Jia Tan were signed by him. Any tarballs signed by me were created by me.
GitHub accounts of both me (Larhzu) and Jia Tan are suspended.

Jia Tan?! Who is that? The plot thickens!

🤖 AI-powered Autofix debugs your code in minutes

Thanks to Sentry for sponsoring Changelog News 💰

Ben Peven:

Sentry knows a lot about the inner workings of an application’s codebase. So we got to thinking, how can we use this rich dataset to make debugging with Sentry even faster?
Many generative AI (GenAI) tools (e.g. GitHub Copilot) improve developer productivity in their dev environment, though few have the contextual data Sentry has to help fix errors in production. Our new AI-enabled Autofix feature understands what your users are doing when an error occurs, analyzes the error, generates a fix and even opens a pull request for your review. It’s like having a junior developer ready to help on-demand.

Give it a try… oh, and don’t forget to use code CHANGELOG when you sign up for Sentry to get $100 off their team plan. ✊

🤿 Everything I know about the XZ backdoor

Evan Boehs looked up the public history of GitHub user JiaT75 (Jia Tan), which goes back to 2021. Tan starts slowly then slowly gains trust alongside a few other accounts that appear to be sock puppets. Evan tried to use other public information to identify who Jia Tan really is, but a potential LinkedIn match seems unlikely:

I have received a few emails alerting me to a LinkedIn of somebody named Jia Tan2. Their bio boasts of large-scale vulnerability management. They claim to live in California. Is this our man? The commits on JiaT75’s GitHub are set to +0800, which would not indicate presence in California. UTC-0800 would be California. Most of the commits were made between UTC 12-17, which is awfully early for California. In my opinion, there is no sufficient evidence that the LinkedIn being discussed is our man.

Analysis of the name has also been performed, but when you include the middle name (Cheong) that was found in one git log it seems unlikely that it’s a real name:

It’s most likely our actor simply mashed plausible sounding Chinese names together.

As of the time of this writing, it’s unknown who Jia tan really is.

💬 A Microcosm of the interactions in Open Source projects

Rob Mensching lays out the process of the attack and focuses in on step 0:

Original maintainer burns out, and only the attacker offers to help (so attacker inherits trust built up by the original maintainer)

Someone found an email thread that captured the individual messages sent when step 0 was taking place, and Rob goes through and picks out salient messages to paint a picture for us.

First, we start with a reasonable request asked reasonably. The question forces the maintainer to address his “failings”. I use “failings” in quotes here because a. the maintainer doesn’t actually owe anything here so he hasn’t actually failed and b. I know exactly how this feels. It feels terrible to let down your “community”.
”Is XZ for Java still maintained? I asked a question here a week ago and have not heard back.” - https://www.mail-archive.com/xz-devel@tukaani.org/msg00562.html
The maintainer acknowledges he’s “behind” and is struggling to keep up. This is a cry in pain. This is a cry for help. Help will not be coming in this thread.

This question does not originate from Jia Tan. Instead, its author eventually points to Jia Tan as a good person to “have a bigger role in the future…” How many of us find ourselves in positions similar to Lasse? I’ve spoken with so many maintainers who would love to pass their project on to someone capable/interested, but it’s darn near impossible. Rob closes with this:

It takes skill and knowledge to write software. And while many skills and some knowledge will transfer, working on a new software project inevitably requires developing new skills and more knowledge.
Software developers are not fungible cogs that you can swap in and out at will.
The email thread ends with the complaining consumers offering no help while continuing to make demands. Only the attacker is left.

🚪 Techies vs spies: the xz backdoor debate

In the wake of this event, many voices have called out the unhealthy relationship between unpaid maintainers and companies that benefit from their work. Don’t get me wrong… YES. That is absolutely a problem. But Substack writer lcamtuf wrote up a different take (that I haven’t heard previously) that I absolutely believe plays a part:

The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.

Unfortunately, sometimes that person with “a pulse and some modcum of skill” is a highly competent, malicious actor.

More fundamentally, the xz backdoor isn’t a technical problem and it probably can’t be solved with technology alone. To a large extent, it’s a counterintelligence challenge — squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft.

There are many, many more comments and blog posts on this event, but I have to stop somewhere. Here’s a shortlist of more reading if you still haven’t had your fix:

🔐 What is Tailscale SSH?

Thanks to Tailscale for sponsoring Changelog News 💰

With Tailscale SSH, you can:

SSH as normal, using Tailscale for authentication. With Tailscale SSH, Tailscale takes over port 22 for SSH connections incoming from the Tailscale network. Tailscale will authenticate and encrypt the connection over WireGuard, using Tailscale node keys. The SSH client and server will still create an encrypted SSH connection, but it will not be further authenticated.
Verify high-risk connections with check mode. Optionally require certain connections, or connections as certain users (for example, root), to re-authenticate before connecting. This allows the user to access these high-risk applications for the next 12 hours or for a specified check period before re-authenticating again.

Your SSH config (/etc/ssh/sshd_config) and keys (~/.ssh/authorized_keys) files will not be modified, which means that other SSH connections to the same host, not made over Tailscale, will still work.

Pretty cool, huh? Learn more…

🎞️ Clip of the week

A tool I love that you might too if you’re a tmux user

🔗 That linked list I promised up top

That’s the news for now, but we have some great episodes coming up: Zeno Rocha from Dracula & Resend joins Adam for this week’s Changelog Interview and Gerhard Lazu returns to Changelog & Friends for Kaizen 14!

Have a great week, forward this to a friend who might dig it & I’ll talk to you again real soon. 💚

–Jerod