Changelog News – Episode #88

Who in the world is Jia Tan?

all about the xz backdoor & what it means

Featuring

All Episodes

The big story right now is the recently uncovered backdoor in liblzma (aka XZ) – a relatively obscure compression library that happens to be a dependency of OpenSSH.

This incident is noteworthy for so many reasons: the exploit itself, how it was deployed, how it was found, what it says about our industry & how the community reacted. Let’s dig in!

Featuring

Sponsors

Sentry – AI-powered Autofix debugs & fixes your code in minutes. Give it a try… oh, and don’t forget to use code CHANGELOG when you sign up for Sentry to get $100 off their team plan. ✊

TailscaleAdam loves Tailscale! Tailscale is programmable networking software that’s private and secure by default. It’s the easiest way to connect devices and services to each other, wherever they are. Secure, remote access to production, databases, servers, kubernetes, and more. Try Tailscale for free for up to 100 devices and 3 users at changelog.com/tailscale, no credit card required.

Notes & Links

📝 Edit Notes

All links mentioned in this episode of Changelog News (and more) are in its companion newsletter.

Chapters

1 00:00 The intro
2 00:56 The discovery
3 02:23 The code
4 03:08 The maintainer
5 03:44 Sponsor: Sentry
6 04:45 The attacker
7 06:04 The pattern
8 08:00 The debate
9 09:27 The outro

Transcript

📝 Edit Transcript

Changelog

Play the audio to listen along while you enjoy the transcript. 🎧

What up, nerds? I’m Jerod and this is Changelog News for the week of Monday, April 1st 2024.

Yes, it’s April Fools Day. The worst day of the year to be an internet denizen. But don’t worry, Changelog News is 100% prank free, AI free and gluten free, too.

The big story right now is the recently uncovered backdoor in liblzma (aka XZ) – a relatively obscure compression library that happens to be a dependency of OpenSSH.

This incident is noteworthy for so many reasons: the exploit itself, how it was deployed, how it was found, what it says about our industry & how the community reacted.

Today’s issue is almost entirely dedicated to this story, looking at it from all those angles. I’ll throw in a list of other interesting links at the end, though, in case you’re not interested in the deep-dive.

Ok, let’s get into the news.

The discovery

Let’s start our story the same way most folks did, on Friday when Microsoft researcher Andres Freund posted an email to Debian’s oss-security list containing this bombshell:

After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer:

The upstream xz repository and the xz tarballs have been backdoored.

At first I thought this was a compromise of debian’s package, but it turns out to be upstream.

Andres goes on to explain his findings in detail. The mind blowing thing is that he decided to shave this particular yak because he was doing some micro-benchmarking and needed the system to be super low load, which made him realize sshd was using a lot of CPU. Go read all the work he put in to find the backdoor and then consider how specific his situation had to be in order to even notice it… Thankfully, he found the backdoor relatively early in its rollout:

Due to the working of the injected code (see below), it is likely the backdoor can only work on glibc based systems.

Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by linux distributions, and where they have, mostly in pre-release versions.

The code

The exploit itself is super interesting as well. I’m not ashamed to say most of it’s over my head, but recurring Changelog guest, Filippo Valsorda, does a great job explaining the nitty gritty details. Follow the link if you’re interested in all of the particulars, but this statement sums it up well:

This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library.

Looks like this got caught by chance. Wonder how long it would have taken otherwise.

So the attacker is competent, malicious & has authorized write access to a widely used library. How’d that happen?!

The maintainer

This page is maintained by Lasse Collin, the solo maintainer of XZ. Can you guess where all this is headed? Lasse says:

XZ Utils 5.6.0 and 5.6.1 release tarballs contain a backdoor. These tarballs were created and signed by Jia Tan.

Tarballs created by Jia Tan were signed by him. Any tarballs signed by me were created by me.

GitHub accounts of both me (Larhzu) and Jia Tan are suspended.

Jia Tan?! Who is that? The plot thickens!

It’s not time for Sponsored News!

AI-powered Autofix debugs & fixes your code in minutes

Ben Peven writing for Sentry:

Sentry knows a lot about the inner workings of an application’s codebase. So we got to thinking, how can we use this rich dataset to make debugging with Sentry even faster?

Many generative AI (GenAI) tools (e.g. GitHub Copilot) improve developer productivity in their dev environment, though few have the contextual data Sentry has to help fix errors in production. Our new AI-enabled Autofix feature understands what your users are doing when an error occurs, analyzes the error, generates a fix and even opens a pull request for your review. It’s like having a junior developer ready to help on-demand.

I’ve used this a couple times and was pleasantly surprised by its “ideas.” Give it a try… oh, and don’t forget to use code CHANGELOG when you sign up for Sentry to get $100 off their team plan. ✊

The attacker

Evan Boehs looked up the public history of GitHub user JiaT75 (Jia Tan), which goes back to 2021. Tan starts slowly then slowly gains trust alongside a few other accounts that appear to be sock puppets. Evan tried to use other public information to identify who Jia Tan really is, but a potential LinkedIn match seems unlikely:

I have received a few emails alerting me to a LinkedIn of somebody named Jia Tan2. Their bio boasts of large-scale vulnerability management. They claim to live in California. Is this our man? The commits on JiaT75’s GitHub are set to +0800, which would not indicate presence in California. UTC-0800 would be California. Most of the commits were made between UTC 12-17, which is awfully early for California. In my opinion, there is no sufficient evidence that the LinkedIn being discussed is our man.

Analysis of the name has also been performed, but when you include the middle name (Cheong) that was found in one git log it seems unlikely that it’s a real name:

It’s most likely our actor simply mashed plausible sounding Chinese names together.

As of the time of this recording, it’s unknown who Jia tan really is.

The pattern

Rob Mensching lays out the process of the attack and focuses in on step 0:

Original maintainer burns out, and only the attacker offers to help (so attacker inherits trust built up by the original maintainer)

Someone found an email thread that captured the individual messages sent when step 0 was taking place, and Rob goes through and picks out salient messages to paint a picture for us.

First, we start with a reasonable request asked reasonably. The question forces the maintainer to address his “failings”. I use “failings” in quotes here because a. the maintainer doesn’t actually owe anything here so he hasn’t actually failed and b. I know exactly how this feels. It feels terrible to let down your “community”.

”Is XZ for Java still maintained? I asked a question here a week ago and have not heard back.” - https://www.mail-archive.com/xz-devel@tukaani.org/msg00562.html

The maintainer acknowledges he’s “behind” and is struggling to keep up. This is a cry in pain. This is a cry for help. Help will not be coming in this thread.

This question does not originate from Jia Tan. Instead, its author eventually points to Jia Tan as a good person to “have a bigger role in the future…” How many of us find ourselves in positions similar to Lasse? I’ve spoken with so many maintainers who would love to pass their project on to someone capable/interested, but it’s darn near impossible. Rob closes with this:

It takes skill and knowledge to write software. And while many skills and some knowledge will transfer, working on a new software project inevitably requires developing new skills and more knowledge.

Software developers are not fungible cogs that you can swap in and out at will.

The email thread ends with the complaining consumers offering no help while continuing to make demands. Only the attacker is left.

The debate

In the wake of this event, many voices have called out the unhealthy relationship between unpaid maintainers and companies that benefit from their work. Don’t get me wrong… YES. That is absolutely a problem. But Substack writer lcamtuf wrote up a different take (that I haven’t heard previously) that I absolutely believe plays a part:

The real issue with a lot of small, foundational OSS libraries is just that there isn’t enough to do. They were written decades ago by a single person — and beyond bugfixes, they are not really supposed to change much. You don’t do major facelifts of zlib or giflib every year; even if you wave some cash around, it’s hard to build a sustainable community around watching paint dry. After a while, the maintainer just isn’t all that into it anymore; they are eager to pass the baton to anyone with a pulse and some modicum of skill.

Unfortunately, sometimes that person with “a pulse and some modicum of skill” is a highly competent, malicious actor.

More fundamentally, the xz backdoor isn’t a technical problem and it probably can’t be solved with technology alone. To a large extent, it’s a counterintelligence challenge — squarely within the competencies of governments and a handful of commercial entities with ecosystem-wide surveillance capabilities. This notably includes Google and Microsoft.

That’s the news for now, but scan this episode’s companion newsletter (link in the show notes) for more commentary on the XZ backdoor, oh and 13 other interesting links that are completely unrelated.

Have a great week, leave us a 5-star review if you dig it & I’ll talk to you again real soon. 💚

Changelog

Our transcripts are open source on GitHub. Improvements are welcome. 💚

Player art
  0:00 / 0:00