Developer (un)happiness

Well, developers are unhappy… That’s the sentiment, right?

That is the sentiment.

Why? Are you happy, Jerod? You’re a developer, right? Are you in the 80% rule, or are you in the 20% rule?

It depends on the minute of the particular day whether or not I’m happy or unhappy. It’s a fleeting thing, happiness. Am I satisfied in my work? Yes. Do I always think that? No. Am I a typical developer? Probably not anymore. We’ve been podcasters now for a long time, and so I don’t hold a 9-to-5 software job, which is probably the people mostly who are being interviewed or surveyed.

I wasn’t in that survey, so… My sentiment was not in there. No.

I did not take the Stack Overflow. We have Abi Noda here with us from DX. Ai, did you take the Stack Overflow survey?

I did not, but I’ve definitely been looking at the results.

Interesting results. 80% is a large number. I mean, that’s an overwhelming number. And it’s not a small survey.

Pareto’s principle says 80/20. That’s the big principle.

Right.

Apparently, it’s true in regards to developers’ happiness.

Or lack thereof, I guess. That would be the 20. The happy would be 20, and the unhappy would be 80. What’s interesting about this - so this came out, as we said, from the 2024 Stack Overflow survey results synthesized by ShiftMag. So shout-out to Anastasia Uspensky at ShiftMag for really highlighting this particular point, and pulling together a few other data points to try to figure out… She was trying to figure out why. Why are they unhappy? So you might think, “Well, it’s the AI. The AI is taking away our joy.” That doesn’t seem like that’s the case. At least that’s what her conclusion is, it’s not the AI. The AI is making us slightly more productive, and maybe a little bit more apprehensive about the future. But currently, I think developers who are in their seats, writing code, know that at least today they aren’t being replaced in large swaths by AI. So if you’re a good software developer today, you’re not too worried about that. At least not in the present.

And it’s not the stuff they’re working on necessarily, but it is other things. Other things like tech debt and complexity. That comes out in all kinds of different ways, but that was her finding. Abi, you run a survey company. You guys create surveys for folks?

Ooh, he said it, Abi. He called you a survey company.

Sorry, is that reductive?

That’s a jab! I’m just kidding. I don’t know.

I don’t mean to reduce. I don’t mean to reduce.

I’m just throwing some jokes out there.

We’ll take that.

It’s Friends, you’ve got to do it.

No, it’s fair. It’s fair.

I don’t mean to reduce, but you all help people do surveys…

Yup.

I’m just curious your thoughts as we kick into this topic.

We help people do surveys, and collect other types of data on their developers. Just to clarify…

Just clarifying. Good job, Abi.

Yeah. The survey is really interesting… One of the first things that came to mind when I read that headline of 80% of developers being unhappy - something we see across organizations we work with is something a little bit similar. We track something that we call attrition risk. So what is the likelihood of a developer actually leaving a company in the next 12 months. And that number typically hovers around 10% to 15%.

Okay.

So one of the first things that came to mind - what are the implications of 80% of developers being unhappy, if only 15% of them are actually going to leave the company. And that amounts to a lot of unhappy employees, who are not doing their best work, who are probably not clocking in the 40, 50 hours that we’re hoping for, who may be phoning it in a little bit. So that was interesting to me, just reading that headline.

Right. I always go bigger when I see something like an industry like software development, and I start thinking – and we don’t have answers necessarily, but I start wondering “Well, how many of workers are happy, just in general?” Is 80% ridiculously large? It is an absolute term. It’s four out of five. That’s a large percentage. But if we compared it to some other industry, medical workers, teachers, plumbers, pick your industry - would they be at like maybe 75%, or 85%? Or are they down there in the forties and fifties and we’re way out of line? That’s the question that I usually ask, and I don’t have the answer ever… So I kind of just twiddle my thumbs and move on.

I was thinking that too, the macro versus the micro, what’s the devs versus the world aspect to this… Because I would imagine that medical workers, as an assumption, are generally - especially since the pandemic, are higher to be unhappy, for obvious reasons. A lot of pressure put on them, a lot of change… I think a lot of bureaucracy, a lot of things in that system… Plumbers, I’m not so sure. Plumbers - if you’re an indie plumber, you’re probably pretty happy. Plumbers make pretty good money.

Yeah.

They generally call their own shots, kind of hard to replace… You call them in a pinch, like “Hey, listen. I’ve got water on my floor, man. You’ve got to come help me out here.” And they jump on it. And they’re like “Hey, 500 bucks. Thank you very much.” “All you did was turn the nut. Come on now.” You know?

[00:08:00.26] It’s a relatively stable industry. I mean, you’re always going to have people with plumbing, new plumbing, plumbing problems etc. So it’s not as much affected by perhaps the Federal Reserve like we are.

The medical industry went through a huge swell, of course, during COVID, where there was just so many needs for medical workers that their salaries went through the roof. They were in huge demand. Of course, they worked ridiculously long and trying hours… And so that was probably not producing happiness. But the pay was really, really good. And now coming down on the other side of it, it’s similar to the software world, where it’s like, demand is waning, jobs are harder to find, you may go unemployed for a while… And so there’s probably a similar chart - if you were to chart - overall demand.

Teachers, though, is a good one to cross-examine on that.

Teachers are never happy, are they? I mean, they’re so under-resourced, they’re struggling… I just feel like most teachers probably would love to be happier. I don’t know.

What’s funny though is I wonder how you can Venn diagram happiness with job happiness. Because I meet a lot of teachers that are very happy, very joyful, very purposeful, serving, loving people, and happy in life. But then you say, “Are you happy with your job?” And I wonder now if we zoom out to this happiness/unhappiness level with devs even - because some of the findings said that code was not what made developers unhappy, because most of them are doing things on the side, either through learning, or for career development, things like that. I just wonder how much - is it job unhappiness? Is it unhappiness generally? Because a lot of people, especially in the United States - that’s where my lens is at, because that’s where I live - are generally unhappy with a lot of things. So does that like spill over the –

Trickle over? I think it does. This particular question was specific to “Are you happy with your job?” And so that is the context that we’re talking about. But of course, nobody just draws a wall up around their job, and as they walk through the door to work, all of a sudden they’re like this different feeling person. These things do affect each other.

It’s interesting… Yeah, the question was “How satisfied are you in your current professional developer role?” And the options were “Not happy at work”, “Complacent at work”, and “Happy at work.” So actually, of that 80% who are reported to be unhappy, 47% are complacent. So they didn’t say they were unhappy. They said “Meh.”

They said “Meh.” So what this number is is you take the happy people, and it’s 20%, and then there’s two categories that make up the 80, and a large part of that is not “I hate my job.” They’re just like “It’s a job”, which isn’t all that bad. I mean, a job is a job because it’s work. I know we have a culture, and of course, the desire to follow your passion and do what you love, and all of these things… But that’s the few - and the proud, usually - who can actually do that. It’s not very many of us who can do what we love all the time, and it feels like “I would do this if I wasn’t getting paid for it.” That’s not the normal. So just being kind of “Meh” with your job is – it could be worse, right? It could be worse.

What I think is really interesting is the why. So why are developers satisfied or unsatisfied in their jobs? And I think the first images that pop into our minds might be pay, or their managers, or layoffs, or AI… But if I’m reading this correctly, the top contributors to satisfaction are actually the developer experience, or technical debt. The tooling, the complexity of the systems and the codebase. Am I reading that correctly? Is that how you guys read it?

[00:12:07.19] That’s how I read it as well. Yes. Technical debt and complexity are the two driving factors to this unhappiness… Which effectively is developer experience. I mean, it’s your work. And how did we get there? I think it’s just like two decades of move fast and break things, isn’t it? I mean, isn’t that just kind of how we’ve gotten here? That’s my best guess.

Maybe.

Yeah. Two decades of move fast, break things, hire a lot of people, churn a lot of people, reorg many times… And now everything’s a mess.

Right. So in this tracking that you do with regard to attrition - 15% to 25%, is that what you said?

Yeah.

…in the next 12 months likely to move somewhere else. Do you also get qualitative information about why? Why are they moving on? Is it similar things?

Yeah. So we’re focused on measuring the developer experience. So a lot of the things listed here - you know, difficulty of understanding code, or developer environments, CI/CD, strategy on the team… A lot of these things are aspects of the developer experience we measure for lots of different companies. And then we correlate the two. So we correlate these different aspects or facets of developer experience against who’s at risk of leaving, and who’s actually left. And our data actually aligns quite a bit with what I’m looking at here with the Stack Overflow report.

Yeah, the difficulty of doing work as a developer seems to be the preeminent cause of regrettable attrition for companies. Not pay, not liking your manager, not stock compensation. It often is just the difficulty of actually doing work.

Right. Which can manifest in technical issues, but also bureaucratic issues.

It makes it harder to be productive.

Right. You’re feeling like you’re not getting anything done, or you’re constantly working JIRA tickets, and you come in in the morning and you’ve got 20 open tickets, and you work eight hours, and you sweat and you bleed, and you leave and you’ve got 22 open tickets, and you’re like “I’m never going to get myself up from here into a place of progress.” You just feel like maintenance. Maintenance is all it is. And I can see how that would be demoralizing, especially over time.

Yeah, when it’s not getting better. I mean, it’s demoralizing for developers, it’s also demoralizing for leaders I talk to, who are getting this type of data at their companies, and quarter after quarter, despite making efforts to make improvements around this stuff, the data keeps coming back that things are slow, people are frustrated, and it’s hard to get work done.

So I guess the question is how do you solve that problem? Is it the organization’s problem? Is it the leadership’s problem? Is it the product’s problem? Is it the market’s problem? Because I think a lot of that complexity comes from the fact that solving software problems are hard, generally. Being blocked is very common. Having to help others level up or answer questions is very common. And that’s going to be pretty much a thing, I guess, potentially, if AI starts to solve some of this for us, that gets to be reduced some… This blockage, so to speak; this spending time looking for answers, spending time answering answers, or repeating answers for people… The blockage that comes from the lack of awareness of where to go next and be productive.

At GitHub - I think I told this story to you before, Adam.

Tell it again.

At GitHub we had a lot of problems. Developer tooling, getting releases out, the builds, developer environments… People were leaving, and they were telling leaders that they were leaving because it was hard to get things done at GitHub. This is back in 2020, if I’m getting my years right.

[00:16:13.01] Well, hindsight is 20/20.

[laughs] And what we ended up doing - we froze features for a quarter. All of GitHub engineering. No features. A whole quarter spent fixing these problems. It was dramatic. And things got a lot better as a result. And another example is actually Atlassian. Their CTO is very public about how they’re focusing on developer productivity, developer experience. At Atlassian not only do they have a pretty substantial portion of their engineering organization that is devoted to this type of stuff, but they give all product engineering teams at Atlassian 10% of their time to be spent fixing things - as they call it, “fixing things that suck.” That get in the way of productivity.

But to answer your question, Adam - what do we do about it, and what’s preventing us from doing things about it? I think it actually boils down to the fact that… You see a survey like this from Stack Overflow, right? People are unhappy.

It’s because of technical debt, and the developer experience. But to actually do something about it as a business, you have to be able to calculate that the cost of doing something about it is outweighed by the return on investment you’re going to get after you do something about it. And I think that’s a really hard problem right now. And no one knows how to actually quantify this set of things, the developer experience. It is something that you can take to the CFO or CEO and say “Hey, we’re slow because of X, Y, and Z. And if we fix X, Y, and Z, we’ll be this much faster, and it’ll be worth it.” That’s the hard problem. So no one can make the case for doing something about a lot of this stuff, because you can’t talk about it in terms of dollars.

Yeah. I think this speaks to our technical debt metaphor and some of the argumentation we’ve had on this show with friends about “Is that a good metaphor or not?”, because you can’t really quantify it like you can actual debt. You can take your debt and your debt service principle and interest, you can take your interest rate and you can put that on a chart, and you can extrapolate it and say “Look, if we don’t pay this debt down now–” Now, maybe not if you’re the United States government, but if you’re like an actual business, you can say “If we don’t pay this debt down, we’re going to go bankrupt in 90 days.” And that convinces leadership to be like “Okay, it’s worth it.” But when it comes to technical debt, we lack that quantitative ability to extrapolate forward and say “We’re going this slow right now. If we don’t dramatically change things, start paying this down, we’re going to grind to a halt in 90 days.”

Well, what has been the happiness level of past surveys? Can we just use Stack Overflow surveys as an example? Because that’s what we’re lensing off of anyways… If that’s a correct adjective or verb. Has the unhappiness changed dramatically, from 40%, 50%, to now 80%? Has it always been 80%? Is that maybe a good baseline? Like, mostly people are unhappy, and that’s a good thing, because – the reason why I ask that question is because innovation comes from angst. Unhappiness is a version of angst. Right? And so you can only innovate and change if you have angst, I opposed, to some degree.

Not necessarily –

Not so much only –

I mean, it’s one place. Greed also drives innovation. Like, “I want to make money. I need to invent something to make more money.”

Of course. But the greed may be causing the angst of the developers, Jerod. So they’re in the same bucket, basically. The angst is there, therefore developers push, organizations push, products change, innovation happens… The new Amazon occurs. Because.

I don’t have past Stack Overflow survey data. Do you, Abi?

[00:20:10.21] Do you, Abi?

I don’t. No.

Dang. Someone in the audience is like “I’ve got it, but I can’t talk to you.”

The way they asked that question - I wonder if that’s how they’ve asked it before. So I’d be curious if we could –

Can you restate that question? Because I think that’s a good point, is the question and the only answer – this is multiple choice. This is not an open-ended question of why. This is a scoped response. And so the 80% is extrapolated from that scoped response. Can you restate the question and the options?

Yeah. “How satisfied are you in your current professional developer role?” “Not happy at work”, “Complacent at work”, “Happy at work.” As someone who spends a lot of time on survey design, I do see a few potential issues with the question. So they’re asking about satisfaction, but then the responses are about happiness, which - satisfaction and happiness are really different constructs. That middle option is complacent… So it’s not happy, happy or complacent. But complacency - it’s not really the perfect middle between not happy and happy. I think it captures the essence of in between not happy and happy, but it’s not necessarily the perfect middle.

So it’s an interesting way that they’ve asked the question, because - is it measuring job satisfaction, or happiness…? Happiness I think is really hard to actually measure. So I think that’s they worded it around satisfaction. There’s a lot of literature about how to actually measure happiness. There’s entire fields where they’ve spent years trying to figure out how to measure happiness, and what happiness is. And usually, happiness is the sum of moments of feeling happy. If you took the day and divided it up into however many minutes, in each minute, or how many times throughout the day did you have a feeling of happiness, as opposed to non-happiness? And that’s kind of how you get to how happy you are. Rather than a point in time, a reflection of happiness, which is pretty difficult. So anyways, I’m nerding out here a little bit on the survey design…

Well, I like that, because you have experience that we don’t have in trying to like craft those so that they are optimal… Because you only have so much time and opportunities to like pull somebody for their thoughts. And if you pull them out incorrectly on accident, then you’re kind of wasting everybody’s time.

Let me rant for a split second here about Stack Overflow and URLs, okay? So first of all, I appreciate y’all do the survey.

No real hate here. But. I was trying to answer Adam’s question, which was “Do we have past year’s results?” So I went to this year’s results, survey.stackoverflow.co/2024/professional developers, found the link to that survey question. And then I went to the URL bar and I changed the year from 2024 to 2023. “404 page not found.” I mean, come on, people.

Respect the URL structure.

This is like what address bar hacking is all about.

Come on…!

Help us get to things in a way that makes sense. I just appreciate good URLs, and that’s not a good one. Anyways, that was my mini rant, because I was going to have answers for you, Adam. I was going to have last year’s answer to this question.

I wanted you to have answers so bad.

I don’t have it, man. I don’t have it.

Maybe ChatGPT or something else might have a hallucinated version of an answer. The reason why I think you’re nerding out on that and camping out on the semantics of the question and the response is because certainly it corners the person. It forces the person in response time. At the same time, there are probably lots of questions in the survey. So they could be experiencing cognitive overload at that particular question, while also being slightly unhappy for their day. They may have measured their happiness moments in that day and like “You know what? I’m unhappy.” Not saying it’s skewed, but it’s important to scrutinize the question and the offered options as a response, because that is what the sentiment is drawn from. And so if it’s skewed or not so much poorly worded - I would prefer you to say that, Abi, versus me, because you’re the professional at crafting these, in quotes, surveys… Just kidding… [laughter]

[00:24:36.26] Poorly designed…

[laughs] Because that’s really important. The way you ask a question and the options you offer is where the sentiment comes from. And if it is ambiguous, or it’s not super-clear, it’s clear why the answer is potentially skewed. And so to understand the efficacy of the answer set based on the question - I think that’s what’s worth scrutinizing.

Alright, real-time follow-up… I used their user interface to find last year’s results. And as far as I can tell, they did not ask this question in 2023. Maybe it was just one year they didn’t, but we did not have last year’s answer to this particular question. That’s not why it 404-ed, okay? They still changed their URL structure, but… Had it stayed the same, it still would have 404-ed, because they didn’t ask that question. So unfortunately, we can’t really go back and say which way is it trending, or is this an anomaly or anything like that.

Okay.

I want to talk about this idea of how can people talk about these problems in terms of dollars.

I would love to hear this, yes.

Yeah. So that’s something we’re working on at DX. It’s funny, Adam posed the question earlier why aren’t people fixing this… Because that’s the same question that we ask, and our customers that we work with ask too, when they surface all this data. So… Many years ago, I read this book called “How to measure anything.” Have you guys read that book?

No.

Okay. The topic is, as the title implies, how to measure anything, and in particular, how to measure things that are seemingly unmeasurable. So when we talk about what is the dollar ROI, or interest rate, or cost of technical debt and poor developer experience - just a few minutes ago we were essentially calling that unmeasurable. In this book they talk about how anything – if you want to measure something in terms of dollars or cost, that you can really do that with anything as long as you take something intangible, like technical debt or developer experience, and then you correlate it to something objective, or something monetary.

So an example of this would be the DORA metrics and the DORA report, which I know you guys have followed. So what they essentially did is –

Give us a primer for those who aren’t caught up on that. What is DORA?

So DORA is the DevOps Research and Assessment. But since I want to say 2013, maybe they’ve been publishing an annual report on the state of DevOps. And right now we’re talking about tech debt and developer experience, but eight years ago people were talking about DevOps, and “Hey, what is the ROI of investing in DevOps?” So it was the same problem; history repeats itself.

And what they did was they said “Here’s some ways we can measure DevOps.” So it was metrics like MTTR and lead time, deployment frequency… So they said “Here’s DevOps.” And what they did is correlate it to companies’ profitability, stock returns and increases, EMPS scores… And by doing that, they were able to “prove” and show the dollar ROI, “Hey, companies, when they invest in DevOps and get X percentage better, their stock prices tend to be X percentage higher.” They tend to be X percentage more profitable. So it wasn’t perfect –

[00:28:26.10] Yeah, that seems a little bit brittle to me.

A little bit brittle… Well, let me tell you what we’re doing with developer experience. So we have this construct of what is developer experience. So we have our version of what Stack Overflow has here, where we have – it’s called the Developer Experience Index. So it’s 14 of the top drivers of developer experience. So we say “Okay, that’s how we measure developer experience.” Then what we’ve been doing is correlating that measure to different outcomes. And one of them is actually self-reported time waste reported by developers. So how much time do you – it’s a series of different questions we ask about. How much time do you lose each week? How much time is wasted each week due to inefficiencies?

And when we correlate the two, we found that a one-point increase in the developer experience index score, which is the average of these 14 different areas of developer experience - a one-point increase in that score, so a one-point increase in developer experience translates to almost a 1% decrease in time wasted. And so - again, this isn’t perfect. You could call this brittle as well.

Well, I think it’s a little bit better, because you’re directly asking the people.

Yeah. And it’s more direct to dollars. It’s not like stock price, which is a little bit of a leap. A lot of things – so many things affect the stock price. So using this approach, folks can say “Hey, if we improve developer experience by X points, that translates to X percentage reduction in waste, which translates to X amount of dollars.” So that’s how we’re approaching it right now.

I like that approach. I think that time waste is reported by the actual people wasting the time… And so it’s probably relatively reliable. Of course, there’s always trolls and thoughtless respondents, but you can’t get around that.

Poor estimators… [laughs]

Yeah. Yeah, “I’m totally just being inefficient because of all these other things. It’s not me. It’s you.” There’s that whole – I mean, maybe you just account for that in your numbers, but… Yeah, if you are saying technical debt, complexity, bureaucracy, whatever it is, all these factors, ultimately for the business are costing money, slowing things down, wasting time is really a decent measure for that, like how much time is actually being wasted. And so if you can track that against this DXIX? What’s this thing called?

DXI. Developer Experience Index.

…at the same time… I don’t know, it seems like a pretty decent approach. Is that bearing fruit?

It is. Yeah. I mean, we can’t think of any other way to do it. I think the feedback we get is this is great. Like, if we can make this an industry standard, then my CEO is going to buy it. But there’s still an education and marketing gap there, where folks – what I just explained to you; it’s hard to get that across in like a five-minute executive summary.

Right. Do you do an annual or semi-annual survey for to the public, like Stack Overflow does?

No. We should. And we already have the data, because we are already – we’re surveying hundreds of thousands of people.

Right. The nice thing about this particular measure or this combination of measures is that if it could be somewhat generalized and made public, it’s now a tool and a resource for people who don’t have those quantitative metrics inside their company to say “Look, this stuff really matters. Look what it did for Walmart and these important companies. It’s moving their bottom line. It’s making them more productive. And they’ve they proved it out over n years. And so if that’s public information that I can take to my leadership and use that, and convince them that “Hey, let’s call a feature freeze”, or whatever it is that I’m trying to get done, right?

Yeah.

Break: [00:32:39.18]

My idea for you, Abi, is a growth hack.

Let’s hear it.

When you do this, it would make sense – if I were you, this is probably how I would, at least, consider it. This is not a perfect one-to-one plan, but… Grain of salt, okay? This Stack Overflow survey is obviously popular. We’re talking about it. The results are shared and examined and analyzed by many… It’s respected. It’s got a core audience. If you have similar data, I would release whatever you’re doing, or whatever data you can do, around the time of this survey’s announcement, and to some degree, Venn diagram. Number one, you associate the brand for DX with a very beloved, mostly beloved brand, Stack Overflow. Some love, some hate.

I thought you were talking about WordPress. [laughs]

Oh, yeah. Yeah, exactly.

Not yet. Not yet. And then you can draw correlators between the questions and the data they siphon from that question, and then the question and/or dataset that you have, that correlates and Venn diagrams across the two. One, to keep them honest - and not so much that they’re not honest, but… To keep this survey, which - all surveys have an optimization opportunity.

Yeah, they’re all flawed.

Right. There’s no perfect survey. And I think you almost better off the entire community, because you give not one dataset, but two datasets. So how true is this? Your findings and cross examination and Venn diagram may say “Well, this is actually pretty close to true, because we have corollary data and we can corroborate these findings.” Second, you get to feature things like DXI, and you get to have an opportunity for “Now way more people know about DX”, and now find the benefit and/or interest in your beliefs, which is this DXI index being such a core thing to you all. One, that’s the idea. How do you like it?

I like it a lot. I’m writing it down. [laughs] I like the cross-examination piece.

I do, too.

And then I would say the second thing - and maybe this gives a foundation to a foundation, which is in order to have an organization support or adopt this DXI, this developer experience index, what do they need to have in place to get to that point? What is a mature data-driven organization look like that has the ability to actually index this index, and have that for themselves?

Right.

Yeah. I love it.

That’s a question.

Taking notes.

That was a question.

Oh, what do they actually need?

I like the question, but that was also a question.

[laughs]

Yeah. I mean –

What’s the foundation to get to this point, to have a –

14 metrics, right? Is that 14 metrics?

Yeah. 14 different metrics. So we haven’t open-sourced it right now. It’s proprietary.

You’ve got to, man.

Yeah. I mean, that’s actually one of the biggest strategic questions I’ve been wrestling with for over a year and a half.

I get it. [unintelligible 00:41:25.09]

Do we open it up, or do we keep it proprietary?

Well, when you have the opportunity to become the index… I think – I mean, obviously you know way more about your business than I do and how important it is internally as a proprietary thing… But I can see a huge upside in the open-sourcing of it.

Yeah. Yeah, agreed. And I think we can open-source it while putting like a copyright on it, so you’re not technically supposed to use it for a commercial, within a – like, Walmart can’t actually use, deploy it…

[00:42:03.01] Right, right. Yeah. I mean, we can get into the licensing discussions, and we’re happy to; we do it all the time.

Yeah, yeah.

And depending on what it is, open source might not even be the right word. Like, maybe it’s creative commons, maybe it’s… But you could still hold trademark and copyright against it. Like, DXI could be a trademarked thing, but also how you go about doing it and how others can go about doing it - you can just let that stuff loose. You don’t let go of the copyright, but you just let other people use it. So… Anyways.

And they can’t call it DXI. It is a product.

Right. You trademark the DXI, and –

Right. [unintelligible 00:42:31.27]

It makes sense.

They can be DXI-compatible, or whatever. But that’s in the weeds. You were talking about the 14…

Yeah. I mean, to deploy it, they – we have the survey items, the measurements for those 14, and they deploy our platform. That’s why it’s proprietary, because they can’t do it without our product.

Yeah, right now they’re buying it from you.

Yeah, DX. But in terms of like who it – you know, we work with the Pinterest, the Dropboxes, the Netflixes… The bleeding edge tech companies. And we also work with – I mean, this isn’t to diminish these organizations, but companies like Pfizer, P&G, Tesco, Bank of New York, BNY. So I think what we’ve seen is the DXI works in all kinds of environments; not just the bleeding edge tech companies, but more legacy, traditional enterprises as well.

Sounds pretty cool. So you have found, then, if you have a DXI - which a lot of these companies do, via deploying your guys’s proprietary platform - and you’re tracking time waste, you’ve found that there is an inverse correlation between the two, that is measurable and repeatable and reliable.

Yeah.

That’s pretty powerful. I mean, we all know it’s true, but like actually proving that it’s true is a whole different thing.

Yeah. So we did a meta analysis, which we’ve published, and I’ll give you guys a link to that… So we actually have a developerexperienceindex.com. I have that vanity URL.

Nice.

So long.

It is a little long.

Gosh.

But anyways, so the R value was point – I think it was like 0.5. That’s a really strong relationship between developer experience and time waste. So then on an individual company basis, we can look at their relationship. We just run that correlation for an individual company, and we always see a moderate to strong relationship at any given organization as well.

And do you find that it follows Pareto’s principle as well, in terms of effort? Like, 20% of your effort gets 80% of your results? Or as you continue to improve your DX, is it trailing off or not?

That’s a great question. Like, do we see a different relationship at different bands…

At the edges. Yeah.

We haven’t studied that, but I’ll add that to my notes as well. That’s really interesting. Yeah.

That’ll be worth knowing. I think it’s logical that that would be the case; in almost any effort, at a certain point you’re squeezing the radish. But what’s the sweet spot for companies, where they can put in this much effort into their developer experience and get that much out… I think that would be worth knowing.

Yeah. Absolutely.

Can we dig into these 14 drivers? It is out there. Can we talk about them at least?

Yeah. Did you go to developerexperienceindex.com?

No, I just googled DX, and then DXI, and it landed me on this page that – you can tell me if this is accurate. The number one –

Yeah. That’s the white paper.

“The one number you need to increase ROI per engineer.”

Yup.

And about two scrolls down, you dig into figure two, which talks about the drivers and the outcomes.

Yeah.

[00:46:01.03] And so I’ll do the work for you, if you don’t mind… The drivers are deep work, dev environment, batch size, local iteration, production, debugging, ease of release, incident response, build and test, code review, documentation, code maintainability, change confidence, cross-team collaboration, and planning. Those are the drivers. Those are the 14 dimensions. And those correlate to five outcomes: speed, ease of delivery, quality, engagement, and efficiency. Dude, that’s a good map. That’s a really good map to maturity in an organization, a dev organization… All those things on the driver’s side are really good. Like, what is my maturity level, and what is my – I don’t know how you would describe it. I’m trying to think on the fly here, but… How good am I? How good are we at these drivers? And then the correlations are obviously awesome, with the outcomes: the speed, the ease of delivery, quality, engagement, efficiency…

Yeah.

But that’s a good map. I like that.

Yeah, it’s taken years to arrive at those 14.

You could almost map software as a service on top of that sucker. I mean, there’s an offering for each of these drivers. I mean, there’s a whole industry around production debugging, incident response, code review etc.

Yeah.

I just find that interesting.

Which one, which SaaS service correlates to change confidence most? That’s the one that stands out to me a lot.

Like, change is hard anyways, and the confidence in change… You could be a senior engineer and feel good about it. You can also be a junior engineer and feel good about it. But what gives you the confidence? How do you measure that with a service, a tool or a SaaS?

Yeah, change confidence is a lot about how easy it is to actually test a change, to get feedback on a change. So I think everything from cloud dev environments… All of these things kind of interrelate, but – so like cloud dev environments that allow you to like quickly spin up a staging environment for just your change to easily manually test stuff… Obviously, things like test coverage… Now AI is coming into the picture there with helping write tests, or even manually QA your work…

So yeah, change confidence is about like when you make a code change, are you kind of like YOLOing it when you ship it, and just hoping it works, or do you actually feel confident that when you make a change…? It also just has to do with code quality. Like, if you make a change in one area of the code, is it a house of cards, or cascading effects, and everything?

So a lot of things go into it, but it’s ultimately about if developers feel like they can actually make changes, or are they just duct taping things and hoping that it works when they deploy it.

Yeah. Another new-ish feature of a lot of cloud hosts are preview branches. That’s another way where you can get change confidence. Netlify, Vercel etc. they’re providing a place where you can have your development branch, and it can be constantly publishing to a preview page, on a sub-domain, on a website. And so now you can both look at it yourself in production-ish, and then also send it to your QA team, or to your boss, or whoever your customer. And I think that definitely helps with change confidence… Because previews are nice. But yeah, there’s so many tools that overlap in these things as well.

Documentation… So if you’re modifying code, can you even understand how that code works, so you’re confident in changing that little bit of code? So yeah, a lot of things. All these factors interrelate. Even something like batch size, which is about incremental delivery. Like, if you’re working on huge changes, huge PRs, there’s so much more risk. They just have a lot more surface area. So if you’re delivering incrementally, your confidence is going to be higher for each unit of change.

[00:50:00.15] Right.

Is this your next big thing, the DXI?

It’s been in the works for a while. DXI is one of the big things. The other is the Core 4, which is the other – if you go to that Research tab…

That one rhymes, so you know it’s true.

DX Core 4. Yeah. So if you go to like that Research tab on our website, and there’s the Developer Experience Index, and above it the DX Core 4 is something else we’ve been developing.

And that is speed, effectiveness, quality, and impact, right?

So that’s the outcome of this, but the real problem we’ve been trying to solve is – I think last year I came on here, Adam, we talked about the DevEx framework I’d published with Nicole Forsgren and others. So ever since we’ve published that, people have been coming to us and saying “Hey, Nicole created DORA, Nicole and Margaret created the Space Framework, and then you, Nicole and Margaret, created the DevEx framework. We have three frameworks now for telling us what we’re supposed to be measuring in our organizations.

So to sum it all up, what should we actually be measuring?” [laughs]

Gotcha. So you add one more… Add one more and you’ve got the Core 4. [laughs]

This is the one to rule them all. This is the one framework to rule them all, because –

[unintelligible 00:51:21.10] replace the other three.

Not replace. This encapsulates all of them.

Well, of course.

This combines them all into one framework… Because – yeah, everyone would ask us that question, and the way we would always answer that question was “Well, it depends.” I was just talking to a CEO at a big tech company, who said “I was talking to Nicole, and I asked her, to sum it all up, what should we measure? And she told me “It depends.” And I get that it’s situationally dependent, but it would be really valuable to have something out of the box, and standardized, that we could benchmark across other companies, and really have a way forward.” And funny enough, I’ve had that same experience. I was talking to a CTO actually at Capital One, who asked me “Hey, I’ve been following your research for two years… So just tell me, what should we actually measure?” And I said, “Uhh, it depends. We can do a consulting engagement on that until I figure it out…” But having a starting point that is out of the box is really valuable. So that’s what the DXCore 4 is.

I love that response, and as a person who has to deliver that response frequently, my next response is always “I have to ask you 20 questions to answer that one question.” So you need to give me more time. If you want my true answer, the only way I can know what to respond with is to ask you several more questions. And those questions may lead to even more questions… And so if you trust me - you’ve been following my data - give me a little bit of your time, and I will answer those questions by asking tons of questions.

The problem is no one has time.

Well, they all want the silver bullet, right?

Yeah…

Of course.

“Give me a yes or no answer, Abi.”

Yeah. They want to go to the doctor and get the diagnosis, not go to the doctor and then have 16 follow-up appointments before you get the diagnosis.

And I get that. I mean, if you were a time waster, then that’s different. But you can only answer that CEO’s of top tech company’s question well if you understand more about their specifics of their business. What are their particular drivers?

Not anymore, baby. Now he says Core4.

Core4. Yeah.

I think the Core4 provides a pretty good answer. I mean, we want people to customize. This isn’t like “Hey, do this and do nothing else.”

Oh, I like this, actually.

Core two, core three, maybe…

[00:53:51.10] But the DX Core4 does – and now we’ve started rolling it out to people, and it’s landed well. I actually asked the CEO, after I showed him – I showed him the Core4 right after hearing about that experience that they had talking with Nicole, and said “Hey, we’ve been working on something for this.” And I asked him, “To you as CEO, does this seem right? Does this seem correct? Does this seem like the right way to be thinking about and measuring developer productivity?” And he said yes.

Wow. I was going to give you an idea, but that might actually be the answer… Because rather than say “It depends”, what if you said “We have a survey that takes you five minutes to answer”? Instead of saying “It depends”, you say –

And it spits it out. It spits out your customized Core4.

Right. This is your on-ramp; your specialized, personalized on-ramp.

Yeah.

Because I’m sure you can take that consulting session and to some degree distill it down into something a CEO who has very limited time can answer in 5 to 10 minutes, right? “Hey, I don’t have an answer for you in this moment, but we have a very fast 10-minute or less…” It really could be 15 if you want it to be, but for most people it’s 10. “And if you answer these questions, I’ll know exactly how to help you.”

That’s like some of those personal wealth front and betterment, some of those robo investment advisor platforms… You go through like a three-minute wizard and then they tell you what your investment portfolio should be.

Totally.

So I like that.

More ideas for you, Abi… Two you’re taking away from here.

Yeah. Yeah. Lots of good ideas.

Write that one down, Abi. Write it down. I’m just kidding.

Write it down.

[laughs]

Can we dig into these a little bit?

Sure.

The core four: speed, effectiveness, quality, impact. You say those are outcomes, not necessarily –

Those are the dimensions. Those are the categories. Think of them as your stocks, bonds, cash… To use the stock portfolio analogy. You need that balance. Because if you only measure speed, and everything else goes to crap, you’re not doing it correctly.

Right. There’s your move fast and break things right there. Like “We’re moving very fast, but we are breaking [unintelligible 00:55:55.22]

Not breaking things. Yeah.

So a balanced portfolio. This is a nice metaphor.

Yeah.

For each of these, you have a key metric. This is something that you’re going to track. And then you have secondary metrics… So there’s some balance there as well. But for speed, the key metric is diffs per engineer.

Yeah.

And I don’t know if I – I might take issue with that one. Tell me more.

Yeah. I mean, probably the last time I was on this show with Adam I was probably dissing that metric, PRs per engineer.

Opinions change, man. Opinions change.

I’m a politician. I flip flop on the issues.

Yeah, it just depends on who you’re talking to.

Yeah. So it’s been a journey for, just for me personally on this topic, because - the whole reason I actually got into spending six, seven years on this whole problem space was because I felt like metrics like diffs per engineer were reductive and not correct and not helpful. But one of the things that the core four optimizes for is – so we work with a lot of technical leaders, engineering leaders. And as we were talking about earlier, one of their big challenges is talking about rationalizing investments in developer productivity in a way that the CEO and CFO are going to agree to. And to do that, you need a shared definition of productivity that your CEO and CFO agree with. And to achieve that, I’ve found that you do need some type of output measure. Like, we’re not at a point in human evolution yet where most CEOs and CFOs are down with this idea that a developer experience index is the one metric that matters for understanding the maturity or effectiveness with the organization. A lot of CFOs and CEOs still think – I mean, there’s Fortune 50 companies still measuring lines of code, benchmarking that.

[00:57:57.13] So we’re still at a point in human state of the art around software engineering where output measures need to be a part of that conversation. It needs to be part of the way you’re framing developer experience and developer productivity if you want the people you’re pitching this to to actually fund it and believe it and buy in. So there’s a marketability optimization here. That’s one of the reasons PRs per engineers is in here. But the other reason is, we have come around to talking with a lot of companies like Uber, Microsoft, top tech companies where they use this metric as an input. It’s not the sole metric. They’re not performance valuing engineers based on this metric. But in aggregate, they are looking at this metric as an input to understanding how developer productivity is trending and compare it to other organizations. And it’s not useless. It is a useful indicator in aggregate, and that’s why in the framework, in the core four, there’s an asterisk that says “Not to measure this at the individual level.” So this is only to be looked at an aggregate, team, group, organization level and benchmark that way… And we’ve found it more useful than not.

So it says diffs per engineer, though. Diffs per engineer, then asterisk.

Yeah. Not at the individual level. [laughs] So the metric is normalized… So you’re looking at aggregate divided by the population. But in terms of like visualizing or reporting this, you’re not looking at a list of people, you’re looking at teams and organizations.

Right.

I do see the contradiction there though, yeah.

Well, certainly at an individual level, at face value, it seems contradictory, but it does make sense. Maybe you could reword that to say like “Averaged across”, whatever.

Yeah. I’ll write that down.

Oh, man. So many good notes for you here. At an individual level, certainly it’s a bad measure. The problem is it becomes a bad measure. That’s Goodhart’s law. Once everybody knows that that’s what’s being measured - well, we all know how to play the game. I mean, it’s lines of code moved to a slightly larger batch, you know?

Yeah.

And so I can criticize that one. I can also criticize lines of code, I can criticize features, or tickets… They can all be criticized. But then at a certain point you’re like “Well, what can we actually do then, if everything sucks?”

Exactly.

“We’re going to have to pick one and go with it.” And I guess if the industry is somewhat standardizing around that, then it’s a decent compromise.

And I think there’s more we can do. I was just talking to a company – actually, I’m working with a Silicon Valley tech company, and all the other core four metrics were quite a bit below like P50, in industry peers, but diffs per engineer was higher. And this was bad for them, because they’re trying to show to their executives that they’re behind peers, so they can get funding to make improvements.

Sure.

So we were just trying to dive into the data, like “Why is your diffs per engineer inflated, even though clearly, empirically and with the other core four data points you’re not like a high-performing organization?” And we couldn’t really figure out an answer. I mean, there was a lot of speculation… It’s just, there are a higher number of like config changes, small PRs that aren’t real changes… But every company has that. That fuzziness should already be kind of accounted for in our benchmarks.

So that led to this idea, could there be a weighted metric? So you’re actually – because not all diffs/PRs are credited equal, like we talked about. Some are one-minute changes, some are one-line changes that are actually eight hours, some are 800 line changes that are two minutes… Like, how do you –

Right.

[01:01:51.13] So if we could apply some kind of weighting to like bucketing all these diffs and PRs… Almost the same way we do estimation, like T-shirt sizes or something like that. I was thinking, could we use gen AI, LLM to basically automatically try to categorize based on the title, the description of the task and the code changes? Was this a big change, or was this actually a small change? And then we could get kind of like a weighted number. That would be an improvement to the signal you’re getting out of like an output measure like this.

Yeah. Even with a confidence score alongside that would be really interesting.

Yeah.

There’s still challenges with that, because the amount of change does not always correlate to the amount of effort.

Yeah.

You can work an entire week on finding a bug, and then you’ve found it, and it’s a one-character change. And you’re so exhausted by then that your commit message says, “I’ve fixed it”, or something, you know? So the LLM just doesn’t have much to work on.

Or just “Boom.” An emoji.

I guess if you can just say “Well, it’s fuzzy, it’s not going to be 100%…” It’s better than merely measuring – so did you come to… My only guess would be like a culture of small diffs, or a culture of –

You can’t figure it out.

Like, why are they so much higher on that one metric? You haven’t figured it out yet?

We don’t know, but they are definitely higher. And I told them, “Look, if I had a little bit more time here, I would take like a random sample of your 200 PRs, and then like random sample of other companies, and try to do what an LLM would.” I would like look at the titles and descriptions and try to figure out, are your PRs generally smaller, lower effort or size tasks than other companies? I mean, that probably has to be the reason… It’s an interesting problem, though.

My guess is you dig into that and you find there’s some sort of scheduled pull request that’s just like changing something that should be in the database, but it’s not…

Well, here’s the thing. Here’s another twist.

Okay.

So the twist is we measure this two ways. We actually measure diffs per engineer self-reported, meaning we just ask developers on average, “How many PRs do you merge, that you were the author of?” And we look at their actual GitHub data. And for this company, both numbers were like within point – they were the same number, which is remarkable in of itself.

Honest people.

Yeah. I mean, that sounds fishy right there. What are the odds that they’re like that close?

Well, not exactly, but within like point two, three. Yeah. I mean, why wouldn’t it?

Well, I thought – okay, it all depends on how exact it is. If you have a vote and you have 99 to 1, you’re like “Okay.” But if you have a vote and it’s 100 to 0, now you’re like “There was some collusion here. Something happened.”

Well, maybe people looked. Maybe people looked at their own GitHub activity and answered the question.

Which is fair, because maybe they don’t know, and so they’re going to look at it. I’m just saying, if it was like exactly the same, then I’d be like “There’s something wrong with our system here.”

Yeah. It wasn’t exactly.

Okay, fair.

It was very close. So that does exclude bot-authored pull requests, for example. And both measures; both for GitHub, the objective one, and the self-reported explicitly exclude bot-authored.

I think you’ve got someone in that org who just doesn’t go to work, and they just have a bot that uses their own SSH key, and just every day merges their stuff. And then you ask that person how much they merged and they went and looked at their bot and they just guessed the right answer.

Yeah. Looking at outliers would be interesting, though the self-reported accounts for that, because there’s an upper bound. The top option is actually I think like 10; 10 or more per week. You can’t put in “I merge a hundred per week.”

You can’t be a self-reported 10x dev.

Yeah, yeah, exactly.

Right. What an interesting problem though.

Yeah, it was fascinating.

Break: [01:05:51.02]

I was thinking though, as you guys were talking about this, that measuring speed is an “It depends.” Because not every team can be measured speed-wise on the exact same metrics, which I think is why you have this key metric and then secondary metrics.

Yeah.

Yeah, round it out.

Because you have the secondary metrics to sort of back up and correlate to what the key metric speaks of. And the collection process is via systems… So collect the data from a Git repo or other intelligence platforms, and then self-report it.

Yeah, that’s a good suggestion. We didn’t look at the secondary metrics. We got really trapped in like “Why is this diffs per engineer inflated?”

Well, I almost wonder if the key metric is what swaps out. Because on one team, the diffs per engineer may actually be the primary driver of the data you’re trying to collect. In a whole different team, lead time, or processes, or deployment frequency is actually the better key metric, and the others are the supporting. I don’t know enough about your business how to do that, but…

Yeah, this makes me want to go look at the perceived rate of delivery, perceived speed; it’s one of those secondary metrics. The analogy here would be for like aerobic athletes heart rate versus perceived rate of exertion. Those are the kind of two… And there’s a lot of flaws in heart rate, because - I mean, just the altitude. You could be training at different altitudes and the heart rate’s different, even though you’re kind of doing the same load. Or you just wake up, you didn’t get as much sleep, so your heart rate is more fluctuating. So yeah, we really like that perceived rate of delivery. We literally just ask people to rate the speed at which their team delivers.

Like 1 through 10, just rate it, or…?

It’s like a five-point scale. It’s from something like extremely fast to slow. It’s the actual speed.

It’s in words, not like a one through five thing.

Yeah, very much inspired by perceived rate of exertion, which is on a 10. There’s 10 options for perceived rate of exertion.

I think so, yeah. Or perceived rate of pain… Have you ever seen that Brian Regan –

For medical? I think they use that in healthcare.

Yeah. There’s a great Brian Regan stand up where he talks about them asking him that question when he goes into the ER, and him thinking through what number should he say in order to get help as fast as possible…

That’s funny.

He’s like “Never pick seven. You’re always an eight.”

[01:11:25.11]

And here’s the full length, unedited clip of Brian Regan on this awesome bit. I was going to edit it, but I was thinking “Gosh, I would just edit this man’s comedy”, and I just can’t do that. So if you don’t want to hear the whole thing, skip to the next chapter. There you go.

And so it’s very much Goodheart’s law in a much funnier context.

Do you think, Abi, that your North Star with DX as an organization, what you’re trying to do, is to define a path to happy developers? What do you think you’re actually trying to accomplish? I mean, I know what you’re doing as a result of giving survey results, and this data, this formulaic and proprietary way to ask questions of an organization, how to disseminate this information and analyze it, that you’re trying to help organizations be optimized. But do you think the true optimization factor is the path to happy developers?

Happy and productive, right? So the Stack Overflow survey, once again, confirms this… People are unhappy because they’re unproductive, is another way to characterize the findings. People are frustrated because it’s hard to get work done because of their tools, systems, whatever. Therefore, they’re unhappy and they’re unproductive. There’s a lot of time being wasted here.

So I would say our North Star is helping every tech organization become the best version of themselves. And that has a different meaning to different people, but… I mean, I’m the CEO of a company, and we have engineers, and so the way I think about it is “With the people we have, are we doing the best we absolutely could be? Are we as good as we could be?”

[01:16:17.09] And we ran the DXI, and all these, the core four, and I’m looking at that, “How can we get better?” To another company, that probably just translates to “We spend a crap ton of money on engineers and we want to make sure we’re maximizing that investment.” Or it might mean “Look, our competitors seem to be creeping ahead of us. How do we go faster without just hiring more people?”

So lots of ways to tell the story in like a one-liner, but yeah, it’s about being good at building software. And as a result of that, people are also happier, because all research repeatedly shows that happy developers are productive, and productive developers are happy, as the Stack Overflow survey also shows.

I go back to the beginning of the conversation, which was 80% are unhappy… And what we’ve failed to ask was why are the other 20% happy?

Yeah.

Because I feel like if your North Star is productivity, but that comes as a result - generally, in my opinion, and I don’t know this qualitatively - is that you have productivity when you’re happy. And you can’t create/make happy developers unless you understand what makes them happy. So why is the 20% of the 80% that’s not unhappy, happy? What is going on? Why are they happy?

Yeah. I mean, whenever we look at this type of data, we’re slicing and dicing. I mean, you do see some – a couple things I could share… We do see some differences cross-culturally. For example, we tend to see higher sentiment around this type of stuff, at least self-reported, from populations in India, for example. We tend to see higher satisfaction with more junior developers, people who just don’t have a frame of reference yet on what is good. They’re new to the profession… So there’s certain things that, if we just looked at this data, it might be that the 20% happy are coming from certain countries, or certain levels of tenure and seniority… That could explain quite a bit of that 20. And some of them are probably legitimately in good situations, with good developer experience, and greenfield projects with no technical debt, where they’re really happy.

Yeah. “I’m in full control. I have autonomy. No one yells at me. I’m getting paid. I’m not getting laid off.”

Yeah.

“I’m not too old.” “I’m getting fired at 25 because I’m too old to code”, that’s the joke now. It’s just, you’ve aged out. “I’m 25.” No, come on.

Well, there’s another interesting data point on their survey, another interesting question, which is about coding outside of work. And if you want an indication of somebody doing something because it makes them happy, it’s something that they would do outside of work. And so the same exact work of developing, while there’s 80% unhappy at work, 68% of respondents said that they write code outside of work as a hobby. That’s like almost 70 out of 100 people. That’s a large number. And 40%, which - there’s some overlap there; these aren’t mutually exclusive - code outside of work for professional development or self-paced learning from online courses. So these are people investing in themselves, caring about getting better at what they do… And that’s kind of amazing.

[01:19:50.22] So we have this dichotomy of people who love to write software, generally speaking, and yet unhappy writing software inside of their organization. And obviously, you can look at your DXI and follow the 14, but the closer you can make your engineering teams feel like they’re doing their hobby… Think about how a hobby works. It is self-directed, first of all. So autonomy is huge. Most likely, unless they have a bunch of kids running around, there’s deep work involved. Like, you can lose yourself in it

You can go into the – I was going to say the garage, but if we’re coding… Well, maybe the garage; wherever it is that you write software, and just pound away at it for four hours without any interruptions, and really lose yourself in it.

A lot of these 14 metrics actually are manifest in hobbies. And so if you can – obviously, a business is a business, and so you can’t just be like “Everybody do what they want.” It worked for a little while for GitHub, till they got to about a hundred, I think; a hundred engineers… I was there for the ride; not at GitHub, but here, podcasting and paying attention and using it as a product, and going to conferences where Zach Coleman was traveling around and talking about their engineering-led development… And everybody pretty much just works on what they want to. That worked for GitHub for a long time. Long meaning in years, not in employee count; up to a hundred is not a large engineering team. They’re way larger now. But at a certain point, that thing falls apart, because there’s work that needs to be done that nobody would just naturally pick, unless it was assigned to them and they’re paid to do it. And so eventually that does. But if you can make your engineering team feel at least approximate like they would be doing this as a hobby, then I think you’re going to have a lot of happy programmers.

That’s how I’ve heard this described… You know, how can you kind of get the same feeling of joy and flow that you do when you’re working on a side project? How do you get that same experience while working in your job? How can you recreate that? And if we could do that, we would unlock a lot more productivity. We would get a lot more out of engineers working at our companies.

Yes.

So I think that’s a good way to think about it.

And a lot more happiness too, which - everybody wins there. Like, there’s no losers. These drivers, these 14 drivers - have you ever done a survey where you’ve like asked developers to rank order those drivers?

We do that. That’s already – yeah, for every organization we work with, that’s one of the… So first, we capture the data on it, and then based on how they stack up on each of the 14, we give them “Hey, here’s how you answered these 14. Now, out of these 14, what are the top one to three that would most benefit your productivity?”

Mm-hm. Do you find that to be pretty subjective, or are there certain ones that always float up to the top?

There are definitely certain ones that tend to float more toward the top.

Such as…?

Documentation.

Really?

For sure, yeah. What’s interesting is the Stack Overflow kind of – technical debt is not one thing. All 14 of these things - well, minus maybe two or three of them, are actually types of technical debt. So documentation is actually a form of technical debt. Complex code is a form of technical debt. Slow CI/CD is a form of technical debt. So all the technical factors do tend to float toward the top, actually… But some of the cultural facts – you know, cross-team collaboration, like delays due to different teams having to coordinate with one another is also, I’d say, a pretty common theme… It makes sense…

Yeah. Is that across engineering teams, or product teams, or is that like ops versus devs?

Yeah, good question. These are the people – my response just now was deriving from how engineers report the friction. So from the perspective - for developers, yeah, waiting on other teams.

Right.

Which could be cross-functional, or it could just be other engineering teams that they have; different services or whatnot. So that tends to be a big area of friction.

Queues. It’s always about queues, right?

Yeah.

CI/CD is a queue, being delayed or whatever is a queue…

Yeah.

[01:24:18.23] “I can’t work on this until you work on that. I can’t work on that until you work on this. We can’t deploy that because of this.” It’s all queues.

Waiting on code review.

That’s right.

Then there’s deep work. That’s just meetings, right?

Not just meetings.

Less meetings, please.

No, not just meetings.

[laughs] What else? Slack.

Yeah, it could be people actually asking you questions, it can be code reviews, it could be “Hey, can you fix this quick thing? A customer asked for this thing. Can you take care of it?” It could be support, it could be incidents… So it’s much more than meetings. Yeah, that’s something we see a lot; companies just say “Oh yeah, we just need like no meetings Wednesday and it’s problem solved”, right?

[unintelligible 01:24:59.01]

And that’s really the case. I was actually looking at our DXI, what our top ranked areas were.

What matters most to your teams.

Well, yeah, what do our developers say are the biggest areas that should actually be improved? And for us, it’s actually that code maintainability. So the ability – how easy it is to actually understand and modify code. It’s actually also a clear project direction. So the projects, they work on having clear goals and direction. And it’s actually that batch size, which we’ve since renamed to incremental delivery. Are you working on kind of small, continuous changes, as opposed to large ships? Those were the top three for us.

Well, those first two are driven by leadership, aren’t they, Abi?

[laughs]

I was thinking you’re showing us cards here. I was like “Dang…!”

Yeah, yeah. Well, I will say this… Our DXI score is three points above P90.

So you’re sitting pretty.

Yeah, we’re sitting really pretty. But yeah, even then, there’s always room for improvement. And actually, I’m looking at the data now, and actually, our clear direction.

He’s smiling, y’all. He’s not upset. He’s smiling.

Yeah, those top three I just mentioned, actually, are not above P90. Those three specifically. Yeah.

Okay. So there’s some room for improvement here.

Yeah. And same with code review turnaround, actually; it’s not above P90.

Here’s the better question, really… And I know you were poking fun, Jerod, but…

I was.

Yeah, and that’s totally cool. And he likes it. You can’t change what you don’t measure. Right? So now that you have this index, and now that you have this awareness… Even as a leader, you couldn’t change it before if you didn’t know it. But now you have awareness, your team has awareness… Your team that is answering these questions feels heard. If you’re going and making change, you say “Hey, because of these results or because of these findings we’re getting from our DXI score, we’re improving these things in these ways.” And the morale changes, the ability to speak to leadership and influence change changes… All those things really come into play.

Yeah. Now, this gives me a lot of reassurance as a leader, actually… Because I wasn’t sure before we ran this last – we call them snapshots. This last kind of benchmarking survey, and data collection exercise… I really wasn’t sure. I was very pleasantly surprised by how good things are right now. I mean, that’s what I would expect out of myself as a leader… But I wasn’t sure. Am I just thinking we’re good? Is it actually terrible working here? Or are we actually as efficient – are we actually kind of at that high level of efficiency that I would expect out of the way we do things here. And we are pretty efficient, so it’s good to see.

[01:28:00.05] Well, we all want to think that we’re doing well in that which we set out to do well… But the worst place to be is to not be doing well and not know it. So at this point – of course, you are reassured because overall you’re doing quite well. But even if you weren’t, at least then you would know, “Okay, I thought I was doing well, but I obviously have some things to fix.”

Now, if we picked one of those three - let’s not do commit change size, or whatever that one is. Let’s go with the other two, and say “Okay, these are room for improvement.” So pick one of those two and just spitballing, what could you, Abi Noda, as a leader, do today, tomorrow, in order to meaningfully move that at your next snapshot? Do you have any ideas?

Yeah… I mean, the project direction - that’s on me. And you were asking earlier Pareto’s principle, but like some of these are tradeoffs. Because yeah, we can improve that, but that would involve a little more process, which would cost time and money… And given that it’s already actually very good, is that – it’s more like something we want to keep in mind and be aware of, so we can just lean in a little bit more there. The code maintainability - that’s already something we’re really focused on. So that was like validating that that’s something we need to continue focusing on.

And how do you focus on that? What are your actual tactics?

Good question. Having clear patterns… I mean, just really pretty strict code review… And not just code review, but just making sure – I don’t know if you’ve seen Addy Osmani’s post “Code is like a love letter to the next developer.”

It rings a bell, but…

Yeah. So that’s in our onboarding doc for engineering. It’s like “Look, the only thing that matters here when you write code is how easy is it for the other people on the team to understand that code?” And we really try to make decisions on how we build things around that principle. So it’s good to see that then reflected in the data. People are saying that it is easy for them to understand and modify other people’s code here… So that’s one of the ways. But yeah, it’s a lot of hands-on driving that principle forward. I’ve vetoed a lot of technical decisions, introducing new technology based on that principle, that every new technology we add, every new pattern we add is something else someone else has to learn, and this is going to slow them down.

I’m not sure where you scored on speed, but I assume it’s pretty well, considering these were the only three that weren’t great. Have you ever considered compromising a little bit of speed? Like, there’s your tradeoff, is like “Let’s slow down a little bit.” Because a lot of times just time to breathe and refactor and maintain actually improves code maintainability. If you have maybe your snow leopard moment, for instance. I’m not saying do a feature freeze or anything, but like small bits.

So get this. I’m looking at the data… So when I look at our core four, we are above industry P50 for speed, effectiveness and impact, but not quality. When I look at some of the secondary metrics, so perceived speed and perceived quality, we are also above P90 on all of them, except quality. So yes, I think we do sacrifice a little bit of quality for the sake of speed. I mean, the data shows that very clearly. Now, whether that’s a problem or not…

That’s up to you.

…is an interesting question.

Yeah, exactly.

[01:31:40.05] As a startup, I’d much rather be P90+ on speed right now, than… Because we have quality issues, but they don’t affect customers that much. That’s the secret about our quality problem. We do have quality problems, but I’ve actually had the principal on the team like “Look, we’re not building payroll software here.” Like, when we have a glitch in a report, it’s not hugely disrupting or impacting our customers’ businesses. And if we can quickly resolve it, then we’re good. So that’s a principle we have here. We have really fast recovery… But yeah, we do have not great quality, objectively speaking.

Yeah. Not abnormal for a company in your situation, I don’t think. Do you have happy developers?

It’s a good question. So we don’t measure – for the reasons… I kind of shared some concerns around measuring happiness. We don’t measure happiness…

Do you have productive developers?

Well, so yeah, we do, based on – I mean, our DXI score is through the charts, as I was saying. But I mentioned earlier, we do measure attrition risk. This is interesting. I haven’t actually looked at this…

Uh-oh. This is a real time.

Real time demo.

Okay. I see. Okay. I mean, this I can’t share out loud, but… So for risk of attrition, we look at it very similar to like a blood test that you might get at the – so when you get a blood test, they tell you “Here’s the healthy range.” If your blood pressure or cholesterol is within this nanomilligrams to this, you’re normal. So with attrition risk, the normal range - and I maybe haven’t had coffee this morning, but I think it’s 7% to 10% is the healthy range. So if your attrition risk is 7% to 10% of your organization has signals of being at risk of attrition, that’s normal. And I’ll just say we’re at the high end of the normal range… But I’m looking at the data; our reporting will tell you where that kind of risk of attrition is. So I’m looking at it now and I’m aware of what is going on here.

[laughs]

Oh, gosh…

It’s good.

He knows the inside story.

Yeah…

Well, how large is your team?

It’s around 20 people.

Well, I mean, just roughly. I’m fine with that. I’m not trying to drill down. My point is, like, smaller teams - these generalized, aggregated numbers, you can see “Oh, well, there’s a skew there because of this one-person situation”, or whatever it is.

Yeah. No, this is meaningful though. I was actually worried about some flight risk in this area, on this team… And this data is actually telling me –

So you’re feeling better.

I’m feeling better about our product, because today we’ve gotten to go through all the data, and it’s been really… To be honest with you, when I originally got the data, I was so busy I didn’t fully go through it. But you guys asking these probing questions…

I’m having fun.

Yeah, me too.

Thanks for going through that with us.

So yeah, 10% risk of attrition.

We’ll plot with you every snapshot.

[laughs] We’ll give you an overall score, the Changelog score [unintelligible 01:34:44.05] your business.

Yeah. This is great.

I do enjoy this. What I kind of find fascinating is you mentioned the year 2020, when you were at GitHub… And I don’t know if you know what year it is, but I do. It’s 2024, just in case you weren’t aware. [laughter] This is four years later, and you’re this far with DX. Congratulations.

Thank you.

I mean, we’ve asked you hard questions, you’ve shared some insider information that is in this report that only probably you and some others get to see these snapshots… So kudos to you for being forthcoming on that. We’ve talked through DXI, we’ve talked to the core four, we’ve asked you a lot of hard questions, and you’re only a few years into this and you’re this steeped in, I would say, trajection and maturity. You’ve got a strong team, operating at speed; not the highest quality, but you understand where the lack of quality is, and why it’s okay to have that… And you have a pretty good foundation and some assurance personally, it seems, on how to take action when you need to take action. That’s a great place to be in.

Yeah, hopefully… I mean, the way I look at this as a startup is “Can we try to not end up like GitHub?” That’s the goal.

[01:35:57.21] Ooh. Say more. Say more.

[laughs] eng velocity specifically. They were at a point where they weren’t shipping, and people were leaving because it was so hard to ship. Honestly, a lot of – quite a few companies we work with are kind of in that position… These big tech companies that went through that hyper growth and churn, and tons of reorgs, and are now confronting “Okay, now we can’t just keep hiring people, but we need to be shipping faster. What are the levers we can pull?”

It’s kind of like health. If you can get ahead of this stuff and not have four decades of poor diet and exercise that hit you in your fifties… If we can kind of stay ahead of it, I would hope that we’re able to scale the business while staying P90 on velocity. That’s the goal here right now.

Would you be like GitHub insofar as they took a $7.5 billion payout?

[laughs] I would take that. I would take that. I would take a few billion dollars.

I was going to say, in that way, I guess it’d be okay. Yeah.

Well-positioned, I would say. I don’t know who would acquire you. Like, who cares about what you care about to the point where you’re an acquisition target?

I mean, anyone in the developer business, I would say. So… Yeah.

GitHub?

GitHub, Microsoft, Amazon, Google… I mean, anyone in the cloud game cares a lot about like benchmarking, assessing maturity…

Right.

I mean, even Salesforce is a little bit –

Could you IPO?

We could. I don’t –

What do you want to do?

I don’t know. I’ll do whatever fate has in store for us. I mean, that’s what I – we we’ve pretty much bootstrapped the whole thing, so we kind of control our own destiny. We don’t have to have an IPO. We don’t have to sell for $8 billion… We just want to keep – I’m just drawn to this problem… And I think I shared this the last time I was on the show, Adam…

Some of it, yeah.

…this all started seven years ago, when I first became the CTO at a startup, and the CEO asked me “Hey, Abi, all the other departments here are reporting metrics each month. Can you start reporting some metrics on the productivity of your engineering team?” That was seven years ago. And so I joke with people, I’m still just trying to answer that question… Because I couldn’t answer that question seven, eight years ago. I asked other CTOs, “What are you reporting?” and got 20 different answers. It was a hard problem.

Yeah, for sure. There’s a Silicon Valley episode about that… Close to the end of the final season, kind of funny. And you’re a – I think this is in the other show we did together, that deep-dive… I think that you’re the only owner of the business; that you’re the solo founder…

I’m the majority.

Majority.

I have a co-founder, but yeah, I’m the majority owner.

Yeah. I thought it was singular owner. My knowledge is – the time between the last time I had this conversation is diffing on me, but…

No, it might’ve been my previous business, the Pull Panda. That was – I was the singular owner of that. But no, this one, I’ve had a business partner since the beginning.

You’ve taken venture capital? You said bootstrap.

Not venture capital. Just – we took a little bit of angel investment at the very beginning from friends.

Friends and family.

Yeah. But we never spent that, you know…

So when I hear bootstrap, that means that you’re –

Cashflow-positive.

Cashflow-positive, or at least reinvested. Like, if not break even, or in the negative, you’re in the negative potentially because you’re reinvesting, not because you’re losing money.

Yeah. No, we’re extremely cashflow-positive. To the point where my biggest concern each year is how to spend some of that so we don’t pay taxes, corporate, 20% tax rate…. Because that just goes in the – so yeah, I try to reinvest it so we don’t… It’s better to reinvest it than lose 20% to –

[01:40:00.12] Do you do much advertising?

I was going to say, we can invest some of that for you.

[laughs]

I mean… There’s lots of opportunity, let’s just say. Lots of opportunity.

Yes. Yes. Let’s take that offline, Adam. Yeah.

Yeah. I’m only kidding with you. Our listeners don’t want to hear about how we get new sponsors and new partners.

But they kind of do…

No, they don’t want to hear that.

They’ll eventually hear.

We won’t let them though.

Nah. This is fun though. I’ve been digging into this. I think that we want developers to be more happy, obviously… I think the question to me is “How? What makes developers more happy?” I think productivity is obviously one key metric. And maybe some secondary metrics could be - what? I don’t know… Just happiness in life, potentially. Other things that influence happiness…

Pay… Perks…

Pay. Well, you know –

Free beer and ping pong… [laughs] Ping pong tables…

Definitely for a certain demographic, yeah.

RTO… No.

Right. I do think that –

That’s my new tagline, Jerod. Not “ug pull, not cool.” It’s “RTO, no.”

It’s a good one. I just was going to say, I do think that like freedom to live your life in a way that suits you, and still work, is a huge driver for a lot of people. More than money. Probably right up there with like productivity and enjoyment of my work is “Do I also get to live my life in a way that suits me?” But that’s maybe just me projecting, because it’s always been my primary driver, even more so than money, is freedom… And I’ve very much enjoyed it for many years, so I’m appreciative that I have it. So maybe I overemphasize that. But I’m sure there’s a survey out there that answers that somewhere. The Stack Overflow, or next year’s DX – what are you guys going to call this thing? Your public thing.

Probably State of…

State of DX.

…DX. Or Developer Productivity. We kind of use those terms interchangeably. Yeah.

Right. Careful now, because there is a State Of organization run by a friend of ours, who does State of JS, State of HTML, State of CSS…

I see.

And they have this whole platform called State Of.

Yeah. We could just call it Developer Experience Index.

You could also team up with them and have them help you run it or something, and then you could – I’m sure they’d be happy to. Because they did create – although you have all your own software… So maybe it would be a square peg, round hole. But I know they have opened it up, and Google runs the State of HTML survey with them. So if you wanted to really use State Of, I think there’s definitely opportunity there. Anyways, in the weeds again… Let’s call it a show. What do you think, Adam?

Yeah, I’m down.

Abi, thanks, man.

It’s been fun, Abi. Thanks for joining us, man. It’s been cool.

Yeah, thanks for the invite.

Alright, bye, friends.

Bye, friends.