Abstractions and implementations

Hello and welcome to Ship It, the podcast all about everything after git push. I’m your host, Justin Garrison, and with me, as always, is a very jealous Autumn Nash. How’s it going, Autumn?

Just – this is our relationship at this point. Like, you just just rub it all in. It’s like, “Look at my 3D printer when you don’t have one.” Oh, I just went to dinner with Angie Jones, your hero in life… And not only did I send you a picture, but I ate dinner next to her.

Justin’s just walking around like “Don’t you wish you had all this male privilege…?”

We talk about that all the time. Okay, you have to say this spicy take because you’re like the white guy, and you can get away with it.

I drop plenty of spicy takes. I use that privilege. That voice - I will loan it to other people if I can.

He uses it so hard, it makes my life.

So it’s actually been really interesting for me, because I got decently successful in tech before I transitioned, and then I transitioned, and so I’ve been able to see in real time; people like literally respected me less.

What?!

But I remember how I used to be treated… So I will walk up to people and I will go “Treat me like you did in 2021.”

Oh my God, we’re best friends forever. I love you so much.

With us on the show today is Hazel Weakly. Hazel, thank you so much for joining us.

That was the best introduction for Hazel ever. Okay? But not only. Wait, wait, wait; it’s an audio podcast, so y’all can’t see, but Hazel’s dressed to the nines. Like, she just won like best Ship It dressed ever.

We now have a competition for dressing up for the Ship It podcast, an audio podcast. This is great. We’re recording this on November 1st, it’s just after Halloween, which all of us tend to get dressed up a little bit… I got dressed up. Autumn, you had a costume, too. I forget.

I went as a skeleton to San Francisco, and… San Francisco. Halloween is just wow. It is. It is wow. I was not prepared.

But also, November 1st is just an off by one of Halloween, so this show for Halloween is fine.

I go straight from Halloween to Christmas. So then like I Halloween for a little bit like later, and then like I just go straight into Christmas. I pretend like Thanksgiving doesn’t happen, and it’s just a free day off.

It’s a free two days for us in the States, right? So enjoy it.

So we know you have a good employer that gives you two days…

That’s true. I guess it’s not free two days everywhere.

Again, you’re damn privileged, Justin. The other week he was like “I didn’t have any meetings today.” I was like “We’re just rubbing things in now?” Like, “Look at my 3D printer that didn’t get lost in the mail in the background.”

I was going to talk a little bit about going to All Things Open, which was fantastic. Autumn, you just went to get GitHub Universe… But I feel like – I mean, by the time this comes out, those are like a month behind us, so we should probably just…

But y’all… GitHub Universe was, like – the detail… Hackable badges… Every detail was on point. Also, the Dungeons and Dragons deployed for Kubernetes, the top 10 [unintelligible 00:06:40.00] was amazing. I have never saw something that was actually so informative, but nerdy, funny, and so many jokes… And you almost forgot that they were telling you a list of like security things, because it was so good. It was like a nerd religious experience, and it was amazing. Also, I met the coolest people. I gave a talk, too. That was okay, because you know, peopling and being in front of people…

My talk got rejected, because I’m not cool enough… And so I cannot even begin –

You are so cool enough.

We’re going to be best friends. But I can’t even begin to tell you how much FOMO I had over GitHub Universe. I didn’t expect to have this much FOMO over it.

I think it’s because it couldn’t handle all of us in the same room. Can you imagine if we were together for GitHub Universe? Oh, my God… It would have been like a neon explosion of just like… We have to make this happen. So you’re submitting a talk for Scale, so we can all go together, right?

I am not committing to anything on camera or a recording.

We almost got her. Just submit the exact same talk from GitHub Universe to Scale. Boom. Copy-paste.

[00:07:58.26] What I will say is allegedly I am planning to submit a CFP for Scale, and SREcon. And if anybody wants me to be hilarious on camera, in front of people, they should definitely ask me to submit a CFP, because that raises my probability of remembering from five percent to more than that.

Us, Hazel. We. We want you to be hilarious. You have to tell us about you, Hazel, for all the people that don’t know you… Even though I feel like everybody knows you, and if they don’t, it’s kind of lame, but… Tell us about you.

So it’s really interesting… Well, especially the “everybody knows me thing” - that is the weirdest true but not true thing I’ve had, because I feel like a lot of cool people know me somehow, and then a whole bunch of other people know me, and then nobody else actually knows me. Like, I’m not even a micro celebrity, I’m not even like a niche micro celebrity… I’m like that one weird kid in the corner that people talk to because they’re like entertaining, and they get invited to some of the parties, and like in two years they’ll be a micro celebrity… But I’m like in a weird twilight zone right now.

But in terms of what I do, and about me etc. I do infrastructure things, computer stuff, I break things… But a lot of the most interesting things I think about are community building, figuring out how to build sustainable structures for people, and generate ways to give them psychological safety, and ways to help them with their innovation, and figure out how to take the concept of wholeness and self actualization and community actualization, and kind of blend all of that into one happy pile, so that everything works well. And what I mean by that is - I used to think that you had to compromise between doing the capitalism and having an effective, healthy, happy team.It was a choose two out of three. But I found out that actually it’s not a choose two out of three. You can actually choose four out of three. You can get even more by leaning into that humanity, the vulnerability, the empathy, the wholeness. And so that’s what I do.

Break: [00:10:20.23]

You’ve kind of already been on the show, on Ship It at least, because on last week’s - or two weeks ago - we were talking to Preston about Hachyderm. You were brought up repeatedly as someone that was helping do the migration, and run the infrastructure, and set that up in a way that was safe and sustainable for a community-run project, where it’s like we actually have on call for this infrastructure that we’re running that builds out a bigger community. So it’s like a sub community enabling a much larger community. So I feel like what we’ve already talked about you on the show has been exactly what you just laid out, of something that you’ve been doing.

It was like Hazel part one, and this is part two, with you actually on here. That’s how much we talked about you.

In this part two, Hazel - you nerd sniped me bad on LinkedIn with one of your posts talking about abstractions. And that was immediately when I was just like “I want her to come talk about this some more.” So today, one of the things I really want to focus on was - started as a LinkedIn post, but then became a blog post, we’ll link it in the show notes, where you talk about home-baked abstractions and store-bought implementations, which I think is just a fascinating way of thinking about platform engineering, and infrastructure in general. Can you talk about that a bit more?

Yeah, absolutely. So I actually ended up nerd sniping myself on this, which is how essentially all of my blog posts have been. I either get really mad and I yell at the Internet, but politely, or I nerd snipe myself and end up writing about it.

That’s what blogs are for.

Yeah. I know, I have ADHD. You couldn’t tell. So the home-baked abstractions and store bought implementations - that started from looking on LinkedIn, and John Cutler actually had a post on LinkedIn that was talking about abstractions, and Charity Majors was talking about it as well… And there’s this concept of “What’s a good abstraction, what’s not a good abstraction? How do you think about this?” Because one of the easiest things to do in engineering is not necessarily even build the wrong thing, but build a thing that doesn’t really work in terms of integrating with other stuff, and integrating with the future. And that’s one of the things that we take for granted with the physical world. The amount of constraints that you have in the physical world because of the laws of physics are enormously useful.

Like, if you want to allow people access into a building, there’s only a couple of general concepts. The building needs to not have a wall somewhere, and you can mess with it a little bit, but you can’t invent a teleportation device, and you can’t do some weird sort of inverted trapdoor. You have so many constraints that allows you to build these deeply reusable abstractions, to the point that you can go into a hardware store and buy screws, and they work with a screwdriver. You don’t need a custom screwdriver, you don’t need these things. And part of that is the thousands of years of evolution of putting things together, but also, the constraints are really interesting.

[00:16:00.05] When you remove those constraints in the digital world, you enter something really fascinating, which is to me one of my favorite topics, and that is “What is it and what does it look like for human thought itself to be the tool that we build on top of, and that we treat as a tangible thing?” That we treat as “What are we building our towers of abstractions of?” It’s thought, it’s patterns, it’s understanding collective knowledge, institutional knowledge of our society. How do we get better at that? Because right now we can handle maybe two and a half layers, maybe, on a good day of abstraction, and then we rewrite it in two years. But that’s not sustainable, and it’s also not effective.

Now, I don’t know if I 100% agree with that, only because we’ve been migrating platforms for pretty much as long as I’ve been in technology. Like, every couple of years it feels like there’s a new understanding of how things work, and then we just say like “Oh, actually, this is the new best practice” or “This is the new common way that people think about this stuff, so we’re just going to rebuild everything on this new common thinking.” Because even like eight years ago it wasn’t containers for most people. And now we’re kind of like “Oh, this containers thing is pretty solid. We’re just going to keep moving that.” So VMs before that, it was physical servers before that… Whether those things are similar or not over time, it was just like “Oh, it’s just a smaller computer, right?” It’s like “Kind of… But not really.”

But there has to be this collective intuition about how the systems work. And we all have to move a little bit together to make sure that everyone is on the same page of “Oh, if you’re going company to company, or environment to environment, you still can operate the system and understand some of those abstractions, just because of the collective knowledge of what technology has available, what computers can do”, and those sorts of things.

Yeah, but I think there’s two different contrasts to that. Look at some of the companies that are still on Cobol, you know what I mean? A lot of government-ran technology is so old. The way that [unintelligible 00:18:08.11] and everything is so old. So some parts are moving.

I mean, legacy runs the world, for sure. Legacy is where all the money’s at, but also –

So it just seems like there are some things that withstand the test of time, and you can very much so get stuck in that realm of what was built. We were talking to – when we did the space episode, if you put something out in space, you’re not going to be upgrading it all the time. So I think that’s like a trade-off. Some things you know you’re going into it and you are not going to be switching things out every couple of years, and it has to be good the first shot, or the first couple of shots.

I mean, there’s a difference between good enough and an upgrade, right? Because it’s like “Oh, those COBOL systems - guess what? They just still make money, and it’s fine, we don’t need to modernize it.” There’s plenty of mainframes out there. There’s all that stuff. It still exists. We don’t need to add a new abstraction to that. But for taking things forward and to try new things, I feel like every few years we kind of reinvent some abstraction.

So that is something I was hoping we would get to, because you fell right into my trap. So there’s two parallel concepts here, or two slightly different ones. One is this notion that Justin’s getting a bit more to, of abstractions being how we approach a problem, how we understand it, how we think about it… For example, we had computers, and then we had VMs, which are portable computers, and then we have containers, and then we have functions, and WASM, and we have serverless, we have different types of runtimes… And what we think of as a program changes all the time.

And we have the other side of things, with “Do we need to rebuild our infrastructure? Do we need to churn things? Do we need to sort of rip things out, almost for the sake of it? But now, is that progress?”

[00:20:00.19] And I think you can actually reconcile those two viewpoints by considering actually the geological process of compactification. So with compactification, you have dirt, and it’s loose. You can move it, you can play with it. It’s there. And then over time, the dirt actually gets sunken into the ground, and becomes more compact, it becomes tighter, it becomes more ossified, and eventually it will actually become stone.

And so one of the biggest things that I’ve noticed in the tech industry in the last, particularly 5 to 10 years, but even in the last 15 years, is it feels like as an industry we have stopped doing this process of compactification. So imagine, for example, it’s 2002, and threading isn’t even a thing in the Linux Kernel. But now we have threading, we have multiprocess, we have container cgroups, and cgroups version 2. Essentially, you have this pattern for most of computing where you would build kind of an abstraction, and the other layers didn’t go away, but you compacted what you needed to consider as a programmer when building things. Before, TCP and UDP and all these other - someone had to invent those. You just used to send packets over an interface, and that was what you did. You just made up your own protocol on the fly. And eventually we came up with TCP, we came up with UDP, and now we have HTTPS, and all these things, and you just don’t think about it anymore. And then you can patch that and keep going. And we have WireGuard, which is one of my favorite examples of this type of process. 10, 20 years ago, a VPN was a big deal, and it was complex. It was annoying. Encryption was a big deal. An encrypted network was complicated. And now you have that literally built into the operating system, to the point where it’s ergonomic and everything to build on top of it. And now you’re seeing all these really, really cool things coming out, and eBPF is the same way. But we don’t, in the more user-facing side of things, have that process of compactification that I notice. And that makes building what I consider real abstractions really, really difficult, if not nearly impossible.

Go into that a little more, about what you mean by a real abstraction.

So I talk about this in the blog post, but what I think about with a real abstraction is - we don’t really have a good name for it in the language, but it’s that process where you take a group of people and you’re trying to understand a problem, a domain, a solution space thing, and as you work on it collectively together, you build vocabulary around things, you build concepts and understanding around different ideas, around different articulations, and then you build on top of those and you develop your understanding of the space through these iterative layers of understanding. And you can actually see this in language development, where for example languages that are in societies, that have like permanent ice, have so many words for snow. And in English, we have basically just snow, because we don’t experience it so deeply to need this nuance. But we do, on the other hand, have in Seattle 20 different words for rain.

And this idea of, okay, if we take this kind of language concept of building these collective understanding layers on top of each other, I would take that into technical concepts. That’s what I think of when I think about abstraction. And it’s this emergent process, and thinking about emergent behavior, and emergent understanding, and how do you build the ability for the emergent things to happen, and how do you let it continue to grow unfettered and evolve organically.

[00:24:06.19] And usually, that starts with just like – the reason we build something on top of it is kind of the convenience factor, right? Going back all the way to COBOL, right? Grace Hopper, key figure in making COBOL as a language, as a common language that you can write to, that then would compile to all the different bare metal implementations of different assembly, different chips underneath the hood, right? COBOL is that first language that is doing that for people. And they’re like “Actually, I’m going to give up some speed instead of writing Assembly code in the actual execution, but I gain speed in writing this common language that is in portable over different chips.” And that’s like the whole reason that exists as, I guess you could say, an abstraction over those different assembly for those chips. And like, we’re doing that again, and it was like “Actually, what if I could write to Kubernetes, and I can deploy it on top of any infrastructure, wherever it’s running, to say “That is my new abstraction”? I’m giving up some control. I’m adding some complexity under the hood to do the compilation, to do this stuff that maybe I don’t understand how it got there, just like compiling a binary… But at the end of the day, I gain speed in the fact that I can consume that interface better, more quickly, just to say “This is the only thing I need to write to. I need to write Kubernetes, and everything else under the hood is going to be figured out somewhere else.”

I think it’s really just a lot of trying to figure out how we allow humans to interact, and to have a way of communicating with computers. It’s just different ways of giving computers instructions, but in a way that you’re meeting humans at different readability levels, and different levels of being able to communicate with computers. Because if you think about it, Java made it where it was easier to write it once, and then compile on different systems, right? And then COBOL, it was different chips. Like, we’re just getting closer – and then Python is more readable for some people than other languages. They’re just these different ways of communicating with the computer to get a certain outcome.

And the Java example is a really good one, because COBOL is abstracting the hardware, Java is abstracting the operating system. It’s just like “Oh, we’re just gonna put this new VM on top of whatever it’s running on.” Same thing with what Wasm is doing. You have this runtime that’s portable, and it’s like “I’m going to write to the runtime, and then I can deploy that on any operating system, and any hardware.” And then how that gets run again is - yeah, it’s just people saying “I want this to be–” I don’t know, I hate the “write once, run anywhere” sort of approach. That’s not 100 percent factual. But that’s the idea, and that’s kind of the goal for a lot of these things. And if we can write that abstraction that is everywhere, then maybe we would have to stop implementing it.

A really funny example that you brought that up, because if you look at the history of DevOps, the word, the concept, the agile movement… And the agile movement, actually, one of the origin stories of it is actually Java, and the failures of Java to truly be a “write once, run anywhere.” And one of the stories of there is a very specific one of when you had these developers running Windows machines, and compiling and writing a Java application, and they did it in a waterfall style of they wrote the whole application, and then they deployed it to production, and production was Solaris.

And it was – if it’s write once, run anywhere, and you test it locally and it works, it should just work. And it turns out it completely fell over, did not work. They had to basically take the entire thing, rip it out and rewrite it. And the people kind of leading the project looked at it and said it was an unmitigated disaster. And how do we make sure that this essentially never happens again? We can’t do the write once, run everywhere, but not run it until it’s – so the abstraction became not necessarily the technology, but the process.

[00:28:04.07] And so the abstraction and the evolution there was “If it’s write once, run anywhere, that’s cool.” But even if it’s not, the better idea around the reusability and around the efficiency is actually that you write something and you can immediately, as soon as possible, figure out whether or not it works.

And so that’s where the agile movement started coming from. And that’s where DevOps started coming from. And a lot of this continuous integration and continuous development concepts came from, is recognizing the fundamental deficiency of not being able to validate something in the context that it’s going to actually operate in.

So how does that apply to what you say in the blog post about having commoditized implementations? Because you have these abstractions that you want to home-build or build yourself in a way that is useful for the humans operating the machine, and then you’re going to apply that in a sense of taking off the shelf components and basically gluing them together. The vendor engineering sort of thing.

Yeah. So the distinction there between the store bot implementation and the home bot abstraction is, if you think of an abstraction as something that’s useful for you, your team, your organization, your company to operate, then the abstraction is really for “How does this company get better at doing the company thing?”

So the most effective abstractions are always going to be so deeply specific to your understanding that they don’t necessarily make sense to anybody else. It’s not that other people couldn’t use them, but it really just does not necessarily make sense.

For example, if you work at a company that deals with a whole bunch of distributed databases, you’re probably going to develop a lot of really nuanced concepts about consistency models. And if you’re a company that just has a generic website and doesn’t do a whole lot of things with it, you probably don’t have a lot of abstractions and concepts and understanding around consistency models. They might have ones around like different types of environments, or A/B testing, or things that are useful for websites. And the implementation, the reason why that’s store bought, is because if you build the abstraction on top of your own implementation, you’re going to run into the not invented here syndrome kind of thing, and you’re going to isolate the entire company from the context of the rest of the development world so much… And it’s actually going to hurt you a lot more in the long term.

Imagine if everybody saw Kubernetes and then said “Oh, this is a great idea, but I’m just going to write it from scratch for my own, and build these nice little concepts on top of it.” You can build the abstractions, and you can build the concept and understanding, but if you rebuild Kubernetes, that’s just a waste of time. It’s not meaningfully helping you understand the problem better… It’s actually hindering you from understanding the problem better, because you can’t work with other people at the implementation level.

I think that’s one of the things that is underrated about open source, is not just the fact that your people are working and making these tools for everyone to use, but the fact that you have this area of thinking, a sounding board to make something better from multiple viewpoints, and people running things in multiple environments… So you get more of a iteration and validation because other people are using this product in different ways than you are.

[00:31:47.17] I mean, how does that apply for like something like Kafka? Kafka has a fundamental way of being used, or a reason to use it, and then everyone uses it in a different way. Everyone’s moving messages from one place to another and having some buffer in between. But the actual thing that they’re trying to buffer, or why they’re buffering it, or how much data they’re putting through, or how fast they need it to go through - all of those vary depending on implementation or use case. And a lot of that comes down to that sort of like abstraction – I almost feel like, Hazel, what you’re saying is like abstraction is more than just the technology bits, right? It’s the process of the business. How we get work done, how the, how the org is structured - that needs to be baked into the abstraction more than just the technology implementation of “Here’s a WordPress website. I’m going to make that as easy to deploy.”

Actually, if you have an internal group creating WordPress websites, versus external customers creating WordPress websites, that abstraction needs to look different because of the interface you have between the people making or using the abstraction, and the people creating and maintaining the abstraction.

Absolutely. And when you take that and you expand beyond just the technology people, it gets even more important, and the distinction between the implementation and the abstraction is a whole lot more profound, and in some cases obvious.

For the WordPress example, you can think of “Well, how do internally we think about websites? How do other people in the company know that they need one? How do they talk to developers? How do they talk to designers? How do developers talk to non-developers about what we need out of what we’re building, and how do we even think about that?” So we can talk about images, and a bunch of images on a page, but that’s not a gallery until you call it a gallery. If you talk about a website, it can be a bunch of pages stuck together, but is it a landing page? That’s different. And what does a landing page mean to you?

Maybe you have a concept of what a funnel is. And if you write funnels often enough, you might actually make a [unintelligible 00:33:56.23] funnel with the construct for WordPress, that works for your particular marketing team’s implementation and your particular sales team’s strategy. And that’s not going to actually be universal. The universal implementation might be “Oh, we need custom types. We need a form. But we’re not going to rebuild our own form software.” But “What does a funnel look like?” is going to be different for every company, and it has to be.

Break: [00:34:27.03]

Every time I see an abstraction get rebuilt is usually when there’s a new consumer of the abstraction. Once we say that – someone’s going to come and ask, “Okay, we have this WordPress abstraction. We run it on top of - who knows - some other infrastructure abstraction.” But then the business comes and says, “How much does it cost us to run each WordPress instance?” And if that wasn’t thought into the abstraction from the beginning, you’re going to bolt on something else that then there’s some external measurements. But if you build that into the abstraction of a total cost that a WordPress, whatever the thing you package as a WordPress site - if you have cost as part of that, then you can have this other consumer easily come in and integrate with it and say, “Oh, I know how much this one costs. We can build different sizes of WordPress abstractions that cost us different amounts. And then we can go in and more easily tie that directly to the business.” And usually, that comes from just like this almost like inflating of the abstraction, and saying “Okay, well, we have a really cool, tight abstraction, and now we need to bolt on like five different things for these different business units to use it, for it to make sense to them.” And you either bring those into the abstraction, or you have other interfaces for those things to work outside of the abstraction.

I think this is also because we put a lot of weight on just being technical, but not a lot of weight on being able to work with other departments to make the business part happen, and to be able to do like processes, and to communicate… Because a lot of times we want to build something really cool, which is great, and we want to be like uber technical, and I think developers are like – it’s just like that argument between like Macs and Linux. We want Linux when you want to get really choosy about all the different things that you want to add to something, and then sometimes you want a Mac or macOS because you want it to be more just use it. That whole conversation we had with that new distribution of Linux that’s more, I guess, user-friendly, that we were talking to.

[unintelligible 00:38:26.04]

Yes. But if developers aren’t given the option and aren’t put in a place to really understand the business needs, and the business needs aren’t really able to understand the technical construction – what they do you call it…?

What they do, how it ties to the rest of the business.

Yeah. It just seems like it’s always this tug of war. And the more – like we have all these different ways to abstract away… Sometimes abstractions are good, because you only want your developers to be working on the things that they need to know. But other times it hides so much that they don’t even know what they’re working on top of. But I feel like if we could get better communication… Communication is just as important as being technical, because really, what we’re doing is we’re communicating with computers. But you also need to be able to communicate and understand the business to make the right technical decisions.

Yeah. And so a concrete example of an abstraction that a lot of companies end up building, in some way, but they would benefit from building it in a more uniform way, would be personas. Because you end up having your customer base, and you can break the customer base into personas. But do people actually take those personas and build a real abstraction out of them? And it turns out a lot of people don’t. Because if you thought about what a real abstraction for personas might look like, in theory you could take your monitoring and SLOs and observability and all this stuff and sort of drill down by persona, and be able to work with, say, your product team to figure out what pricing makes sense based on the volume, the reliability needs, the support team, and what the product success team is doing by persona.

[00:40:15.05] And then people can start talking about, “Oh, we think we have this persona, but it turns out that this persona has very weird behavior, where they fit into this bucket, in this area, and this bucket in this one. So we have two personas.” Or it turns out that this persona is 10% of our revenue and 90% of our traffic. How do we either optimize that, or throttle that, or renegotiate? What is that? Is that a missed opportunity? Is that an overinvestment? What is that?

And you end up, in reality, in the case where people are working on performance, and they’re just kind of working on it without tying it to the business contest. And so they might go “Oh, yes, this group of people just kind of really use the website a lot, and whatever. We’re just going to fix this and make it so that it doesn’t fall over. We’re going to handle multi-tenancy over here, but we’re just going to kind of handle it in an ad hoc way, so that quality of life for everybody in the application is pretty okay.” But you can’t communicate that to the rest of the company, and so you just end up with these meaningless sounding OKRs like “We’re going to take the 99th percentile of response times and bring them down here”, and nobody cares. But if you say something like “Oh, this persona accounts for 80% of revenue and 20% of volume. So we’re going to focus on making their 99th percentile performance really, really smooth. And to do that, we’re going to take this whale type of persona, and we’re going to handle them with a throttling strategy. And this other persona is actually a very small one, except it has one customer that is 97 percent of that entire persona’s usage. So that customer is going to get a little bit extra treatment, and we’re going to actually talk with the product team and the marketing team sales and figure out what we need to do with our pricing strategy to account for that, because it doesn’t reflect what’s actually happening. Maybe we need a new enterprise, too.” And this is a sign, because this is weird. But that is far more compelling than any OKR you could pull up that’s just numbers.

That’s so true, because sometimes you need the story and the context, but also, in any other business you could not know your audience to make a good product for them. And then that’s the whole point of the iteration and the agile… If you can go back and get the results and figure out what’s working and what’s not working, you can better serve your customers, but also better serve your business at the same time.

I mean, but we get blinded by numbers in technology, right? The numbers are the easy part.

I think we could do such a better job as a service to the people working in the business, and our customers, if we would take that approach of marrying the two and remembering “This is a business, and you need to –”

I do feel like looking back just on my own experiences, where developers have been paid so much, they’ve been seen as this high, off to the side, like, you’re not allowed to disturb them if you don’t work as a developer, that it’s been really difficult to bring them back down to earth and say we’re all working on a business together, and we need to work together on the same goal of making customers happy by making whatever it is that the business does make that side of it better. And in infrastructure and developers, you just get blinded by the easy thing of like “That number is really easy to chase. I can lower that number by 10 percent. I know it.” And it’s like, that actually doesn’t matter. At the end of the day, let’s see what is that actually going to do.

[00:43:45.04] So at some point you have to look back at the harder thing of going and talking to people, the people that are your customers, or the people that you work with that are the interface to the customers. And that’s fair. I worked in help desk for a long, long time, and if I don’t have to – but also, I still work in help desk, right? Even if my title changes, I’m still basically the help desk. I’m still out there, talking to people.

Specialist solutions architects are really well-paid help desk, you know what I mean? If you look at it, so much of our jobs, if you’re doing it right, you are the help desk. Because if you don’t know what your customers need and you don’t know if you’re fulfilling those needs, and what is – like, half the time observability into your product and into your customers is the problem. It’s like when you are worried if like DNS is the problem, and 90 percent of the time it is DNS… It also is people just completely – like, sometimes I think as like engineers, people get so into the technical details, and like the flex of like being so technical… And I’m just like, there’s so much communication.

It’s also such a cultural thing as well.

Yes. And it’s like icky.

We have this huge problem in technology where people think that they want to be left in the corner to just do their own thing, and they think that being technical is better. And the more technical, the more nerdy you are, the more like all of these non-human elements, the better. Like, if you could just unplug my brain and stick it in a [unintelligible 00:45:11.00] “Yes, amazing.” And trying to take these types of people and say “You would actually be a lot better at your job if you talked to the non-technical people instead of just insulting them, or instead of just looking down at them.” And they would go, “What? Oh, no.”

Even the way that roles talk about each other, the way engineers talk about product, the way that people talk about solutions architects… It’s always like this ridiculous pissing contest, which makes it so much harder to work together.

I love reading people’s bios that say “I love to solve hard problems”, and I’m like “Oh, you like fixing printers for your family?” Level of hard problems is just like go fix a printer for your dad on the phone, right? Like, you’re not even there…

Or explaining any technology to people that were born before like Google and cell phones, and all of those things were popular… The way that my kids will pick up technology and just go, and then the way I’m trying to teach it to an older family member is…

Hard problems. Hard, hard problems. And I think what we’ve kind of all agreed on here is like everyone at the company is part of customer service, everyone is part of sales… And in my opinion, everyone’s also part of recruiting. Those are the three things that if you work at a company, you’re doing it no matter what. It doesn’t matter what your title is. You have those three things.

And if you’re not doing it, or you don’t think you’re doing it, you are doing it, but you’re doing it badly, and you’re probably doing the opposite of what you should be doing.

You’re doing the negative.

And that’s not great for the company, or for you either. But like even just going back into - yeah, you’re part of the help desk, you’re part of… And you know what’s really great about being part of the help desk? You have this really good, really refined intuition of what can be automated, what can’t be automated, and you should be working on that. And one of the best ways to work on that, one of the highest leverage ways to do it is internal tooling for non-technical people. Nothing humbles you more than going “I built this thing. It made sense”, and everybody’s “Why the f**k is the button in the wrong spot?” And you’re like “Okay, what?” And they’re like “Okay, I don’t get it at all, actually. I get none of this. What do you mean this is basically just like [unintelligible 00:47:18.12]

And so you have to make things understandable for people. But also, when you work on internal tooling for non-technical people, it ends up being such a massive leverage point, such a massive influencing point, and such a massive return on investment that it’s shocking that people don’t do this in companies. Like, I’ve yet to see a company where you have someone saying independently, “Oh, I’m going to build an internal tool for the help desk, people, because people spend 60% of their time on this one type of problem, and I could just build this in an hour or two, and then it would help so many people.” And you can’t do that, because that’s a waste of time. And I’m like “No.”

[00:48:07.11] At a previous company we had DNS records, and we had S3 buckets. A lot of clients would want to do S3 bucket direct transfer, but that requires clicking a couple of buttons, setting up a couple of things, and touching AWS somewhere. And so the help desk people and the solutions people would talk to the DevOps types of teams, and those teams were overloaded with busy work of basically helping the help desk. And I’m like “These teams are underwater. They don’t have time for this.” It would take literally two days to just build something that you take a bunch of forms, you fill it out, and then on the backside it does a bunch of hideous selling out to Bash script that just creates all this stuff. It’s fine. It doesn’t need to be amazing. And I never actually got permission to build that. And I was like “This is stupid.”

Companies that don’t let people fix their own problem have other issues…

But there’s also no incentive at most companies to work together in that manner.

Well, if anything, there is disincentive; you’ll be actively punished.

Exactly. I just wonder, since we’re in this weird flunking place of tech, when will people realize that you do need those cross-team skills, and those communication skills, and process skills to also make you a better engineer?

I was hoping that the zero interest rate phenomenon would give people the space and the time and the ease of exploration, because investment is so low risk, to figure out a lot of these problems and these social problems and these “How do you actually help the business and how do you actually do all these things and incentivize the right results?” And it ended up having the opposite effect.

And so my new hope is that in all of the hard times around the whiplash of the market, transitioning from zero interest rate to higher interest rates, and all these tightening of the belts, it’s going to get people hopefully forced to look beyond tomorrow, and look beyond their neighbors, and go “What actually makes the business more effective outside of my preconceived notions of what I should be doing, and how I deliver value?” If I can look outside that box, what are the options? What are the possibilities? And I hope we get there as an industry. And if not, I’m willing to drag them, kicking and screaming.

Drag them, Hazel. Drag them.

I know we will have more conversations in the future, whether in person or more podcasts. Hazel, where can people find you online?

You can find me on hazelweakly.me. You can also find me on Blue Sky, Mastodon, and LinkedIn. I’m sure you can find me in other places, but I wouldn’t really recommend it, because that’s kind of weird.

You can also find them at coffee with me a lot, because we’re just about to be besties.

We’re going to be besties. This is going to be great.

Thank you so much for coming on the show, and thank you, everyone, for listening. If you have other ideas, your own ideas about platform, and maybe even - I would love to hear how people stretched across those boundaries of where they were working and teams they worked with to add business value, or make an abstraction that kind of crossed those barriers… I love hearing those stories, so please reach out to us. It’s shipit [at] Changelog.com. And we will talk to you all again soon.

Also, can we vote Hazel like president of all infrastructure and platforms and technology? Because it would be such a better place if we let them.

I mean, I am wearing a very queenly outfit…

Look, I’m your first vote. I’ll do the campaigning for you.

Let’s do it.

Alright, thank you so much.