Don't make things worse!

Alright, today I am joined by Taylor Troesh . Taylor, thanks for coming on the Changelog.

Thanks for having me.

I’ve been really enjoying some of the stuff you’ve been writing lately. Your “How to be a -10x developer” actually has spawned and inspired a funny episode we did on our talk show last week, where we had 10 tips to be a 10x, and they were all pretty ridiculous tips. Yours, of course, is good advice of things not to do, and I love that post, I loved some of your other writings lately, so I’m excited to have you on, excited to hear some of the stuff you’ve been thinking about in the software world.

Yeah, thanks. With the -10x stuff I personally have done a lot of those. I think at different points in time, a lot of us are -10x, -100x… And that’s really where it came from. I feel like – I hope nobody misinterpreted that as me thinking these were the teams that I’ve worked with. It was mostly me. [laughs]

Right. There you go. Well, it’s easier to pick on yourself than your past teammates, for sure. Well, maybe not easier, but maybe just more beneficial for everyone. Why don’t you tell us a little bit about yourself, where you’re coming from, and where you’re going?

Right. So my name is Taylor, I write at Taylor.town. That’s my little corner of the internet.

Love it.

And over there I write about humor, software, lots of design stuff… And time. I’m a little bit obsessed with time, so you’ll see a lot of posts about time. I am currently doing freelance stuff; you can find more about hiring me on my page; always doing random projects… Just open sourced a time management utility called Nowify. A lot of people have been interested in that. I’m working on a programming language called Scrapscript… I’m just all over the place.

You’re just doing all sorts of stuff. What about time? I mean, why do you obsess with time? Why do you like this concept? Tell me more.

You know, my grandpa growing up, I think it was him. He’s like an engine of a human being, and he would always growing up throughout my lifetime talk about how many weekends he thought he had left.

Ohhh…

Yeah, yeah. He would just be like “Yeah, I’ve got a few hundred weekends left.” He is still living, and the count’s down real low.

Oh, man, I was gonna say - is he now gone past zero, because he’s outlived what he thought he might? Or is he still on the countdown?

I think he’s far outlived what he thought he would.

So that’s the good news, is we don’t know how much time we have left, do we?

Yeah, he’s kind of a reckless man…

That reminds me of a site by Scott Hanselman. I can’t remember what it’s called; everybody out there who’s heard of it probably knows exactly what I’m talking about now. It’s kind of like how many keystrokes you have left in your life… And he does the math. And the whole purpose of that is that Scott Hanselman gets a lot of email, and he doesn’t want to respond to an email, because he only has so many keystrokes left in his life… So when he gets a private email, somebody asking him a question, or advice, or whatever, he would then turn that into a blog post, and then reply to the email with the blog post. Because he only has so many keystrokes left, and he doesn’t want to waste I guess - “waste”, maybe not the way he would cast it… But he doesn’t want to use them on a private individual, he wants to use them publicly. And so you create a whole website, like HowManyKeystrokesYouHaveLeft.com, or something like this. I don’t know what the website is, but… Similar to weekends, I suppose.

Reminds me of Quentin Tarantino, I believe he’s about to make his 10th and final film… He kind of put a self-imposed limit there. I kind of do that on a daily basis, where - you have like 16 hours of waking time per day, and if you chunk that up, 10 minutes is about 1% of your waking day. So every 10 minutes goes by, it goes “Oh, there goes –” Half an hour, that’s 3% of your day. When you think of it like that, it stresses you out.

Oh, man… I was gonna say, that sounds anxiety-ridden, doesn’t it? Just thinking in those ways?

Yeah. Sorry for anybody listening who I’ve gotten in their brain…

Right. So here’s the question listener, “How much time are you going to waste?” I mean, how much are you going to spend on listening to this conversation? We might be taking a large percentage of your day and just throwing it down the drain. Now, the nice thing about podcasts - and I’m not just a creator of podcasts, I’m a junkie. I listen to podcasts all day, every day. That’s hyperbole, but you know what I mean… Is that they’re just - they’re a parable. Like, you’re not actually wasting, you’re actually complimenting something else. So you’re mowing, you’re exercising, you’re walking, you’re showering, as we learn… Many of our listeners do listen in the shower. That one’s a little strange to me, but there you have it. You’re not actually just dedicated, which is why video for me is a much bigger ask, if I might just use that word the way business guys use it… It’s like, do I have to sit here and stare at the screen? Or not? And so a 30-minute video is way more intimidating than a 30-minute podcast. Does that speak to you, or do you look at it differently?

[08:27] 100%. I actually just did a 545 mile bike ride…

Oh, wow.

…from San Francisco to Los Angeles over a seven-day period. And the great thing about biking is you can secretly leave one earphone in; you do want to be able to hear the road.

Right. You’re not supposed to, but everybody does it anyway.

Yeah. I listened to The Grapes of Wrath by Steinbeck. That was an amazing book to listen while going through the California countryside.

Wow…

This ride, by the way - great organization, the AIDS Lifecycle; if you want to donate to them in order to fight AIDS -great, great organization.

That’s cool. So will that book now always be somehow linked to that bike ride in your memory, do you think?

Oh, 100%, because a lot of it is also about like – like, half the book is about California, and like deeply about California, and all these areas going up and down. So that was a complete accident on my part, but it vibed.

Right. It’s weird how sound and word or music kind of attach like that in our lives… I know when I was a young man - I wasn’t even a man; a young boy, probably 10 or 12… One summer I read It by Stephen King. And I also happened to have for the first time acquired Metallica’s Black album… And I listened to Nothing Else Matters, that particular track, because that was my favorite one on that album, on repeat, while reading the book It. And to this day, when Nothing Else Matters comes on, I’m for a moment transported into those woods, where those kids were in the book of It. And it’s a very strange experience, where it’s like that song triggers for me intense memories of a book that I read when I was 12, or whatever.

Yeah, I feel that.

Alright. Well, speaking of time - it’s time now to talk about some more of your writings… I want to talk about this one, because this is one of those blog posts that I haven’t actually linked up yet in Changelog News, because I knew I was gonna talk to you about it… But it’s kind of one of those posts where it’s so good to me - maybe I’m your intended audience - that I wish I wrote it. I read it – even the conceit. “11 ways to shave a yak.” That’s something that I would love to write. And I didn’t, Taylor. You wrote it, darn it. The conceit is awesome, and then you also executed quite well. So first of all, as we open this one up, for those who are uninitiated about yak shaving, do you want to just give the rundown on what it means to shave a yak? And then we’ll go through this again, not like your -10x, a list of anti-patterns; things really you shouldn’t do in your opinion, because yak shaving, at the end of the day - talk about time - you end up wasting time. But first of all, define yak shave for folks.

Right. So I don’t know originally where it comes from, but the saying of shaving a yak is like “Hey, you want to change a light bulb, but - oh, you lent your ladder to your next door neighbor… So you borrowed a sweater from them, and the sweater has a hole in it that you made, and it was made yak fur.” So you wanted to change a light bulb, but you end up, find yourself shaving a yak, so that you could repair the sweater, so that you could get the ladder from the neighbor, so that you can change the light bulb.

Right.

So it’s a kind of accidental complexity. That’s the definition.

Yeah, it’s kind of a series of unfortunate events, or maybe they’re not – sometimes what’s interesting about a yak shave is sometimes it ends up productive, and like “Oh, I actually needed to get that done in the first place, and so I’m happy.” But oftentimes it’s not; oftentimes it’s this complexity that really, if you think about it critically, is procrastination, to a certain degree, because you find out that the yak shave is enjoyable. You’d rather be shaving the yak. And you’re shaving a yak to do another thing, but you’re also kind of just not doing that other thing that you know you should be doing.

[12:07] Maybe I’m just projecting… But there’s a little bit of that in there as well, where it’s like you really enjoy shaving yaks, and not so much this thing that you set out to do, because that’s hard, or boring, or whatever it is. And so you’re like “I’m gonna do these 11 things and feel productive, but I actually never accomplished my goal.”

Oh, 100%. There are the fun parts of every codebase that draw you in… And man, sometimes it’s like you spawn off a second project that you’re like “Oh, this will help the first”, and you never complete either. Classic…

[laughs] Well, like the Slack team - they were building a video game, and they created this chat system in order to build their video game… And this, I believe, is the second game they tried to create. The first one became Flickr; I’m maybe munging the history slightly, but the same people behind Flickr were the people behind Slack. And both of them were efforts to build video games. And Slack became this internal communication tool they built to build the video game, or alongside it, I don’t know. It ended up being the main thing, obviously, and long story short - what’s the name? Stuart Butterworth, or Stewart Butterfield…? He got rich a couple of times, but never got that video game created in the meantime. So I guess, you know…

Yeah. I guess you could say that Slack was the golden yak.

Yeah, there you go. Sometimes you shave a golden yak. But you have tips here, ways to shave one. These are things not to do. And so these are things that you think are, I guess, not the best ideas, or they end you up down a path of complexity… And I’d like to talk about a few of them. The first one you say – well, not the first one, but the first one I’d like to talk about, is “Create thirsty systems.” Can you tell us what you mean by that? Of course, the takeaway is “Don’t create thirsty systems.” But what’s a thirsty system in the first place?

Thirsty systems - they require more and more just to keep going. And when you exhaust the resources necessary to sustain a thirsty system, you will find yourself out on the road, shaving a yak in order to keep that supply line going. That’s how I view it.

Okay. So what are some examples of systems that you create that are thirsty?

Let’s see… I don’t remember what I put in the essay, but in my own life - or I guess let’s do a development example. This is the Changelog. Credits. Credits would be an example of that, where you forgot to leave the credits going… There could be – let’s say that you require a bunch of training data to keep your business going; getting the training data – if the training data stops, then that might require you to go do this whole excursion to get more training data.

Yeah, that’s a good one. I know that as a listener and a producer of Practical AI, a lot of their conversation is around data labeling, and how much time and effort and systems are designed, sometimes services, just around labeling data, and making data clean and useful for those models to be trained in a way that’s productive. So yeah, that’s a good one, because you know that train never stops. Once you start it, you just keep going and going.

Yeah. A secret - I have been working on a data labeling startup for the past few months…

Oh, you have?

So maybe expect some news about that by the end of the year. [laughs]

Okay, cool. I guess on the business side, maybe you do want to work on thirsty systems, because they need constant feeding, and so people will continue to put their credits into your system in order to have that solved for them.

I mean, yeah; if you’re okay with it, I do view this as the business model for me. If I’m in the training business, then every single time I get a customer, it’s like “Okay, well, now I’ve gotta go find people to–” There’s a lot of yak shaving in that. So it’s not something I necessarily recommend, but I do see an opportunity there, so I’m going to negotiate me some yaks. [laughs]

Yeah. Well, I mean, let’s face it - if we’re going to extend the metaphor… I mean, yaks need to be shaved. And if you’re being paid to shave a yak, and you don’t mind shaving a yak and getting paid for it, then this is no problem at all, is it? This is just a business. It’s a yak shaving business.

Yeah, exactly. [laughs]

The other one you talk about is clever architecture… And you say “Clever systems produce clever problems.” Is there more to say about that, or does it speak for itself? I love that saying.

[16:12] No, that’s a great one. So there’s this building out in France, I cannot remember the name of it, but it was made by this famous architect, and one of its cool ideas was it made all of the plumbing and the wiring and stuff color-coded on the outside of the building, so that they could theoretically maintain it easier. Well, the problem was it actually, for some reason, was a very hard to maintain building, and they had to kind of do major repairs every few years… And it has cost a lot of money to maintain this building… Because it’s clever, right? It doesn’t follow the normal building design, so they have to get architects and stuff to kind of like help with any small change… So yeah, clever designs require clever solutions.

Yeah, totally. It reminds me also of like certain cars - and I’m not a car guy, so I’m going to quickly get outside my depth… But I know you’re not going to have any hard times finding parts for a Honda Civic. But these small, limited quantity cars - where the car is rare, all the parts are rare. And so anytime you have a problem with that car, you’ve got a bigger problem that you’d have with any other car… And it’s kind of the same situation, where their design is really clever, and they’re worth more because they’re rare, or unique, or they drive faster etc. But all the problems are like equally bigger than they would be if you go down that happy path. So it’s just this idea of like straying from the normal in order to have something that’s not necessarily worth it - you’re going to end up with bigger problems than if you would have just taken the straight, or the broadway.

Yeah. I think Kubernetes was in that zone for a while, where there weren’t enough people using it, and so whenever you had a problem with it, you needed to get like an expert out. I think it has become mainstream enough where maybe it’s not a clever system. I’m not actually sure, I haven’t used it in a while… But there are definitely architectures that are so unique that you need to get the experts in… Like “Yeah, I wouldn’t do that.”

I’ve heard Marco Arment talk about this. He really yells or screams about boring systems, and choosing boring systems… Specifically, he’s been on MySQL forever. And the reason is - and I guess I’ve probably been on Postgres an equally long time… And you see a lot of databases come and go, and they have various allures to them, and they’re very interesting for this reason, or they embarrass your current database in this particular vertical, and you have that desire, that wanderlust to be like “What would it be like if I was running Cockroach, if I was running Mongo, if I was running Fauna?” …whatever it is, insert your newer database here. And he says, “I never want to be that first person using a new database.” Or even like the first 10,000 people. He’s like “I’ll be the last person, using the oldest, most worn down, beaten paths ever, because I just don’t want to have database problems that are bespoke. I don’t want to be the first one to find this/any bug.” I think that there’s some real wisdom in that. That being said, new databases are fun to play with, so it’s kind of hard to decide.

Yeah. I’ve been thinking a lot about boring technology. I just wrote a piece called “IKEA-oriented development.” Yeah, I think IKEA has boring furniture to an extent, where they don’t use any fancy tools; you have everything you need. And I like making software like that. Minimal dependencies. And if the dependencies are there, they’re either bundled, or they are in everybody’s toolkit.

For me, I think Python is one that people mistake for a tool that’s in everybody’s toolkit, but it’s more like a hex wrench. There’s so many different varieties and sizes, and it takes a little bit to kind of get the exact right thing. It’s not a screwdriver. Python’s not a screwdriver.

[19:56] Right. So it’s more like a hex wrench. I like that analogy. I’m not sure how – I guess I don’t know that it applies to Python. I’ll take your word for it, I’m not a Python user, but I do like that analogy, because immediately I think of all the different-sized hex wrenches that come with whatever furniture you’re buying… And they’re all just – and I have a collection of these over the years, having installed tons of furniture in my life… And I’m always like “I’ll hold on to this one, because I’m going to use it later to install something else.” And they never fit. They never fit the next thing that you’re putting together. They’re always just like a 16th of an inch smaller or bigger. It’s maddening.

Yeah. I find this in the JavaScript ecosystem, too. But my platonic ideal for software that I make is I want everything to feel like Mario Kart 64, where like - it’s been how long since that’s produced, and there’s like zero chance that you plug that thing in and it’s going to ask you to like download Python 3.9, right? Like, it’s going to work. I wish all software was like that, where it just works after time. I don’t know.

Yeah. I mean, bitrot is a real thing. Sorry, I got stuck on Mario Kart 64. I just went back to a happy place in my mind, and I was just playing the game as you talked there… So that one’s almost too good of an analogy. But yeah, I mean, you pop in an old NES cartridge - okay, you might have to blow on it first, even though I think that’s a wife’s tale, but we still blow on it, pop it in and play it… And the sucker is just gonna work.

Yeah. I wish software just worked sometimes.

I’m with you. So do you have any practical tips? I mean, you mentioned dependencies. How do we write software that works most of the times? How do you do it? How do we do it as an industry, me as an individual? Do you have any advice on that? Dependencies is a big part of it, but I’m sure there’s other things.

Yeah. So actually, dependencies is the one that I think is usually the worst offender. So I tend to pick stuff that’s been around for a while, like Postgres. Postgres is going to be around for a while in terms of web technology. But seriously, if you can get away with not even using Postgres, and maybe using, I don’t know, filesystem stuff - that’s gonna last way longer, just like not even including Postgres as a dependency.

Right.

Now, you have to do the trade-off, of course, of your business. And if you’re handling any sensitive data or anything important, then you’re gonna want the power of Postgres. But in general – so like in the JavaScript ecosystem; if I’m like “Oh, I need to make a call to GitHub”, I just use HTTP. I just use the Fetch API. I don’t go and use their official package, because that’s prone to break. But in the first place, if you’re going to be connecting to somebody else’s API, that’s going to break anyways, right? So I don’t know, it’s hard in the web world to create things that last.

I’m the same way, so that spoke to me. If we look inside of – so our website’s built on the Elixir, and I do have an HTTP library that’s called HTTPoison; it’s a third party library. But I speak to pretty much every web service there is just using that one HTTP library. So if you look in our codebase, there’s little clients for Buffer, for GitHub, or Hacker News, for Mastodon, for Twitter, for Typesense, for Campaign Monitor, Shopify… And they’re all pretty much the same thing, it’s just like “Here’s the details for this API, and here’s the details for that API.” And each of these different services has like a client library that you could go at, right? Like, I could go out and get the GitHub client library, or - especially in the JavaScript world, I mean, there’s a billion packages that you can install instead. But if you could just have one dependency, like a good HTTP library, and in the case of the web, fetch is all you need, basically… Well, let’s say you need one library - sometimes you need more; OAuth, or something - and you can use that to build 17 integrations, you’re gonna be way better off than having 17 libraries, just on plain surface area alone. Right?

Right. And pro tip - you can actually do payments without Stripe. You can go directly to the bank APIs and use NACHA files, and all that stuff… It’s annoying, but if you want things that last - the banking API is not going anywhere in 50 years.

Okay… Have you done that before? Because man, I don’t know if I’d go –

[24:04] Yeah, I’ve done it for a few companies. A lot of these – yeah, I think you can get really deep into a lot of these systems and build them out yourself. I don’t think it’s usually worth it.

Yeah. That’s the hard part, right? I mean, the hard part is deciding where that trade-off is. Where do you draw those boundaries? And I think that these are things that every developer struggles with, because like “When do I pull the dependency in?” and when do I say “No. You know what I’m gonna do? I’m gonna code against the bank API, because it’s worth it”? I think it’s a hard decision to make, isn’t it?

Oh, 100%. I encourage everyone to go watch some Greg Young videos. He does a lot of stuff and talks about – you know, kind of, engineers need to think more from the business perspective. One such example that I’ve been thinking about recently is like meetings. I think that engineers forget to look at things from the business point of view sometimes… And we’re talking about like the platonic ideal of software that lasts - that’s an engineering thing. The business views your code as a disposable thing. It’s not this elegant statue that’s going to be lasting a long time. So this is more for like the love of software, making something that lasts. But from a business perspective, they don’t care.

No.

That’s fine. That’s completely fine.

And sometimes it’s smarter not to care. I mean, in the case of a startup, when you don’t have an established business that you know is going to generate revenue, if you’re building software that lasts, you’re building the wrong thing, because the business might not even last, right? So at a certain point, it’s foolish to build a statue when you’re trying to change the thing constantly… But at a certain point, this starts to now formalize, and coagulate, and then you’re like “Okay, this thing that I built not to last - it turns out the business is working, and it’s going to last.” And now, when do I know that’s the case? How do I know when to start taking it more seriously? These are hard problems, and oftentimes we err on the side of moving fast and breaking things, and end up with systems that are very, very difficult to maintain, difficult to change, and end up having some sort of consultant come in and say, “We need to rewrite this from first principles.” Right?

Yeah. Yeah. I think this actually brings me back to the 10x post… Kind of what I’ve been thinking about recently is not making things worse. So you go into a business, and just don’t make things worse. That, I think, fits the business needs most of the time. Make sure that the database doesn’t go down. They don’t care if it’s faster, they just don’t want things to get worse.

Right.

And I think a lot of the reasons I end up using these “how to shave a yak” and -10x, the inversion principle, is because for messy systems - I have a 10-month old, and I want to be a really good father, but there’s no guide on how to be a good father. However, there’s a lot of advice on there on how to be a bad father, right? I know, to an extent, how to not be a bad father, so I’m just trying not to make things worse. I’m just trying not to mess her up. Same thing kind of goes for, I think, doctors. You have the exceptional surgeon - what was his name? Robert Liston, who did amputations in 30 seconds. He would start his surgeries by saying “Time me, gentlemen.”

Oh, wow.

And you compare that to somebody like Semmelweis, who essentially invented hand washing in hospitals… And he was of the mind just like “Yeah, let’s not make things worse. Let’s not kill people”, versus trying to optimize and make things as fast as possible. I tend to be more on the hand-washing side of things.

Yeah, that’s the kind of doctor you want to be put on there before he does any surgery… Because when you hear the doctor – imagine, you’re like slowly drifting off and you hear him say “Time me, gentlemen.” That’s not comforting.

I think this was pre anesthesia, which is why he went so fast, which was to–

Oh, okay. It’s making more sense.

I could be wrong on that one. It’s been a while since I’ve looked into him.

Break: [27:56]

I do like this idea… And you know, we make fun of the 10x thing around here quite a bit, because it can be quite silly, but software is interesting in that to be quite a bit more productive than other people sometimes all you have to do is not make things worse, like you’re saying… Because it’s so easy, and maybe uniquely easy, in software; maybe not uniquely, I don’t know. I always compare software to like construction, because people try to; you can’t really compare the two, but kind of a close analog, is like physical engineering and construction, like building buildings, bridges etc. And the thing about that is you can hide mistakes for a little while, you can cover things up, but it’s really hard to unbeknownst to you and the team be going in the wrong direction for a long time. Whereas in software, one developer doing something wrong, or short-sighted, or whatever it is, can do so much harm, so quickly, and so secretly. Even maybe they don’t know how much they screwed up… That to be 10x, sometimes all you have to be is 1x, meaning like “I’m actually pushing things forward”, because so many of us make mistakes that are just headed in the wrong direction, and make things worse, maybe the -10x thing is possible, to actually go backwards at such a clip… Which is unfortunate, but I think it is true that you can screw up a lot of things fast. And we do from time to time, because it’s hard, and it’s complicated, and there’s so many pressures. You’re not just writing software; you’re trying to please your teammates, you’re trying to please your boss, maybe you have a deadline that’s ridiculous. Maybe you have a 10-month old at home and you’re not getting very much sleep. Maybe you got put into a position that you’re not actually good at yet, and you don’t really know Lisp all that well, but now you’re writing Lisp, and you didn’t realize that when you miss a parenthesis, this thing happens. You know, there’s just so many factors that make this job so hard in many ways… And we can secretly, unbeknownst to anybody, be causing harm. That’s kind of scary.

Yeah, I totally agree. Yeah. Now that I’ve been thinking a lot about not making things worse, my coding has changed a little bit, where I think I’m a little bit more interested in testing, a little bit more interested in really, really simple deployments, that aren’t going to break on me later on… I really like fly.io, by the way. A little plug for them.

They’re a sponsor of ours, so plug away.

Oh, they are?

They are. But we did not pay Taylor to say that. Yeah, we’re fans as well.

No, you didn’t. I love Fly.

Yeah, our site’s deployed on Fly as well. Yeah, I mean, good tools is one thing you can do… Rely on reliable systems. Dependency selection… Because you’re not going to write the entire system yourself. I think – I can’t remember the stats; Amal Hussein over on our JS Party podcast always cites in the JavaScript world I think it’s like… I’ll just use big and little. For every one line of code you write in a typical JavaScript web app, Express backend, whatever frontend you want, React, it’s like 1 to 10, 1 to 1000… I don’t know, it’s a little amount of code that you write compared to how much code somebody else has written, in any typical application. Like, astoundingly so. There’s so much code that somebody else has written. And if you’re building your app on top of giants’ shoulders, as they say, it’s really important to pick the right giants, because so much of that code you rely upon. And so dependency selection - not just “Do I pull in a dependency?” but “What kind of dependencies do I trust or not trust?”

Yeah. It’s a little easier in some languages to find good dependencies. Elixir is something we mentioned earlier… Elixir, I would say all of their packages were pretty nice before it became popular, and that makes a huge difference. npm was the opposite. It was already popular, and then everyone just kind of came in, and there was a lot of competition… And that ends up - you get a lot of fragmented design decisions across the ecosystem, because things aren’t baked, and made a little bit slowly in the beginning… So I do actually think your language matters in that regard. How your language started out kind of does matter.

That’s interesting. And it can change as well over time. As an ecosystem changes, the package ecosystem around that ecosystem also changes with time.

[34:07] Completely. Deno I know is trying to rethink a lot of the JavaScript ecosystem, and I have found that their community packages are consistently pretty good. I use it for some hobby stuff… It’s obviously like, if you want something to last, don’t use something that new… It’s at, what - 0.19 right now? But it’s delightful, and the community I think is taking it seriously, and I’m just finding the packages out there to be a lot higher quality than the average thing on npm thus far.

Side note - hook me up with that link to Greg Young’s videos. I’ll put in the show notes. Because I can’t find them because there’s a local car dealer here called Greg Young Chevrolet. [laughs]

His discoverability is so bad… But yeah, I’ll send you some of his stuff.

Awesome. Alright, so we’re talking about dependencies… Let’s also talk about tooling. You mentioned Fly as a service you like, you mentioned Stripe is something that you would rather go against the bank – no, you didn’t say that, but… Have you really coated against a bank API?

I have. I’ve done it for multiple companies. I worked in cryptocurrency before the second wave.

Okay. Like 2018 timeframe?

Yeah, that was – so I had to do some banking stuff there. But yeah, I’m currently out of that ecosystem.

Sure. I can’t leave this topic… So when you’re coding directly against the bank APIs - the decision to do that, was it because it’s like “Because we’re a cryptocurrency thing, and it makes more sense for us”? Or was it like “We want to cut out that 3%” or whatever Stripe takes and it’s more important? Or was it really about just like long-term resiliency?

So the first company I worked for was a real estate company. I think we had to do it because our money transfers were too large. We were working with really large deposits and withdrawals. In the cryptocurrency space, I think Stripe doesn’t support or allow cryptocurrency companies. So I think that’s why we had to integrate directly. And I think we had to – it’s like an SFTP type; you have to put things, and it gets picked up…

Oh, really?

Yeah, yeah.

You’re like just FTP-ing stuff to a thing…

Yeah. It’s been a while, but - yeah, I think there was only like a few banks that you could use, so we had to go directly to the bank, and the bank has a crappy API.

So another thing you had mentioned here on the yak shave post is something that I hadn’t heard of, which is Lindsay’s law.

Oh, yeah.

And in the post, you’re saying “Well, you can build on other people’s property.” Of course, we’ve all seen and heard about somebody on the App Store who’s built their business on Apple’s App Store, and has to deal with Apple problems… There’s similar problems with Facebook Marketplace… I mean, you name the platform, you’re going to have to deal with the platform provider. And so that was no surprise to me. But this Lindy effect I hadn’t heard of.

Yeah, it’s super-cool. I can’t remember where I found out about it, but it says that the lifetime of a thing is proportional to its existing lifetime. So in other words, if you want to try to predict how long a thing will be around for, just look at how long it’s already been around.

Right.

Books will be around for a very long time. Deno, at this point, it’s very new. Based on Lindy’s law, I’m not going to expect it to be around in another 20 years, right? Look for old technology for things that are going to be around in a long time.

I love that. So it’s just like, the longer it’s been here, the longer it will be here.

Exactly.

And the shorter it’s been here, the shorter it will be here. That’s so easy to hold on to and use in decision-making. Not always true, though. It’s kind of a distribution, isn’t it? Because things that have lasted this long, by de facto at one point hadn’t lasted as long as they do now.

[37:57] 100%. I think it’s a rule of thumb. And there might be a little bit of mathematic… I don’t remember how Lindy came up with it, but I would say that if you were to make bets and try to get like expected value, Lindy’s law applies. But yeah, sometimes if you want to forge the future, you want it you want to invent the future, as Alan Kay says, then you’ve gotta ignore it. You’ve gotta just follow your passion.

Right. So when it comes to tooling, platforms, Deno… I think they call it Deno now, by the way. Not to correct you on air here, but…

Oh, was it Deno?

Yeah, well, even Ryan couldn’t figure it out at first. It was Deno, then it was Deno, then it was Deno… I think they land on Deno, because that dinosaur is just so stinking cute that they just like are gonna roll with it… But it definitely started off Deno. That project, Fly.io - which hasn’t been around very long, by the way; it’s relatively new. The app store now is [38:50] Do you have any insights on when to break from that general rule of thumb? What do you see in a technology or a platform that says, “I’m gonna give it a go on this”? Because even when I was talking about boring databases, sometimes you come along and you find something new, and that new thing actually allows you to build something you couldn’t build previously. I mean, even the App Store – I mean, even we could look at just like commercial apps like Instagram. Instagram couldn’t have existed before the iPhone. The iPhone made Instagram possible. And had the Instagram founders said, “Well, this iPhone’s not going to be around” or, “I don’t like Objective C, it’s not going to be around–” or whatever… If they had not done that, then obviously that would not have existed. So there are times where, like you said, you have to break from that. Do you have any insights on when, how? When can you tell?

So I’m kind of working on a bunch of different projects right now… There are a few that I want to last, and those I’m very careful to make sure that I’m thinking about not breaking links from the beginning. I don’t know, I think if you go into something from the beginning thinking like “Okay, I want this to last a little bit longer”, you kind of have that in mind, then it does inform some of your decisions. I think for – I used to work for a company whose CEO always talked about type one and type two decisions, where like “Is this a permanent decision, or is this something that could be changed later?”

Oh, yeah.

And when it comes to a lot of these longevity things, like choosing Fly.io - I can move somewhere later. That’s not that big of a deal. But the architecture does matter. Those broken links - that’s hard to change. So I think that’s more important, and when you’re trying to decide whether or not you want to make something that lasts - I think making something that lasts takes longer. Making one of those KitchenAid mixers that lasts 50 years and still works - it’s more work. It is.

Right.

So you as an organization or project planner, open source maintainer just need to decide your shelf life. Where I see this a lot is people’s choice of timestamps. Some timestamps - you can, at least for like certain IDEs… I was at a company who was like “Okay, do we want to use 32 bits and give this app 10 years? Or do we want to do 64 bits, double that, and then we get unlimited?” But I asked the CEO, I was like “Should we do 10 years? Are we gonna be around in 10 years? Do you want to save that money on AWS costs?” These are business decisions.

Yeah. What did they say?

They said 50 years.

Okay.

I think he said “Let’s pretend like this is gonna go on 50 years.”

Yeah, but then you’ve gotta decide, “Is it a type one or type two?”, like you said; because if it’s a small decision, it’s like “Well, let’s go for the gusto.”

No, no, that was not a small decision.

Oh, it wasn’t?

No, no. That was a very important IDE for the hundreds of terabytes of data, and a very large Postgres database.

Oh, wow.

Yeah, that was an important one. That one was not going to be changed.

That was an important one. Was it the right decision?

Yes, I think so. I think taking up a little bit more space to go longer term was the right decision.

Are they still around and going at it?

Yes.

[42:02] Okay. [laughter] Because if they were gone by now, then the answer was “No, it wasn’t actually worth it.” But since they’re still going… I mean, 50 years - that’s a long time in life, let alone in software. Right? I mean, gosh, software changes so fast that a 50-year company… I mean, are there 50-year companies? How old’s IBM? I don’t know. In software it’s like 10-year-old is old. Stripe is old at this point.

It’s weird, because software moves so fast, and yet doesn’t… Like, if you go search on YouTube, you can watch “The mother of all demos”, where Douglas Engelbart in a one-hour presentation shows the first GUI, the first mouse… He has like a video display for like a Google Docs type thing… This was in ’58, ‘60… Everything kind of was designed, and things have not really changed a whole lot since then… So things change so fast, and yet don’t change at all. I don’t know. It’s bonkers.

Isn’t there an old cliché, like “The more things change, the more they stay the same”?

Yeah. It’s hard to pick out – like, in the ‘60s I think it would have been really hard to pick out which parts were going to be around in another 50 years. Because you can ask them, none of them knew what they were doing. The people in Xerox PARC, they’re like “Yeah, this sounds cool. How about a–” And you have Metcalf. He’s like “Oh yeah, Ethernet. We just need to plug these things together.” You didn’t know Ethernet was going to be around, or the defining thing that connects the world together.

Yeah, that’s totally true. I was thinking about that a little bit with TCP and UDP. The new version of H3 and QUIC and all that stuff, they’re building it over UDP.

It took us a long time to get – there were a lot of problems with the TCP and HTTP stack. I am glad that we’re kind of moving over to this UDP thing. That’s an example of a design decision – I don’t think the person or the group that invented TCP really intended this much information being sent over it. Like, if they knew how serious it was going to be, they would have just thought of something different.

What’s funny about that is like when I was in university, one of the things they taught us in our networking classes is like TCP is for important information, and UDP is for not important; like, you don’t care if it really gets there, whatever. And it’s like - so that’s the serious one. Like, “UDP is fine for whatever, DNS, and I don’t know what else you guys are gonna use it for. It’s just over here, it’s simple, but we don’t use it very much. TCP - we’re going to spend half of a semester learning about it.” And now it’s like “Well… It turns out UDP is pretty important, because we’re gonna reimplement a whole bunch of stuff on top of it.” It gives us way more flexibility. The simplicity actually pays off in droves. Now that we realize how we’re using these things - which, like you said, they didn’t know back when it was designed. It’s amazing sometimes that it works at all, isn’t it?

Oh, I describe it – like my wife every once in a while asks me what I do for work… It’s like “Oh, I create bugs, and like mess with duct tape.”

[laughs]

I think software barely works sometimes… [laughs]

Absolutely. Alright, well, we’ve plumbed the depths of yak shaving, we talked a little bit about IKEA-driven development… Is there anything else you want to say on there to bring out beyond the simplicity and the modularity about IKEA-oriented development? We just touched on it.

Yeah, I do. On the IKEA one I have a phrase, “Composable and disposable.” If you look at like the IKEA tables - you can see the shelves behind me if you’re watching on video. The shelves have these pluggable tables, and so you could have a desk/shelf hybrid… And so things are composable within the IKEA ecosystem. And yet, if I wanted to kind of edit it, and maybe chop something off, it’s disposable. I don’t have a ton invested in this work of art, and so I feel free to experiment with it.

[46:11] I think - back to Greg Young; I love this guy. He coined this phrase, “Expendable over extendable.” You make throwaway code, or code that can be easily thrown away. It’s not necessarily throwaway code, but it’s code that can be thrown away.

Right…

And that makes you create the system a little bit differently. His rule of thumb is that no area of your codebase should take more than 10 days to rewrite from scratch. I like that rule; I think it’s a little lofty, but it’s something to shoot for. If you needed to, if there was some intractable bug, to an extent it should take less time to just rewrite that module, rather than go hunt down this weird distributed system type thing going on.

What are some attributes of code that it’s easy to delete, or expendable code? I’m trying to think of what it would look like. Would it have - clear boundaries I guess is the only one I can think of. What makes code easy to delete?

Two things. Number one is the internal state of it. If it’s connecting to a database or something, and it has its own persistence inside of its boundaries, then way harder to do anything with it, really. That’s code that is a lot more sensitive than everything else in the system. Even just storing things in memory can have the same effect.

The other thing is not so much – so it is the connections between it and other parts of the code, but it’s the pieces of code that connect to it, rather than vice-versa. So it’s the number of places where code is referencing it. The actual API for it - I don’t know, would you rather have a good API referenced 1,000 times, or a bad API referenced once? I think the API is just less important than the sheer number of times it’s being called, sometimes.

Isn’t number of calls somehow indicative of usefulness though? Like, if code is useful, it’s going to be called more. And so if you have a lot of call sites, then what you’re dealing with is useful code. I guess useful code by definition is hard to delete, because it’s being useful.

Exactly.

But I’m just wondering if that’s like a bad attribute of code. Like, if I can write a function that gets called in 1,000 places, I feel pretty good. Like, “That was good function. Look how many places we’re using it.”

If it’s not scary, like if it’s easy to understand - oh yeah, that’s great. That’s like a severe win.

I see. But if there be dragons in that function, then that’s when –

Oh, yeah. I think that - back to “Do not make things worse.” If there’s any part of your codebase that is starting to scare you, it needs immediate attention. Things will just get worse if you have that one part of your codebase that all the engineers are afraid to touch, because then it just kind of - you pile around it, and it kind of grows in…

It metastasizes.

…disgustingness. [laughs] Everyone just kind of wants to like get in there and get out, and it just kind of accumulates. Yeah, I think when you start getting to that point, even if you have 1,000 references to it, you want to start thinking about how you can throw that thing away, and not break everything.

That’s well said. I have an idea in my codebase that no longer serves our business, and it served our business very well for six years… So I made the decision - this is in the system that powers all of our podcasts, and distribution etc, etc, etc. And I made the decision back when I first started it that everything was going to be a news item. I kind of liked this idea of like when you design a system, there has to be a core idea, a core tenet. and the core tenet for like Unix - like, everything’s a file. I was like “Everything’s a news item.” So what do we do? Well, we point people to news. That’s kind of the core of what we do here.

[50:01] And it’s like - well, so we have a news item; that’s a piece of data, and it’s gonna point somewhere. Everything’s a news item, and every news item points to somewhere. It could point to taylor.town, it could point to an episode of the Changelog. It could point to anywhere; everything’s gonna be a news item. And we have different kinds of news items; some point to our blog posts, some point to our podcasts, some point to other people’s websites… But everything’s a news item. That’s just how it’s gonna work. And that system had a lot of implications as we built out the system that were really nice, and worked really well, especially because we published a newsfeed on the homepage for years. And it’s like “Here’s News, News, News, News, News, News.” And we would decorate them based on their internal attributes.

And then we decided we got sick of posting news items to the home feed every day, and instead we’re going to write in Markdown and publish a newsletter once a week, Changelog News, and we’re only going to publish episodes and posts to the website. And now, all sudden, this core decision I made all these years ago, that has served us very well, kind of feels useless. It’s just there. It’s just there in the system… And I’m having to work around it a little bit, but mostly it’s fine. But there’s always a layer of indirection between me and what I’m actually trying to get at, which is the episode, or the blog post, or the whatever… And it no longer serves a useful purpose. Those are the kinds of changes where like business changed; it wasn’t a 50-year decision, but it was a nice six-year decision, but here I am year seven, year eight, and I’m sitting here thinking “Do I rip out the cardiovascular system in my codebase and simplify everything?” Because my life would be simpler now that we’re done with this idea, if I ripped it out. But also, it’s my cardiovascular system of my codebase… [laughs]

So I’m just sharing a little bit of my thoughts around this as we talk about usefulness too, because it’s like, it was very useful for a long time… And I don’t think it was a bad decision. It’s just a decision that no longer applies to the way that we run things. And so now it’s a liability.

Yeah. So I like to call this architecture archaeology, where you go back in time to think “When they designed this, what were they thinking?” It’s so fun, because you can go back to old Postgres code and be like “Oh, I understand why they made certain decisions.” So for you, I think don’t beat yourself up too bad. Like –

I’m not. I’m just sharing because it’s interesting. Yeah, I’m not mad at myself. I just know that I got myself here for reasons… And here I am.

Hm… So in terms of trying to redo that, that mental model that you have - because it sounds like the code… The code itself I think is sometimes incidental; it’s the mental model itself that’s hard to change. Have you played around with different database designs? Because when I have things like this, I tend to just think about the database. And maybe it’s like “Okay, well, if I move these tables like this and migrate it”, right? “If I just did this, what effect would it have, from the backend all the way to the front end?” Just everywhere. So that’s how I tend to do it.

Sometimes you can find a compromise that isn’t so bad. Sometimes it’s just like a single column. In your case, it sounds like it’s like RSS, where RSS you can have the content or the link. Or/and. I’m not sure. But maybe just adding a content column would fix the problem somewhat easily… I don’t know. Yeah.

Yeah.

I’ll start with the database, though.

I like that. I do think in databases - I do think that data structures, and the form of the data, and the actual data itself is the most important part of most systems. And I don’t really feel bad about the way it works now, even though it doesn’t really map to the way we’re doing things exactly. It’s not causing any problems. It’s mostly just like, I know it no longer maps to the way I’m doing things, and so now every time I have to step over that architecture in the code, I’m like “Oh, there it is again. This is useless. I’m just doing it because that’s the way we do it.”

[54:06] And so that over time just begins to nag, and you start wondering – you know, because I kind of take the gardening approach to software development; like, any area that codebase that I’m in, I’m going to pull the weeds. I’m not going to spend a whole day just pulling weeds, but when I’m over here fixing this, or adding something, changing something, I’m going to leave that code better than I’ve found it… So now I always had this one thing that could be better anytime I’m in a certain section, where I’m like “This doesn’t make sense anymore. Why am I doing it this way?” And I know why, but I wish I wasn’t. And then I’m like “What would the refactor look like? How big of an effort is it? Could I replace this subsystem in 10 days?” It’s not really a subsystem. Like I said, it’s more like a cardiovascular system where it’s running through the entire codebase… So I don’t know exactly my point here. I’m just kind of sharing a little bit of the architectural archaeology with you…

Oh, yeah, exactly.

…that I like to think about.

My advice to you is just don’t make things worse. [laughs]

I like that. Don’t make things worse.

In this situation, it sounds like it’s really hard to make things better, but it’s really easy to make things worse… So, yeah, proceed with caution.

Yeah. Which is why I’m with you, and why my tactic thus far has been just leave it alone. It’s not causing harm; it’s not a problem. It’s not even really slowing me down. There’ll be a point where it starts to slow me down more. It’s not currently slowing me down, because I understand it. If I hired an outside developer to come work on this, it would be slowing us down quite a bit, because I would have to give them the mental model that I have. And it’s such a way that they would be like “Why is this working like this?” We come across that a lot.

But yeah, I’m trying not to make things worse, and so I’m just leaving it in there… And I probably will leave it in there. But part of me is the perfectionist, purist architect guy, who’s like “Dude, your software doesn’t map to your business anymore. What’s wrong with you?” So I have that little niggle of like “You should fix it, Jerod…” But you’re right. I just should not make things worse.

So I have a static – I made my own static page generator for taylor.town, and at first, I think I had some Postgres database to store like “Okay, I want this released on this date [unintelligible 00:56:19.04] and then - yeah, the mental model wasn’t even right from the beginning, so I ended up redoing it. So I just used YAML front matter. So I have a little YAML thing on top of all my Markdown files… But now, it’s like my entire site is just a bunch of Markdown files and a 30-line Elixir script that parses the markdown that creates the RSS feed, and generates a few dynamic pages based on everything… But yeah, if you can get rid of a big dependency like that, and just move things down into simpler subsystems, it is really nice. It’s so much nicer, because now if I want to make changes to the website, it’s like “Oh yeah, the whole website is dictated by 30 lines” and I’m just like “Okay, throw that away.”

That’s nice. That’s beautiful. I think every engineer should build their own personal static site generator at some point, right? That’s kind of like becoming an adult kind of thing; like your rite of passage. It’s like “I built my own website generator, and so now I can be a big boy or girl.”

Yeah. I think HTML CSS is so much easier now than it was a while back, especially with the CSS Grid and Flex stuff.

Oh, yeah.

I’m very efficient with just plain CSS now. If you haven’t given HTML CSS a try without React or whatever, go for it. JS is also really nice. Like, a lot of these new query selectors they have, vanilla JS - pretty dang good right now if you’re making websites.

It is.

[58:01] Also, if you haven’t ever made your own programming language, Rust makes it super-easy. Like, that’s another project, like a static site generator where – I think everyone needs to give that a try. And especially because we have this book, “Crafting Interpreters.” Excellent book. It’s a long read, but it’s a good one. It walks you through making your own programming language; something I think everyone should try.

So you’ve been making your own Scrapscript, coming in 2024, according to the website… So still in development.

Yeah. Scrapscript.

Tell us about it. What is this thing?

I hope it comes out in 2024. I just put that there because it’s not coming out this year. [laughs]

Because it’s a different year. Yeah, you can just put an incrementer on that, and every year it increments by one. You’ll be good.

Yeah. So let’s see, Scrapscript. I have made a bunch of different versions of this over the years… I said early on in the podcast I used to work with cryptocurrency. I was less interested in the currency and more interested in the crypto, and so I was like “Oh, what if you made a programming language where everything was content-addressable?” And making that nice to use is really difficult. That’s why I think you’ll see Unison, which I think they started working on it around the same time I did, but they actually worked on it, where I just kind of twiddled my thumbs for a while…

Sure.

But getting content addressable where every little section of the language is hashed, getting that nice to use is a large paradigm shift. You need a lot of additional tooling. So for instance, Unison uses a lot of Git type stuff. It’s like, okay, we’re going to store these files in this kind of database thing in Git… If I understand correctly. I might be wrong about that. Whereas Scrapscript, where I’m using it, it has its own Git type thing that you use, that’s kind of like a distributed key store of all of the little snippets in the language which are called scraps.

Gotcha.

So it’s just a whole weird space that I think is fun to explore; not sure how useful it’s going to be, though.

So with everything being content-addressable, what’s the implications of that? What falls out from that?

So the design goals of Scrapscript were to essentially make JSON+++.

Well, that’s a lot of pluses.

Yeah. Like, think of like a nice to use JSON that has types; you can essentially define a schema right there in the JSON. And so it’s typed, so you can describe other types, and it also has functions. So it’s a full programming language, but it’s really small. Now, where it gets even better is you add that one little thing in the language, where you can take any expression and replace it with a hash of that expression, and store it in this global namespace, so you can name it, and other people can name it. So you get this really shareable language. You can send the scraps to each other via Socket, or whatever… You can send it as a message. But you also can upload it and have people reference it by hash, kind of like it’s a web page. So you have like a bunch of different options.

And Scrapscript also comes with its own compact byte format. Think of like MessagePack, where you can compress it really small… JSON is not very compressible. Whereas if you give it a little bit of forethought, you can make things a lot nicer on the bandwidth. And that seems to be the problem right now, is bandwidth, and having these systems that degrade connection over time. The APIs don’t match after a while. So I’m kind of working in that space, is making software shareable.

What’s the status? Where are you at? how far along?

Let’s see… I have made four working compilers for this thing. Each time I find something I don’t like. I’ve been working on it since like 2017, 2018.

Wow.

[01:02:01.25] So right now, I think I’m working on the final compiler. I’m really happy with the design and ergonomics. I’ve used a few different versions of this. I have a ton of really talented people that have reached out and are on the mailing list at news.scrapscript.org. I am trying to get that spec out… Hopefully, in the next few months. I was supposed to get it out last month.

And I think this is one of those 50-year problems. We were talking a lot about trying to make software that lasts… I want to make a spec more like JSON, where JSON doesn’t need to change. The tooling surrounding it - there’s always new tools that use JSON, and I want an evolving ecosystem around Scrapscript. I don’t want the spec to change. I don’t want new language features. It’s more of like a data change format, but you can also code in it.

So I am working on getting a spec out that’s coherent. The language itself, you can write a crappy interpreter in a weekend. It’s not that complicated. So I’m trying to get the important things done right. So that it last 50 years. 50 years is the hope.

Okay. Well, you do have a website out there, so people who are inspired by the idea, Scrapscript.org. Like you said, there’s a news feed people can hook into… That’s cool. Is it going to be community? Is it going to be open source? Is it going to be – what.

Yeah, totally open source. The only reason it’s private right now is I’m trying to get – I’ve already been getting some feedback and trying to tune a few more things in private, so that when I put it out, I think everyone can have like a little bit more fruitful discussion… But oh yeah, definitely, this thing is going to be out in the open. I don’t know, it’s hard as an open source person trying to make something… If you go back to what I said earlier about Elixir - Elixir started out very small, and it had a chance to make everything right before it got popular. I am somewhat trying to copy that success. Like, I want this thing to be really small, until it works really well, and all the packages and stuff are nice, and then we can throw everybody on. It’s like “Hey, everything already works. Don’t make a bunch of crappy packages. Everything is already made for you.” So I don’t know, maybe I’m just moving too slow and I’m a little bit neurotic. We’ll see.

Time will tell. Well, if you’ve got a 50-year timeline… I mean, come on.

I do have time, I do have time. [laughter]

And time is something that you love. Well, the website is taylor.town. Of course, as I said, scrapscript.org. If you’re picking up what Taylor’s putting down, definitely check out his website, definitely subscribe to the feed, or to the newsletter, or however you get Taylor’s content. I’ve been enjoying, as I said at the top, a lot of the stuff you’ve been writing, so I just encourage you to keep writing, keep coming up with blog posts that are just envious, or enviable, that I envy them and I wish that I wrote them myself… So the rest of us can learn from the things that you know, Taylor. Thanks for coming on the show today, this has been a blast. Any final words, or anything you want to say before we call it a show?

Yeah. I have one thing to say. Don’t make things worse. [laughs]

Don’t make things worse. Wise words to end on. Alright, we’ll call that a show, and we’ll talk to you all next time.