Robert Ross: Our approaches is within incident management we have communications, so status pages are key; internal status pages, product related status pages, external status pages… When we say communication, it’s really like even just sending an email to someone. Because when you communicate about an incident quickly, you build trust internally with the team. So for us, every incident that happens through our tool, you automatically get a status page. That can be sent to anyone in your company. It doesn’t require a license. Because if there’s a fire and you need to tell people, tell them. So that’s part of our solution.

And then the other angle that we take is we have a built-in Service Catalog with service ownership, with teams and team assignments. So when you do encounter an incident, our belief is that you should be able to very quickly get the right people to that incident as fast as possible. And I’m a big analogies kind of guy, and we’re called FireHydrant… So imagine you live in Brooklyn, you call 911 and you say, “There’s a fire in my apartment.” They’re not going to send a fire truck from the Bronx. They’re going to send a fire truck from around you. And the way they know how to do that is because they understand your neighborhood, they understand who owns your neighborhood, and the best people suited to get there quickly, and know where the fire hydrants are, right?

There’s so many things that come from service ownership and service catalog that you have to have in place. And we’ve been building that central pillar of FireHydrant from day one. We’ve had a service catalog in our product from the first few lines of code.

Alex Sims: Yeah. So I’m completely new to all this stuff. I thought the status page was driven by those SLIs and SLOs. Is that not something that – that’d be really cool.

Jerod Santo: Hello friends, I’m Jerod and this is Changelog News for the week of Monday, July 4th, 2022. But we’re shipping out on Tuesday instead of Monday, because [Freedom…! 00:00:16.11]

First up, thanks to everyone for sharing your thoughts on this little experiment of ours. We are listening and adapting, so please continue with the feedback. Here’s what’s different this time around. One, almost everyone said to make it longer. [As you wish… 00:00:37.10] This episode is about twice as long as last week’s.

Two - many of you also read Changelog Weekly, and since we’re covering the most interesting links from Sunday’s edition of the newsletter, it makes the audio version less new or interesting. So we are now mixing in stories that we haven’t previously covered. Hopefully, that keeps it fresh for everybody.

Three, many of you think the music bed is distracting, and asked us to turn it lower, or turn it off altogether. So let’s try it without, and see how that sounds. Nice and quiet.

Okay, let’s get into the news.

On DX tips, Swyx writes about the 4.5 kinds of dev tools platforms. This is his attempt to make sense of the overwhelming world of developer tooling. If you work for or invest in dev tool startups, I think this mental model is a good one. What are the 4.5? One, application platforms that make the user productive. Two, infrastructure platforms make the application run. Three, data platforms make the data useful. Four, developer platforms make developers productive. And the point five, internal services with cross-cutting needs to interface with every platform. Those are his high-level conclusions. Read the whole article for his thinking behind the matter.

Did you know there are things you should know about databases? Lots of things. And did you also know there is an excellent - and I mean, excellent post on architecturenotes.co to teach you some of those things. It’s focused mostly on indexes and transactions, explaining them in great detail with some seriously impressive diagrams to make sure it all clicks. Not much else to say about this one, except that it was the number one clicked link in the newsletter, and there’s a link for you to click in the show notes.

Brandon Rhodes wants you to start all of your commands with a comma. Why is that? If you have your own Bin directory somewhere in your path, which you probably should, and you put all your own little scripts and executable bits in there, like you probably should, eventually you’ll hit a problem or your script names start to conflict with the built-in and bundled commands. But if you start all your commands with a comma, like Brandon does, this problem melts away. [You cursed brat! Look what you’ve done! I’m melting, melting. Ohhhhh, what a world, what a world. 00:02:56.22]

As a bonus, team this technique with tab completion, and it’s super-easy to browse your collection of commands. Just type comma, then tab, and there they are.

Lobsters turns ten. Over the weekend, the website’s creator, Joshua Stein, tweeted “Ten years ago today, I “pooped out” a better version of Hacker News in my spare time.” As Thomas Ptacek eloquently put it. It didn’t really mature into the thing that I hoped it would, and I don’t use it much anymore, but I’m happy some people are still finding it a nice place to be.

Over on the site itself, current administrator Peter Bhat Harkins says “Over that time, our community of 15,013 users have submitted 87,530 stories, written 381,460 comments, and cast 2,536,117 votes.” That’s a lot. I am a fan of the site, which usually features more esoteric and engineering-heavy posts than other tech aggregators. The conversations there can still get toxic from time to time, but were on the internet is that not the case? Seriously, if you know of such a place, holler at me. [I want to go to there 00:04:18.09]

[04:20] James Hawkins and his team at PostHog have interviewed 725 people, and one of his big takeaways is “It’s normal for candidates not to ask harder questions about our company, so they usually miss out on a chance to 1) de-risk our company’s performance, and 2) increase the chances they’ll like working here.” So he shares some really important job interview questions engineers should ask, but don’t. Questions like “Does the company have product-market fit? How much runway does the company have? What’s the culture like?” and a whole bunch more. Definitely worth a bookmark for the next time you’re job shopping.

Go Time listeners already know this, but for the rest of you, our recent episode with Ron Evans a.k.a. @deadprogram is just too good to miss, even if Go isn’t your thing. Here’s the conceit - the year is 2053; the tabs versus spaces wars are long over. Ron Evans is the only Go programmer still alive on earth. All he does is maintain old Go code. It’s terrible. He must find a way to warn his fellow gophers before it’s too late. Good thing he finally got that PDQ transmission system working. What results is a hilarious and somehow still insightful conversation with so many funny moments like this one about the future of social media.

“You know, when you’re building something that’s gonna survive a two year trip to Mars, believe me, your mp3s sound pretty funny by the time the ship gets to its destination… Or so I’ve been told. I don’t know, actually those might be AIs sending back those reports. There might even be no humans that survived the trip. There’s a rumor going around they’re all just AIs.” “How’s it going around? Who’s it going around?” “Social media still exists in 2053.” “Oh, thank goodness. I don’t know what you’d do without it.” “I use Minder. You know, it’s where you’re allowed to dump your actual mind directly.” “That’s cool.” “Is it text? Is it visual?” It’s more like a feeling.” “It’s just Hex.” “Remember the feeling you used to get when there was somebody being wrong on the internet? It’s like that all the time.” “Is it XML though?” “No, you just plug directly into your brain-computer interface, and you’re just really mad right away.” “Oh, I love it.” “Yeah, it’s beautiful.”

Listen to the rest in our Go time feed, or on the web at Gotime.fm/235.

Last up for today. Upptime. Upptime. No, I wasn’t stuttering. [Did I stutter? 00:06:43.00] That’s U-p-p-t-i-m-e. That’s how you say that, right,? A super-cool uptime monitor and status page powered entirely by GitHub. It uses GitHub Actions as the uptime monitor itself, checking your website every five minutes. GitHub issues for incident reports, opening new ones when endpoints are down, and assigning team members appropriately, and GitHub Pages for a status website built with Svelte and Sapper. This project is not affiliated with or endorsed by GitHub, but it is super-cool, super-free, and endorsed by over 1,000 people who have set it up for themselves.

That’s the news for now. Let us know in the comments if you liked this better than last week; or worse, or whatever. Seriously, we do want to hear from you. We have an excellent conversation coming up for you on Friday. Brian Cantrell from Oxide Computer joins the show for a deep-dive on their attempt to build servers as they should be - hardware, with the software baked in for running infrastructure at scale. Adam’s on vacation this week, so I have a special guest/co-host joining us as well. Can you guess who? We’ll talk to you then.

Chris Evans: Yeah, I think you’ve hit the nail on the head. I think using Incident.io incidents for low, low severity things has many benefits. It has the benefit of you just leaving a really, really good trail, so someone else who can come along and first of all see what you’ve done, if you’ve reached a solution and understand your thought process and learn a lot…

There was an engineer at Monzo who used to do this repeatedly, where he would dive into some of the gnarliest bugs, that would scare most people away… And it was just a fascinating read, being able to go “I have this channel. I can look at that, I can look at a timeline”, and you can sort of scan through and catch up on those things… It also acts as like a really nice, structured way to hand over work. So if you are picking up some lower-severity bug or issue in production, but you have to go somewhere, you can be like “Cool. I’ve left all the context in this channel. Pick it up and run with it kind of thing.” So I think all of those things kind of lean towards that… It’s just useful and helpful. There’s very few downsides. I think that’s the main thing - there is such a low cost to starting an incident. You’re talking one slash command, and you’ve then got everything at your fingertips. And lo and behold, if the worst does happen and you’re investigating and you go “Oh, this is really bad”, you are suddenly now already in the place where you need to deal with your incident, with a heap of context that people can then pick up and run with… And surrounded by all of the support and tooling that we’ve got in place there. So if you need to escalate to engineers, they’re a button away. If you need to communicate with your customers via your status page, the same sort of thing.

And that’s the approach we use at Incident.io. As Stephen says, we are just using it for any kind of structured, but interrupt-driven approach to dealing with things.

Raj Dutt: All those things, absolutely. The Loki teams, the Cortex teams, the Prometheus teams - they’re relatively small teams within Grafana Labs. We’ve traditionally put most of our engineering resources behind Grafana. But those other things, like Prometheus, like Loki, Cortex - we’ll have some interesting announcements around tracing soon. You know, Grafana Cloud… We really wanna invest more on the wider ecosystem, if you will.

So it’s really not one thing in particular. It’s more on Grafana, scale up on the big tent, wider ecosystem, scale up on empowering the community, investing in the community, investing in open source, also building commercial differentiation, launching new products in our cloud platform that are very different to what we’re doing today… It’s really about completing the picture from a developer’s standpoint. When you want to be able to troubleshoot some of these very complex systems.

Everything from getting an alert at 3 o’clock in the morning - you’re blurry-eyed, it’s a high-stakes game; something’s down, your boss is calling you every ten minutes… From the time you get an alert to the time that you have to look at your metrics, you have to look at your logs, you have to look at your traces, you have to switch between all these disparate systems across different vendors, till the time that you find the resolution, find the problem, update your status page, create an RFO… How did you violate your internal SLOs, what does it mean for your external SLAs… There’s all these systems, all this data, all this process that you have to go through, and there’s so many different avenues along the way to optimize for that experience and to make that experience really slick.

[01:24:20.05] So that’s really what we’re focusing on. It goes so much beyond Grafana itself. And if you can kind of optimize for that experience across different vendors, do it in a way that’s really composable, prioritize interoperability, but also just create a flow, if you will, for that poor developer or SRE. She’s woken up at 3 o’clock in the morning; why does she have to have 12 different tabs, and 10 different systems to deal with 8 different vendors? There’s so much sub-optimal workflow in that, and we just wanna take the pain out of that, and do it in a way that is interoperable and composable. So we’re just kind of focused on that problem, and there’s so many areas in that problem that we can innovate on.

So that’s the majority of the answer to where we’re gonna focus. But then the other part of the answer is – Grafana itself has been pushed in all these interesting directions by the community, in ways that we couldn’t even imagine. And that’s the magic of open source. Most people are using Grafana for IT operations, whether it’s cloud-native, or legacy IT… But they’re using it to run their infrastructure. But we’re seeing Grafana used in all sorts of crazy different ways, that are beyond really our imagination.

There’s an emergency room system in Tokyo that’s using it to track wait times across all the ERs in Tokyo. The German rails system is using it to help with metrics there. There’s a community of people who run beehives that are using it to monitor the activity and the weather around all their beehives. SpaceX is using it to help with launch control. So there’s all these use cases that are outside of the normal problem domain that we’re solving, whether you wanna say that that’s industrial internet of things, or it’s business intelligence, or whatever…

Kelvin Omereshone: Yeah, for sure. Thank you so much. First of all, I started building the Boring Stack way before people now know it as the Boring Stack. So I started working on it as my answer to the very shiny, but not so productive world of modern JavaScript full-stack web development. So at the time I was building a platform called sailsconf.com. I still work on the platform; it’s the place where I educate JavaScript developers to build full-stack apps using the sales framework, or whatnot.

So at the time I was building the app with like a sales API, and then Nuxt SPA, which is like a Vue meta framework. So the promise I had from Node.js was using JavaScript across the stack, one language makes everything really faster… But having to have this separation of concerns with two repos - that’s two codebases, double routing, because after routing the Vue router, then I route in my API… Don’t get me started with authentication nightmare, because you know how many people really invalidate the JWT really well, right? So I was [unintelligible 00:07:14.15] I was like “Okay, this is good.” And especially when they’re loading spinners, when there is a page – for example, on sailsconf.com there’s a Courses page.

The page is useless without the Courses list. But with the current status quo, I will have to have like a loading spinner. Why do I need a loading spinner? Why can’t the page just load with the courses? Because without the courses, there is no Courses page. But that’s the status quo of modern SPAs, right?

[00:07:45.21] So I got dissatisfied, so I started looking for alternative. Then I found Inertia.js, which is this thing that came out from the Laravel world, because of course – the maker of the tool, Jonathan Reinink, had the same problem I have, which is I want to build modern SPAs, but I don’t want a separate API. Can’t my backend drive the SPA experience? And that’s what the Boring Stack is all about; it’s where you have this solid MVC, stable, predictable web framework, and then you can bring in your React, your Vue, your Svelte as your pages. So you get the SPA experience, you’re routing this on the server, the data the page needs comes with the page each time the users request the page, so there’s no loading spinners at all. So in a nutshell, that’s what it is. There’s other things that goes into it, but that’s just what it is.

Fred K. Schott: Yeah, that’s a great question. I go into that in a bit of little different places, but it’s always hard to get a kind of succinct place where you wanna look for it. The website uses your module entry point in a packaged json manifest. So if your package is published and you’ve exported an ES module build, basically, you’ll traditionally add a little module entry point, similar to main, if you’re familiar with that, where it points to the ESM build. So packages will actually build side by side if they are targeting both Node and a modern browser.

[11:56] An ES module build, which Webpack, Rollup, they all understand - it will be a little more modern, it will be a little faster because of that usually, and main will remain the Node long-term support; “What is the latest version of Node we wanna support?” It lets you decouple the requiremens of both, where Node v4, v6, v8 maybe doesn’t have the same support that a modern browser that supports ES modules would.

That has been traditional in the community, so I actually have a stats page on PikaPkg. That was one of the first things… If you go up to the top right of the site, there’s a nice little link that will take you to a stats site, and it’s actually tracking module usage over time. You can see going back to 2015 - there’s actually a fun, little history section of that page, where the first package to ever use a module entry point appears to have happened by accident… To this day, I would love to know the answer of why they did this, what tool they were building for. Who knows. They just said “module=” and there was like the name of the package. Maybe it was for their own testing, maybe they were just confused… I don’t know.

But that was a year before this became the standard, which Rollup, and – actually, I believe it was first proposed in a post called “In defense of .js” as a pushback against the Node communities or the people working on ES module support in Node, their proposal to add a .mjs file extension. Their take was “We would like to support ES modules. This is the new modern thing, we’re excited. However, as Node, we don’t know how to tell if you are using a file that is old CommonJS, or new ES module syntax… So we need this new file extension; that’s how we’ll know, and that’s how it’ll work.”

They’ve since walked that back a bit - I don’t believe it’s anymore required - but at the time, that caused a lot of frustration, especially among the web community, that said “This is a Node problem. Why is the solution going to have to affect everyone? It affects me as a web developer. That’s not my problem.” There was a lot of friction there.

So this concept of having a module build came out of that proposal to “How could this work without adding a new file extension?” Since then, pretty much immediately after - that was around mid-May of 2016 - D3 started, Rollup started, all these big projects started buying into this idea, and that has just grown and grown over time. So about 6% of the packages on npm now support an ES module build. Fairly low, but a lot of packages are older, deprecated, not maintained, or even Node-specific, where it doesn’t make sense. Even today, only bundlers are really looking at this property.

Nadia Odunayo: But two things. One is I wanted to kind of avoid anything that was user data, even though it would have been anonymized, and would have been more trends… But two, I had a chat with a friend, a colleague, kind of sometimes mentor, [unintelligible 00:43:38.21] and he said to me “The best businesses always make money or get value directly from their customers. They don’t have a middle person, a middle entity. And if you’re going down this publisher plan, you’re gonna have a middle entity. It’s not direct.”

Oh, sorry, the other thing is I didn’t want to do ads. That was another thing. If I could avoid ads, that would be great. One of the things that people hated on Goodreads were the ads. So I was thinking “Okay, maybe it’s freemium then.” And so really, the thing that made us start to build out the paid plan was just our costs. We were spending thousands of dollars a month to keep everything running on the highest Heroku plans… And the database - every time we had a spike, we needed to upgrade to the next database. We were getting to the highest tier. What happens when the highest tier runs out? What do we do? And it felt like every time we got an influx of users, our costs went up, so we needed to do something about it.

And so that’s when I just made up a paid plan. Like, I literally made a page… It didn’t exist. I said the “Plus plan coming.” At this point it would have been 2022 – it was in 2021. Oh, wow. I’m like, is it 2021 or 2022? It must have been 2021, because we became profitable in 2022. So in 2020, at the end of 2020, October, I made a page, that app.storygraph.com/Plus. And I said, “Hm…” And I went through and I looked at the features we had, and I thought “How can we enhance these?” Okay, similar books… We’re gonna give you personalized similar books. So you have a unique page just for you, that takes into account your unique preferences, your unique themes, topics and tropes that you’ve told us you liked, and shows you the books similar to this one, but also highlighting the ones that have the things that you like. Cool.

And then it was stats. Everyone loves the stats, but what if you could compare two of the stats pages side by side? So I was just taking the features we had, and just like made up a list of features. And then I said “We’re also going to have a public roadmap on which you can vote, and post. You’re going to get priority support, so we’re gonna [unintelligible 00:45:53.27] And I said, “Pre-order now, it’s $30. After that is $50 [unintelligible 00:45:58.24] And we got over like three months, we got 1,400 pre-orders. 1,400 people paid $30 for something that didn’t exist.

Gerhard Lazu: It runs just test, and when we run it, it creates a report. Why does it create a report? It uses this amazing tool - many have heard of it - Hurk. Not sure if it’s the best name, but anyway; we didn’t pick it. Hurl.dev. And Hurl.dev is orange open source. It’s a CLI that – actually, it’s more than a CLI, but you interact with it through a CLI, which allows you to write tests and assertions about HTTP requests. So the focus is on testing HTTP endpoints. And it’s written in Rust, which means it’s really fast… It has a very simple DSL. You put it in .hurl files. And what does it look like? Well, let’s have a look at the files changed. So we’re going to look at files changed… There’s the workflow which runs it, and it’s just tests. That’s it. And when it comes to the hurl file - I’ve put it in test, by the way; there’s the just file, we’ll scroll past that… Let’s look at this one. Testadmin.hurl. We get the host admin., we repeat it twice, so that we can confirm the caching behavior… We expect the HTTP status code to be 302, and we say we want HTTP/2. You can specify this in the file, when you do your configuration… And then you can write a bunch of assertions.

[01:13:57.29] In this case, we make sure that the response comes back in a second, by 1,000 milliseconds in this case. We check that the header, the location header sends us back to home page, because we’re not authenticated…

Chris Ferdinandi: Yup. So I’m going to totally butcher his last name, and I’m so sorry… Jason Lengstorf from Netlify, he’s over on the developer experience team… He tweeted like a week or two ago that he ported a Next.js site into Astro. He used 90% of the same component code, he obviously had to make some nudges and tweaks to accommodate the build tool a little bit… But he ended up reducing the amount of client-side JavaScript that got shipped by 90%, and the page load time went down by 30%. So just really, really huge wins all around. And this is with almost entirely the same code. So using Next, but no client-side runtime on that. It’s just HTML and platform-native.

John Graham-Cumming: [44:48] Yeah, sure. I had lived abroad for a long time, as I am now doing again. But in 2009 I came back to the U.K, and I think I saw a tweet from probably Stephen Fry saying “It would have been Alan Turing’s 90-something birthday today, if he hadn’t killed himself.” And I knew the story of Turing because first of all I’m a computer scientist, and I’m interested in computer security… So you end up – Turing pops up all over the place.

And I knew, of course, that he had killed himself, and I knew a lot about the history of the code-breaking stuff, and artificial intelligence stuff… And I was at home, and I just got really annoyed about it. I honestly thought to myself, you know, the problem here is that we don’t talk about Alan Turing in general, because it’s a kind of shameful thing which is sort of like “Well, he killed himself because he was gay, and because we prosecuted him and we treated him very badly…” So in a very typically British way we sort of forgot to talk about him.

I thought, “Well, if we could get this out once and for all, we could all talk about it, and admit to ourselves it was a bad thing.” Then we could celebrate what he did. So I wrote a blog post on my blog saying “This is terrible. Britain should apologize”, and somebody in the comments said “You know, you could start a petition on the number ten website.” So I immediately went there and created this petition. Now, I had to wait about a month for it to be approved, and I honestly thought maybe 500 people would sign it. You know, “Who cares about this intersection of gay rights, and computer science?” and I was like “There’s probably 500 people in the world who care about this…” Well, in Britain anyway, because it’s only open to residents.

So sure enough, 500 people did sign quite quickly, and I thought “Well, I’m gonna give it a go. I’m gonna try and get the press to talk about it”, and I just worked on it by myself, just trying to get people to write about it. Eventually, The Manchester Guardian wrote about it. Then The Independent, and so on, and it kind of snowballed a little bit.

One of the first famous people who signed it was Richard Dawkins. The cool thing about Richard Dawkins signing it was that I could then go back to the press and say “I know I told you about this before, but now Richard Dawkins has signed it, so what you should write is “Richard Dawkins has signed this.” So I did that, and eventually I wrote this cool postscript – at the time, the signature names were public, and my father was actually reading them every day and saying “I think this is so-and-so (famous person).” And eventually, I automated my father.

So what I did was I wrote this ugly perl script that took the names and then searched on Wikipedia to see if that person had a Wikipedia page, and if on that Wikipedia page it said something like “So-and-so is a British/Scottish/English/Welsh blah-blah-blah…”, so trying to see if they were a notable person from the U.K. And if they were, then it would email me and then I could get a hold of them and ask them…

I did that with Ian McEwan, the writer; his name was on there, and I found his email address. I emailed him and said “Are you the person who signed this?” He’s like “Yes.” And then I was like “Can I tell the press?” “Yes.” I did stuff like that, and it grew and grew and grew.

[48:00] Then just before the bank holiday at the end of August I emailed somebody I had met from the BBC, a journalist called Zoe Kleinman. I’d met her because I’d written this book before, and she’s written a little bit about my book… And I sent her this really cheeky email which said “This is a really important story. You should write about this.” So I was telling the BBC, “Get on with it.” And she very kindly wrote back and said “You know what - I’m gonna write something. I’m going away for the bank holiday weekend; I’m gonna submit it to the editor. I have no idea if it’s gonna get published.” So she did, and I never saw it.

I went to sleep on the Sunday night, bank holiday Monday, I wake up… I have a graph of the number of signatures, and there’s almost this vertical line in the number of signatures, because it’s suddenly thousands and thousands of people signing… Because of course, BBC has this incredible authority. If the BBC says this is something, people read it, it gets copied around the world… After that I was on the TV and all over the place, talking about it. At that point, 30,000 people signed it.

Parham Doustdar: Well, I actually… One of the very neat features of a screen reader is… Well, there are actually two. But the first one is that you can control the mouse, that’s really cool. So I can actually focus on a line and then I tell my screen reader to bring the mouse pointer here and I can just tell it to activate the left click functionality on this side, so that’s really cool. But the second thing is that most people think of screen readers as a software that just starts reading a window from top to bottom, without actually giving you much control, but in reality you can actually jump around. What I actually do when it comes to reading stack traces is I just read the first line - well, depending on what language I’m using. In Python, I just read the last line, because that’s where… The whole stack doesn’t make sense, but then you read the last line and you’re like, “Oh, okay…” But when I wasn’t familiar with the Python stack traces, I would just look at that and be like, “Wow, where does this even start? I had no idea.”

So it’s more of a habit. You need to form a lot of habits as a blind person, to get around really fast. Because there is all this useless information and you kind of have to filter that out. It’s mostly mental. Your screen reader doesn’t do it, your eyes don’t do it, because there is no highlighting. For example, one thing that… You know, I read at 530 words per minute, right now - it’s going up to 550 pretty soon - and most people say, “Wow that’s really amazing”, and what I usually point out at this time is that you actually do read more than that, and that’s because you have the ability to skim.

Like, imagine that you want to have a list of transactions, like 100 transactions on a page, and you want to quickly look at a list of successful ones, for example. Most applications provide the successful ones in green and the failed ones in red, so what you can do is just scroll down, have a quick look and you’re like, “Okay, so most of that has failed.” But me - no, I have to go through every single row of that data. The most I can do is just focus on the status column and just tell my screen reader to read that column, so what I hear is “Success - success - failed - failed - failed - success…” But still, there’s no way of actually doing that as quickly as you can with sight, so there’s a…

Zach Latta: Yeah. So Hack Club today is a few key programs. The first is there’s a massive online community. It’s all ran through Slack. There are 25,000 teenagers that are a part of it. We’re about to cross 10 million messages sent, and it’s one of the most active online discussion spaces for teenage coders anywhere. And the discussions range from what it’s like being a teenager, to like people do really highly technical software in there.

One of the projects that was built now a few years ago by a Hack Clubber was called Nearly.js. It’s a parsing library for JavaScript. It is now downloaded 2 million times a week on npm. And jQuery is downloaded 6 million times a week on npm, just to give that some perspective. And this is something where it’s like – that was built by an 18-year-old at the time, in their Hack Club meetings, and talking about some of that work on the Hack Club’s Slack as they were doing it.

The second part of that club is just hackathons. So these are 24-hour-long coding marathons that happen on weekends, and they’re all teenager-organized. There’s roughly 50 to 100 that happen a year regionally, and those are all led by teenagers.

The third is there’s hundreds of after school Hack Club chapters where teenagers get together weekly to code together. These tend to be more beginner-oriented, because again, over 50% of high schools in the US don’t offer a single coding class, and then in a lot of the schools this is the coding thing that exists. And what’s cool is when you come to a meeting, it’s not like you’re signing up for a semester-long commitment as a young person; you’re just seeing “Is coding something I’m into for an hour?” And as a result, you’re also writing code that’s meaningful and relevant to you. You’re shipping a project every week. So it’s real contextual, everything you’re doing.

And then finally - you know, the areas where I think Hack Club is really interesting and is really unique is we’re really the first major educational organization, structured and formed after the internet already existed. And what that means is that the internet is part of Hack Club’s DNA in a way where you look at other organizations, they’re still kind of trying to figure out how the Internet affects their organizing. And one thing that happens at Hack Club is anytime teenagers run into problems, internal tools that are open source get built by the community, that everyone starts using.

And that brings us to our final program, which we call Hack Club Bank. This is a financial tool. It’s almost like Stripe Atlas, but for nonprofits, where if you want to start a nonprofit, or if you need a way to receive donations - and we originally formed it because our teenagers kept trying to run these events that had no way to receive money. Because if you’re under the age of 18, you can’t open a bank account in most of the country. It’s a financial tool. If you go to hackclub.com/bank, with one click you can proceed – you get 501(C)3 nonprofit status, you can receive donations, you get physical cards for spending funds, you can manage it with your team… And now there’s 1010 organizations, many of them led by teenagers, that run through Hack Club Bank, and there are millions of dollars that we process on behalf of these groups all over the country each year.

[41:54] So those are kind of our key programs today. So there’s the online community, there’s clubs, there’s hackathons, there’s Hack Club Bank… And then we also do seasonal events and activities. Like, one thing we did a few months ago was we we did a project called Winter Hardware Wonderland. If you go to HackClub.com/winter, where we did an open call and we said “Hey, if you’re a teenager and you want to build a hardware project, and you’ve never done that before, buying components is expensive. So we’ll buy all the components you need up to $250 per project if you submit a pull request to this GitHub repo with your stuff, if you meet the requirements and whatnot.” And in total, we have hundreds of projects built from like dozens of countries all over the world.

The projects ranged from like - there was this one student I think in Greece who built a plant soil monitoring system for their parents’ garden, that helps you understand if the soil has a right components and the right setup to grow the plants that you’re trying to grow. There’s this one student in New York City who built a foldable kayak from scratch; they wanted to get into woodworking, so they wanted – it’s kind of crazy. Their final video submission was them in the kayak, in the Hudson.

Mark Bates: That was, that was good. You should keep making those jokes. Anywho… And then, of course, things like this. This is nearly 600 pages of technical writing, the book. And Cory said that we wrote this book by accident, because we were working on a 12-week course that we were going to do. We did a 12-week fundamental course, and it went really, really well. It was like a master-style college-level course; it was so good. But then we got so busy with the corporate and then writing the book, we unfortunately were never able to actually get back to that. But that class that I was writing became this book, accidentally. And it’s all because of this tool that Cory was talking about, that we’ve built to help us fix these problems we have with all this documentation we were just talking about - the maintenance, the upkeep… And portability, right? We write code, we write documentation, but there are many places where we need our documentation. So for us, we’re writing all this content, we’re doing all this heavy, deep technical writing, we’re publishing it to the web, maybe as blog posts, or in a slide format, or republishing it to Canvas, which is like the world’s leading learning management system, via a Rails API that they have. Or LaTeX, when we did the book, the publishing company Addison-Wesley needed LaTeX, so we had to take our content and give it to them LaTeX.

So we have all these different places we need to put it: databases, and APIs, and HTML, LaTeX. And the last thing we wanted to do was rewrite all of that same content for all those different places, and have to spend a lot of time reworking that content. Because every time you do that, if you keep splintering that content out to all these different places, and it’s not centralized, then that maintenance nightmare we were just talking about becomes exponential. So if I want to update now a course in Canvas, that LMS system, I now have to go there and do all the updates there. And then I have to make the same updates to LaTeX and the same – whatever it is.

As documentation writers, as technical writers, we need to be able to disassociate what we write from the presentation layer, and where it goes. And a lot of tools out there connect those two, or they disconnect them in such an abstract way that it’s – like, it’s all via JSON, and all these like configuration stuff, that you’re now caught up in that.

[26:08] What we wanted to do, and what we had to do is we wanted to work in editors, and we wanted to work in our editors, in Vim, or VS Code, or what have you, we wanted to work with local files, we wanted to be able to write in something simple and easy, that we know, like Markdown… We all write our readmes in Markdown, it’s a nice format for doing that, but it’s limited. It doesn’t allow us to do things like insert code samples, snippets from files, or run output and include it automatically; or inject go doc directly into the documentation, or any of those sorts of things. Or validate it, or allow us to pull it apart and do all these things.

So essentially, what we did was we just built that system. We took markdown, and added the ability to put some custom HTML tags in there, and then from there, we were able to parse it and execute the document and do all sorts of different things, and then put them wherever we want. We can execute this document and have all of the code samples, and we can put in there “Okay, I want the Go doc for io.EOF, and it’s going to go, and it’s going to run Go doc, and it’s going to capture that output and it’s going to stick it in for us.”

If I say “I want to run here, I want to show this piece of code, and then I want to run go build, or go run, dot, or whatever it is I want to run, and I expect that this code sample is going to fail, because I’m showing something that panics.” So I can even say “I want to make sure that it exits with a one or a two”, or whatever.

And so we can put these nice little simple HTML tags in there, we can execute the document, and it’s going to fill in all of that. And if there are problems, we’re going to get back errors immediately, telling us our code was wrong, the exit status was wrong, the image didn’t exist that we were looking for, the included file didn’t exist… Like, there’s all these different validations and kind of understanding we can get by compiling the markdown document like this. And then we can take all those different elements and post them up via APIs, put them into a database, write HTML, create LaTeX, whatever we want.

But the key for us was really writing and keeping things very, very simple. Local files, Markdown, and HTML. There are tools already in every editor for Markdown, HTML; we all know Markdown and HTML. There’s no learning curve; it’s just practically a very little learning curve to do it. And we just work in our editors, and everything just kind of falls into place, and it’s just such a nice way to work, to know that every time we make a change to that code, that we’re being told whether our documentation is valid or not. And that’s the big thing for us.

Feross Aboukhadijeh: Yeah, I can get the link. I think it’s Chrome UX Report, maybe. So search for “Chrome UX Report.” It’s a little bit of a process to set this up initially for your site, but there’s a little tutorial that they have up there. What it’ll do is basically create a little PowerPoint type thing that shows you the historical data. It’s unnecessarily complicated the way you set it up, but the data is really good.

You can also use this thing called Google Page Speed Insights. That one’s a lot easier to use. You just type in a URL and it’ll tell you the real-world page load times for that site… But it’s only the current stats. So as of yesterday, what are the numbers for that site. You can’t get the historical data if you just use that tool, but it’s pretty good to get an idea of where your site falls on the user experience spectrum, I guess.

Jerod Santo: I do enjoy that, yes. So that was one big piece of work… The other thing - Adam, you mentioned it; it’s in flight right now - we’re swapping out Algolia for Typesense, which is a very cool C++ based search index, search engine, open source, that we had on the Changelog… Jason Bosco, we had him on the Changelog last year. I really liked the guy, got really interested in the product. Algolia has been kind of – we were on the Algolia, and we still are on the Algolia open source plan, which sets us a limit… And so when we’ve hit that limit, and we’ve been putting new things into the Algolia index ever since, but it won’t search them until we upgrade our plan… So we’re happy to be replacing Algolia with Typesense. Of course, that’s an open source thing, but we’re working on a partnership with Jason and his team, so that we’ll be using Typesense Cloud. All that’s very close to at least being swap-out-ready, and then we’re going to build from there and start to use some of the things that make Typesense interesting. So I’ve been coding that…

And then the third thing is trying to rejigger the way that our feeds are generated and cached and stored in order to get to this clustered world of multiple nodes running the apps, without having to change the way we use Erlang’s built-in caching system, because I’ve just had some issues with that… And I just started thinking, “Why are we caching stuff if we have a very fast application, that can just run close to the user? Let’s just figure out a way not to cache stuff as much.” But we have these very expensive pages, specifically the feeds: Master feed, Changelog feed… I mean, the XML that gets generated is like 2.3 megabytes. It’s not going to be fast on any system, unless it’s literally pre-computed.

So I started thinking about different ways of pre-computing and storing files on S3, and fronting that… And there’s just lots of concerns with publishing immediately; we like to publish fast. And we even had a problem - thanks to a listener who pointed it out - with our Overcast ping, because Overcast as a specific app allows you to ping it immediately on publish, and they’ll just push notify, and people will get their things immediately… Which some people really like that. I’m always surprised - there’s some listeners who listen like right when it drops, and there’s others who listen like six months later. And that’s all well and good, but for the ones who want it now - it’s cool, we add the Overcast Bing. Well, there’s an issue there, because Overcast pings, but we’re caching our feeds for a few minutes, maybe just a minute. And so Overcast says there’s a new episode, and so you click on it, and you go there, and there isn’t a new episode. And then you refresh, it’s not there, then you refresh, it’s not there, then you refresh it and it is there, and it was like 60 seconds… Because we’re caching.

[46:14] So I just turned that thing off and thought, “Well, people can just wait for Overcast to crawl us again, for now, but I would love to solve that problem…” And so then I started thinking, you know, we already have a place where we store data, that’s a single instance, but is a service, so to speak, and it’s called Postgres. And instead of adding like a memcached, or Redis, or figuring out these caching issues inside of the Erlang system, which was not trivial in my research, I was like “What if we just precompute and throw stuff into Postgres?” And I did a test run of that, the feeds; just the feeds. And just turn off all other caching, because I don’t think we actually need any other caching. It’s just like, I already had caching setup, so I cached a few popular pages… But what if I just did it on the feeds? And every time you publish, you just blow it away, rerun it, and put it in Postgres. And you just serve it as static content out of Postgres.

I did some initial testing on that locally, and it’s like consistently 50-millisecond responses with like Apache Bench, it was not a problem. It’s never super-fast, like what you get with Erlang, where it’s like microseconds… Which I always like to see those stats. But that’s not what we need, right? Consistently 50 milliseconds is great.

Justin Garrison: Like “Oh, I can do this at my house with an old laptop, and then I can show people I did it. I could show people, and they actually use it and consume it and say “Oh, this is something that is available a lot of times.” And even when you have outages, how do you communicate that? Because those are all things that you have to do at work, and you have to do those things for a job if you’re running infrastructure, and being able to do that in like a voluntary way, of saying like “Oh, I’m interested in this thing. Let’s see how it works. Oh, do I have a status page?”

Marques Johansson: You kind of don’t wanna know that, right? …when there’s some sort of status page outage, something wrong in an Equinix facility, it is a big deal, so it’s good to not hear about those things, to have them not actually happen.

Guillermo Rauch: And also, sites of a certain size can not just pre-render every page… And together with all of that, when we’re talking about mobile web versus native, a tremendous advantage that I see for the web is that when you tap on a link, it can render just in time the page just for you, without you having to have downloaded a massive application with all the possible code paths, not just for you, but for every known language, every known variant, every experiment, and every other cohort of users.

The web has this magical – and it goes back to my concept of the hyperlink… You go to this hyperlink, and it’s almost like a black box; you don’t know what’s gonna come out on the other side. And that has this beauty of “I can give you the specific set of code, I can give you the specific HTML, and the specific CSS for what you need, just in time, at that very time.” And that’s only possible if you combine ideas of JAMstack with more traditional ideas, like server rendering.

So this concept that we pioneered with Next.js, this is what frankly drove and got us the attention from Google Chrome’s team, that started contributing directly to Next.js, because it matched very well what they’d seen internally at Google as being the best ways to scale really large sites and really large applications.

[32:14] It’s always with a combination of CDN caching for the parts that are known to be static, but also leveraging what I call the power asymmetry of the world, which is - the average mobile phone in India, which is the Xiaomi Redmi phone, is not gonna be as powerful as a server’s pre-warmed with all your application code, sitting on a data center. And not only will it not be as powerful, but by transferring some of the rendering load to that powerful server, you reduce the battery life, you reduce the download amount they are putting on that device.

So I think one of the ways forward for the web will be to acknowledge that there is no silver bullet where you say “Oh, if I make everything static, and if I JAMstack this or that, I’m gonna get great performance”, because you might also limit the ability to customize, personalize and create very dynamic content just for a certain user, context, country, language at a certain moment… But not only that. As I mentioned with Web Vitals, one thing that we’ve learned as well in 2020 - and before that, but it became very clear in 2020 - is just because a CDN can give you HTML really quick doesn’t mean that you’re gonna have a fast website.

The example I like to give here is when Facebook launched the new Facebook.com built entirely on top of React, which was supposed to be this engineering marvel, because the open web is so introspectible and debuggable, a developer went ahead and inspected it and realized that they weren’t even using CSS that much. They weren’t using box shadow, and they were using spacer divs.

And the box shadow one was particularly puzzling to me, because I remember the old days of the web, where we didn’t have the box shadow property, and we had to use background images… Or not even background images, we had these image tags. And Facebook was doing that because they realized that for that particular element that was floating on the screen, box shadow had really bad performance.

So performance is not just a matter of like you downloaded that box shadow property in-line into the HTML really quickly from the edge. That obviously is gonna help. But then begins this other complete new box of surprises, or Pandora box of performance. And that’s where the Web Vitals metrics that Google has created are extremely helpful… Because if you are experienced with backend engineering, you know that one of the easy things to measure in this pipeline is “Okay, I’m gonna measure the p99 percentile distribution of response times and render times.” In fact, github.com used to include this in their status page. Status.github.com would show you the mean page render time, the p95 page render time etc. And you would say “Okay, it’s 300 milliseconds.” And you’re like “Okay, that’s really good.” It’s like two blinks of an eye, or whatever. The threshold of attention, or whatever.

But then you realize “Okay, I gave that to the web browser, and now the web browser is struggling with rendering box shadows, or Intercom widgets, and GDPR pop-ups, and all the layout is shifting, and there’s three cookie banners, and another one to accept the terms of service… Actually, that one is fresh from today.

[35:54] Today I went to a website… This is a marketing website. And it was presenting two blocking modals. One to accept Terms of Service, and I was like “Terms of Service of what? I’m just visiting this website.” And the other one is a massive GDPR modal. And both of those things kind of came in a little later…