Jamie Tanna: Yes. So there’s a really good article titled “One year in the indie web”, and this is the one that I usually recommend people read, because there’s a really good distinction there between IndieWeb, with a capital I and a capital W, versus indie web all lowercase, or with a space between indie and web. So there’s a lot of people who think indie web is “I need to do these particular things to be part of the indie web.” Whereas at its base, it’s just you own a URL, and you publish content to it. It could be that URL is in front of Substack, or Medium, it could be in front of wordpress.com, or it could be infrastructure that you are hosting in your home lab, and you’ve got a load of stuff for that. But then as well as that, it has also morphed into things like owning data across different services.. So one of the things you reference is POSSEE, Publish on Your Own Site, Syndicate Everywhere Else, or Elsewhere. And so that is a lot of what I do, or what I try to do, which is in the before times, before Twitter’s API got taken down, I would reply to people’s tweets from my website, and so I would actually go through a lot more pain and effort to reply to someone’s post… Which kind of showed that I actually did care a little bit more, because I would – so using the Twitter app, I would share with an indie web app, I would write my reply, and then hit Send, that would go to my website. In about 10 minutes, my site would have rebuilt. It would have sent a notification to an app called Bridgy. And Bridgy would then say “Oh, cool. This has replaced something in Twitter, so I’m now going to send an API request to Twitter, and I’m going to syndicate that post. And that will then appear on twitter.com.” And in all that time, someone’s probably not bothered by my reply anyway…

Gerhard Lazu: So you can run just a single one… So a couple of things. First of all, this migration was like a progressive rollout. So certain apps of ours - we’ve received notifications like “Hey, these apps will be migrated to from v1 to v2 within like the next week.” And then our app, the Changelog app - it required a blue/green deployment strategy. Blue/green was not implemented for Apps v2 until maybe a month ago, maybe two months ago… Something – and actually, it was like a month ago. Because two months ago they didn’t have this option. So a month ago, this was enabled. Shortly after, I think a week after, we received this notification, “Hey, this app will be migrated to v2.” But the problem for us is that our deployment configuration - I wasn’t sure whether it will work with v2. Because what they say is, “Hey, if you have a chance, try and save the Fly config after the upgrade, because some things may have changed in the configuration.”

So in our case, we didn’t have to do that. Everything continued working, which was a nice surprise… But when you go to v2 and when you go to Machines, or Apps v2, their strong recommendation is to run more than one. And the reason for that is the app will not be automatically placed on another host if a host was to have a physical failure.

Michael Grinich: When I was an undergrad in college at MIT, I met a bunch of the folks that were working at Dropbox early on. And this is when Dropbox was pretty small. They were like 50 people, or something. Dropbox was started by MIT alums. So they would come to campus, and come to the recruiting fair, and throw parties, and stuff like that… And I ended up spending about six months working at Dropbox, doing some mobile engineering there and building stuff. This is back in 2011. So the company was just starting to grow faster. I think when I went back to school, it had like doubled in size. And the thing I really thought about when I was there - it was not just the mobile app, but how Dropbox was getting used by companies, really businesses, and the positioning we had against other products like Box. We’d actually hired a sales guy from Box back then, and just started thinking about why is it like Box is winning in these business contexts when me and all my friends used Dropbox and it was clearly a better product.

So I thought about that a little bit, but obviously not too much. I went back to school, graduated, and then after graduating, I had been working on this email project. It was actually like my capstone project in the computer science department. So the codebase was there… And it was this thing to synchronize email and build data kind of visualizations on top of email. And I’d always wanted to build an email client. I felt like the email client is sort of the view into the database of your life. At MIT everything runs through email, and I was just like obsessed with email clients.

And so I started this company to try to rebuild Outlook or Gmail with a much more modern take. And to build something that’s really fast, and elegant, easy to use, and something hopefully that would help people communicate and get their job done a lot better. So I figured if you can help people communicate faster, help people kind of structure their thinking better, and help teams work better together, you have this huge effect on productivity, and it’s a high-leverage thing you can do.

So I started working on that, took that codebase, and that became the core of the mailsync engine, and then built this app on top of it. And we obsessed around the design. Really obsessed around the experience of this app. I spent like three weeks looking at fonts, just typefaces – because email’s all text, right? The interface has mostly fonts. Working with a couple designers that – I really, really loved that time working with them, on every single individual icon.

I hired a musician that did sounds for video games, and went to their studio for a couple days and just listened to like notification mail sounds that we were going to put in the app… Because the new mail notification sound you hear all the time. So we really went deep on this, right? Like we went really, really deep on making the experience great. We did all this amazing engineering to make it extremely performant on multiple devices. We had everything from MacBooks to like little Surface tablets, underpowered, that we were making sure it rendered properly, and worked offline.

So anyway, we obsessed around the design of this product. We launch it, and people love it. And it was also open source. So developers loved it, it became a top project on GitHub, and we got all these people signing up. And I had just gone through the Dropbox playbook, right? Build a great experience, that people really like, that’s really useful, put it out there in the world, and people will start using it at work, as well as their personal life.

[07:46] So we started getting users everywhere from like NASA, to Uber, to Facebook, to the city of Minneapolis… Every single kind of imaginable type of user was just downloading it, signing in and using it. And the playbook I had was like “Hey, do the same thing as Dropbox. Just go find the organizations that are adopting this and we’ll sell it to them.” It’s kind of the bottom-up freemium type of business model.

And what happened is a few years later, as we started getting this adoption, I would go in those companies, like Uber, for example, and say, “Hey, there’s 100 people here using Nylas mail. Let’s get you guys on a team plan. Let’s get you to pay for it for the whole organization.” And this was the first time I really learned about any of these things around enterprise features. I had never talked to like a person in IT before that, that type of moment. And what I realized is the same thing that was constraining Dropbox’s growth back then, I was starting to have as well, which is that you could get this amazing adoption and really product-market fit, with the end users signing up and using the product, and using it every day, in many cases… But when it came to actually commercializing and building a business around it, there was this totally different audience, which was the IT admin, or the procurement team, and they didn’t care anything about the design of it, or the fonts, or the sounds I’d put in it, or why users liked it. They pretty much said that they can’t use it unless it had all these seemingly obtuse enterprise features. And in some cases, some organizations actually shut it down. They didn’t know people were using it, and after they’ve learned about it, because it wasn’t enterprise-ready, they shut it down.

Ultimately, this meant we didn’t commercialize that product successfully. We got some people to pay for it, but it really wasn’t growing in that means… And we’ve found that “Hey, in order to actually build this enterprise stuff, it’s going to take an incredible amount of time.” And during that moment, I actually went and talked to friends of mine working at companies like Asana, or Slack early on, and they were all building this enterprise stuff. They’d kind of stopped working on the product features.

So we ended up just fully open sourcing it, shutting it down, pivoting the company… Nylas is still around, and doing well, and growing… But that idea of building a mail client essentially stopped. That died. And I took some time off, and when I was really trying to think about what to do next, I just kept coming back to this problem. I was like “Why did we build the product everyone loved it?” I still talk to people who say “I used that app, it was really great. I really loved it.” Why did we build the thing that everyone loved, but that wasn’t enough?

Paul Graham famously has written and said “Make something people want. If you do that, everything else will take care of itself.” And it obviously didn’t. There was this missing step. And I realized that getting product-market fit - it’s necessary, but it’s really not sufficient to building a business. And if you don’t think about the business side, you don’t really think about how you’re going to commercialize, and who is the buyer, you get stuck. And it might be in year one, it could be in year five… You might be able to squeeze through for a little while, but ultimately, you’ve got to figure out how to go to those actual commercial buyers and satisfy them.

And what I realized was – why WorkOS exists is all this stuff that you need, you can really put it into a software layer. And that’s what we’ve done. So things like authentication, and user management, security features, things like audit logging… All this kind of under-the-hood stuff that IT admins need - it’s pretty complex to build, it turns out. It takes a lot of time, you have to have really good engineers to do it… But the features at Dropbox that they ended up building for enterprise are the exact same stuff that Slack built, it’s the exact same stuff that Asana built. People just kept doing it again, and again, and again. And so I was like “Hey, what if I could take all these features and turn it into a product, kind of like Stripe, or something like that? …they’ve done that for payments, but let’s do it for enterprise stuff.” And that’s WorkOS. That’s the idea.

So I’m not clairvoyant, or anything, I just looked at the hard thing that everyone was doing, the hard thing that I had struggled to do at my first company, and then said, “Hey, let’s just kind of go a couple notches down the abstraction chain and solve this from an infrastructure perspective for everyone.” And it turns out we were so far not too wrong. People are using it, and it’s helping people… But I kind of stumbled into this world of infrastructure for enterprise apps really through building an app myself.

Jon Calhoun: So counter-argument, I guess… What happens when you don’t know the color? Like, the example I’ll give is I have Substack on my phone. And for whatever reason, the notification icon on Substack is green, but the app icon is orange. So for the longest time, when I was looking for it, I was looking for a green icon, until I realized that it wasn’t green. I’m pretty sure it’s an orange app or icon on my phone.

Robert Ross: Yeah, I think you nailed it. I mean, there’s so many surprising companies coming out now… And sorry, by surprising companies I mean companies that you wouldn’t expect to have amazing software. I have a Roomba. And the Roomba app is fantastic. It’s internet-connected, it does a push notification to me when it’s done… My vacuum cleaner has no business having amazing software behind it, as far as I’m concerned…

Break: The top story from Sunday’s edition of Changelog Weekly was Matt Healy announcing, “Your next smart home device is a $30 used Kindle”. He says:

“… a lot of tools that used to be separate have now converged into one: the phone.

You might not buy another camera again, and you probably won’t buy a calculator. For the most part, it’s a good trade-off: what we lose in the tactile pleasures of a good single-purpose device, we gain back tenfold in having all those devices in our pocket all the time.

That’s all fine for calculators, cameras, music players, and the like — things that are there when we need to use them, and can forget about when we don’t. But our phones are also the nexus of all kinds of data we want pushed to us — news, messages, notifications, alerts, status updates. Things we want to know about without going looking for them.

A phone isn’t actually great for this kind of ambient awareness. It’s usually in a pocket or purse, so the ways it can get our attention are limited: a buzz, a notification on the lock screen, a red dot on an app icon. I can choose to let myself be interrupted, or I can compulsively check in to see what the latest alerts might be.

I’ve found myself wanting a less intrusive channel for ambient data. Something I can glance at in passing, without giving my full attention or opening myself up to whatever other interruptions my phone might have in store. It should be easy to check, but just as easy to ignore.”

The device he landed on is, you guessed it, a wifi-connected Kindle 4 mounted on the wall. In the post he walks you through why this is the perfect setup: why e-ink, why a kindle, why a screenshot of a web page… he also provides instructions so you can set up a $30 Kindle to do the same.

Philipp Heckel: Yeah, so it’s interesting, you’re hitting on the “How does it work under the hood?”, the message delivery, which maybe is a good small segue. So ntfy.sh does use Firebase, or Firebase Messaging - what is it called; FCM, Firebase Cloud Messaging - to deliver messages through Google. And then for Apple, for iOS devices, Firebase itself will forward to APNS, which is Apple’s Apple Push Notification Service, which in turn will then deliver that to iPhones. There is other ways to deliver messages. For self-hosted instances, they don’t go through Google. They go through regular WebSockets or HTTP stream. So obviously, there’s no limits there. Interestingly, Google doesn’t have limits on FCM. I’m literally pushing –

Sam Lambert: Yeah, so the adoption at GitHub was bumpy, like it is with every – so Vitess is an amazingly powerful tool, but it’s not perfect. And what it does for you is way more than – you know, you just couldn’t build it yourself; starting from scratch to do all this yourself would just be wild, and take way too long. So what Vitess gives you in return is you cut to the chase very quickly on scaling, but if you’re just installing it yourself, you’re likely gonna need a team of engineers to learn. It’s like with every system, you have to learn the quirks, you have to learn its failure modes.

So we did it incrementally… We have a great team of application engineers at GitHub that started to split specific tables out of the database and put it in Vitess. So there was a lot of very, very large tables, that just kept growing and growing… And everyone has these in their application. Your notification tables was one that was just a complete pain. And it was just like, every notification on GitHub just got put in this table, so it just started to get massive. The others is like statuses, or whatever.

So you know like at the bottom of your pull request, when all the CI checks come back, and GitHub was developing the product around this, and all of these statuses for pull requests, like pass, fail, whatever, was just, again, all piling up into a massive table. And people look at pull requests that are like three years old and they might wanna resurrect them, so these statuses needed to be there… So again, it was just like massive amounts of data. And I think around the time GitHub was probably around 30 million users at the time, with a lot of heavy usage.

My favorite thing to look at in the usage graphs at GitHub was you could see all of the cron ticks for the world, because everyone had crons that would pull their repo down and do testing. And you could see this spike through all of the graphs, whether it was like frontend, CPU load, whether it was database queries… Every single performance graph at GitHub had these ticks, had these large spikes. On the hour was the largest, then half an hour, then 15, 10, 5, 1 minute… And you would just see these little – because all of the world’s crons are running to pull code down from GitHub and start up CI processes.

[24:21] Anyway, we were trying to scale that, so we adopted things incrementally, table by table, and moved them over to Vitess, and it just worked fantastically, and we still see the enthusiasm from the different engineering teams that get to use it… That’s how I got involved.

So then when I was here at PlanetScale, I thought “We can democratize this tech. We can give it to everybody”, and then it came to the team. And I have to give all of the credit to the great team that we have at PlanetScale. Not only do we have the Vitess team, that are just literal wizards… They’re writing query planners, and you look at the annotation and their notes in the formula that they’re working, and I just look at this and just think “I don’t know anything about computers.” [laughter] You just think, “Wow, these people are phenomenal.”

Marcos Nils: So we made a lot of mistakes by building the platform. I guess a lot of the mistakes that we made were also contextual to the infrastructure that we had, the decisions that we made… But one of the things that I really recall - I don’t know if regretting, or like learning a lot of things the hard way, is that because we had an on-premise deployment or an on-premise thing, we wanted to build a lot of managed services for teams. For instance, if you had to deploy a memcache cluster, if you wanted a MySQL cluster, or those basic services, Elasticsearch as well - so there were two ways that you could do it. And remember, I’m talking about sub-1000 engineer organizations here. So there’s a bunch of people that usually don’t communicate with each other, and they reinvent the wheel multiple times; that usually happens.

[30:20] So what we did is that we tried to – from the architecture team, we tried to come up with services similar to whatever you can find in any cloud today RDS, Elasticache, whatever… But we tried to come up with those services ourselves. And we tried to do it in a fast-paced kind of way, where we could show developers that we were shipping fast.

So I recall that the first thing that we did was called BQ, which was an analogous thing to SQS, I would say; it was like a queue service where you could push a message and then consume it from somewhere else, and it was our initial approach to deliver an eventful system, event-driven storage system between applications, so you could get a notification when an entity changed, and then that got replicated all over the place, and you don’t lose a message, whatever…

And the thing that I remember the most was, first of all, it is very difficult to build a high scalable, high throughput distributed systems yourself, right? There’s PhDs out there that are actually doing this S3, SQS, whatever, and we were just a bunch of senior software engineers, but trying to tackle very challenging problems. So we found a lot of issues across that way. And even though we solved the problem for some time, eventually you had to basically re-adopt or basically redo all that work, because it wasn’t scaling. So we were basically trying to chew a little bit of a bigger bite than we can actually [unintelligible 00:32:11.16]

So yeah, we made some of those mistakes multiple times. So we tried to come up with this service, with [unintelligible 00:32:19.04], which - it worked pretty well for some time, but then it wasn’t scaling anymore. It was built also on top of Node.js, which was around that time 0.4, very early days, we were adopting very edgy technologies as well… So I guess we fell into the trap of, “Hey, we are at the very top of the of the wave, we’re riding the wave on the very, very edge, so let’s try to do crazy things. Let’s try to copy Amazon in what we’re trying to do”, and then those weren’t really good decisions.

Alex Rickabaugh: So Angular JS to Angular - that transition was almost 10 years ago now, that we were going through this rethinking of how the framework would work. And 10 years is about the right time for those kinds of introspection moments. So around 2021, post pandemic, the Angular team was able to be together in-person for the first time since switching to remote work. We sat down and held a – I to call it a design review after the fact, where we did a lot of reflection on what state the framework was in, all of the different design decisions that had gone into making Angular what it was over the years… Those foundational pieces, and whether they were still working for us. Some things were no longer relevant, because the web had evolved in that timeframe, the job of the framework had shifted significantly…

Frameworks - their main job used to be making the experience the same across all of the different browsers, especially Internet Explorer. And now, in the 2020 era, frameworks are more – like, their primary focus is on offering the best possible developer experience and velocity, really being a tool that enables developers to code and move much more quickly, and ship features much more quickly.

So we were looking at Angular through this lens, of what still will work for us for the next 10 years, and which pieces might need changing. And one of the things that we identified as needing to invest in is the framework’s reactivity story, which was built around this library called Zone.js… And how Zone interacted with the browser, and monkey-patched all the different browser APIs to give us this notification that something might have happened in the application that the framework needs to handle. And for a variety of different reasons, this pattern we didn’t think was going to be viable for the next 10 years. Some of that is related to browsers and browser technology; Zone physically can’t interact with async/await. So we were seeing new browser features that were incompatible with this model.

At the same time, Angular applications, Angular as a community was scaling, our applications were getting bigger, people were writing larger and larger projects with it, and we were seeing some of the cracks of the Zone.js model, where this approach of “You just code, you just write your state updates, and the framework will figure it out”, that scales to a point, and then it stops working. You start having to actually really care about what’s going on. Or you end up with situations where the performance of the application gets really bad, because you’ve tripped over some edge case without realizing it.

[00:10:03.15] And so we started looking around for what is the foundational piece that we can replace Zone.js with. How can we overhaul the reactivity model of the framework to carry us for the next 10 years, and address the core problems with the Zone.js designs. And after a number of sessions with Pavel, where we sat down and tried out all kinds of different ideas, Signals came to the forefront as checking all of the boxes for what we were really looking for.

Jonathan Norris: [50:09] Yeah. Don’t get me wrong, those VMs have gotten very good over many years, and we’ve been relying on them for a lot of our high-scale systems… But yeah, I think there’s just an order of magnitude difference between the size of containers – like, yeah, you can optimize your size of your containers to be pretty small, like tens of megabytes size… But WebAssembly is, at its core, designed to be more portable than that, where it’s – you’re talking about tens of kilobytes, instead of tens of megabytes. And the boot-up times can be measured in microseconds, instead of milliseconds, or tens of milliseconds, or even seconds for containers. So there’s just an order of magnitude change there by using WebAssembly, that I think it’s gonna be really hard for a lot of containerized systems to match.

You can think about a big platform running at the edge, at scale, where – say for us, for our use case, we have a lot of SDKs that hit our edge APIs. And we have certain customers, say our big mobile mobile apps… And they may send out a push notification and get hundreds of thousands of people, or even millions of people who all open their app at exactly the same time, when that sports score, or that big news event lands on their phone; they’re opening their app at exactly the same time, and we see massive deluges of traffic - literally, a hundred times our steady state traffic - hit our edge endpoints in those points in time. And because we’re using these edge platforms, they’re able to spin up thousands of runtimes of Wasm and edge runtimes in milliseconds to serve that traffic. And having to do that with VMs is possible, but there’s a lot more latency in that toolchain.

So that’s why I think the power of not only the really tight security model, but the boot-up times, the small size of the Wasm modules really can power that. And for certain use cases it makes a lot of sense. I’m not gonna say it’s gonna replace every use case; it’s clearly not. But for certain high-performance latency-sensitive use cases, trying to deliver feature flags globally to mobile apps, or web apps around the world - that is our use case, and it’s definitely very applicable to this problem.

Alex Moore: Before the Blink fork. So at this point we were still developing web apps, and we were waiting for these major features… But then Blink and WebKit fork. So suddenly you have all the features that were getting piled into WebKit suddenly sort of get cut off, and now they’re only going into Chromium and Blink. And then the gap between what was possible on Android and Firefox, and what was possible in Safari started to get bigger and bigger as the time went on.

Then what we experienced was just an enormous number of bugs. Literally almost every second release Safari would break something critical. So there’s IndexDB, which the listeners will be familiar with; it’s for local storage. That would quite often break all the time, to the point there James and I used to joke it was built by the work experience kid at Apple… Because it was just so unstable, and parts of it didn’t work.

Now, that was unfair. The reality actually is that Apple’s investment in Safari was just so small that you couldn’t possibly work on all these different things at the same time and have them work. So it wasn’t a skill issue, it was simply not investing enough resources… Because they had no incentive; they moved parts of their team off onto their native app ecosystem to focus on that instead.

But we persisted, and we were still building web apps, including for iOS… Because you don’t get a choice where your customers are. You have to build for your customers. And in particular, you have to build for the phone of the CEO, or the CTO of the company you’re building for.

So that kept on going till 2015-2016, but the noise from developers was getting bigger and bigger. They’re like “Right, we want push notifications.” But Apple would pretty much – they’d give these sort of proforma non-responses, like “We hear you” or “We wanna hear about the use cases”, for which our response would always be “Well, the use cases are the same as they are for all the native apps. They’re identical.” Do you want a chat app without notifications? Do you want a social media app without chat notifications…?

[00:13:47.27] We got all the way to 2020, and James and I at the time were writing software, doing a point of sale system… And the first thing our customer asked us was “Hang on… These printers you’re making us buy are really expensive. Can we use Bluetooth printers?” And we’re like “Nah, sorry. Apple doesn’t support web Bluetooth. We can do it for your Android devices, we can’t do it for all the iPads.” Because it was a franchisee kind of arrangement, they obviously couldn’t get all the franchisees to swap out all of their iPads with Android devices, so it just wasn’t an option.

Then about a few months later he came up and he said “Oh, for the product ordering system we really need notifications when products turn up.” And again, we said to him “Well, we can’t do that, because Apple hasn’t implemented that functionality”, to which he said “Why can’t I just install another browser and get notifications on that browser?” And we’re like “Ah, because they’re all essentially the same.” And that got me thinking, I was like “Yeah, that’s actually ridiculous.” Obviously, if there was another browser, you could just tell your users to switch to that browser. And the fact there isn’t a browser means that Apple has no competitive pressure to implement any of these features, or invest in Safari, or make it stable, because there is no fear of losing users to another browser.

If there’s a huge bug in Safari, that breaks lots of apps, then the apps rightly - and this is a good use case for it - could say “Look, it’s broken in this one. Go use one of these other browsers instead.” But that doesn’t happen, because as developers we know what’s functionally identical. We’re not gonna push our users to go use another browser, because that’s annoying, and bad practice, and all… So that competitive pressure never turns up, which means that the Apple management – and I wanna be really clear here; this isn’t stuff which is within the Safari WebKit’s team control. It’s management deciding what budget they get. And presumably, in some cases, what they can invest in.

So it was at that point I started reaching out to people that worked at Apple, and I’m like “Look, we really need notifications. It’s been 10 years… What can we do to get this of the ground?” And we got no response. So we kept pushing, and eventually we created a bit post at WWDC, and we said “Look, Safari is years behind by Firefox and the Chromium browsers. We’re missing all these core features for web apps. Safari needs a much bigger budget. What can we do to make sure that happens?” And it was the most upvoted, viewed post at WWDC that year.

Adam Stacoviak: You also have a segment here on the homepage that says “The hard parts made easy.” As deep as you can, take us into those things. You’ve got automatic updates, native menus and notifications, you’ve got app crash reporting - these are things that typically if you built a native app on any platform, you’d have to figure these things out on your own pretty much. You’ve got debugging and profiling and then, the super hard one, Windows installer. Help us break down the hard parts made easy, what can you tell us about that?

Cory Doctorow: They bought everyone else’s ideas and operationalized them. Now, those mergers historically would have been prohibited. Right up until like the Reagan era, you weren’t allowed to buy your nascent competitors to create vertical monopolies. You would have had to compete with them, or source them as a supplier.

So imagine if the entire ad stack or the mobile stack was not underneath Google, right? Because Google couldn’t build an ad stack on its own. It had to buy it from someone else. So imagine if that was not underneath Google. And then you would have multiple competing vendors, all using that ad stack. And because they were all competing with each other, you would have multiple competing ad stacks as well.

So right now, Google, because it’s stitched up the ads stack, because it’s the demand side platform, the sell side platform, the marketplace, an advertiser and a publisher… Right? They represent the seller, the buyer, they are the marketplace where they meet, and they compete with both of them… This is like when you go for a divorce and your lawyer is also your partner’s lawyer, and they’re also the judge, and they’re also trying to match with both of you on Tinder… And then when the whole transaction is done, who gets the house? Oh, it turns out the lawyer got the house.

So Google and Facebook - which, again, are a non-competing duopoly - they take 51 cents out of every ad dollar. Historically, it was like 10% was the rake that intermediaries in ads used to get. So that’s money out of the pockets of publishers, right? So you have to have competition, and we don’t have that when companies buy their competitors. As Mark Zuckerberg once said in a breathtaking act of self-incrimination, “It is better to buy than to compete.”

And then the next thing that happens when companies get this concentrated is you lose the second way that companies are disciplined, which is regulation. So companies that break the law, if they’re small, generally get punished pretty bad. We saw that when the GDPR passed in Europe - all the little European ad tech startups, they all went out of business, because they either couldn’t comply with the rules, or when they failed to comply with the rules got slapped around by the commission and put out of business. Google and Facebook just ignored the rules, and 10 years now they’re still ignoring them. Maybe they’ll eventually get brought up, but they can just like delay, move venue, move forums… They keep insisting all evidence to the contrary that they’re Irish companies, because Ireland is a crime haven, where they don’t really have a working data commissioner… There is a guy who has that job, but most days he doesn’t get out of bed, and when he does, he doesn’t put on his pants. He just sits around in his underwear, eating breakfast cereal and watching cartoons. So Google and Facebook get away with breaking the law forever in Europe. And so they’re not disciplined by regulation. So the historic contours of privacy, labor, fair trading - they’re just not in there.

[00:36:16.13] Like, if you walked into a store, and you said “Give me part number XYZ for my dryer filter”, and they give you a different part, or they handed you five parts, none of which were that part, and then when you bought one of them and walked out of the store, they didn’t mention that none of them were that part, that would be fraud. That’s how Amazon works. Amazon makes $31 billion a year letting people pay to match your queries when they wouldn’t be the best match otherwise. That’s Amazon advertising. And so we throw away consumer protection, and we throw away labor protection, which is the other thing that would discipline companies. So if you think about Uber, Uber does this thing called algorithmic wage discrimination, where Uber drivers, they sort themselves into these two buckets; they call themselves either ants or pickers. So an ant is indiscriminate, they take any job, and a picker is very choosy. And if you’re a picker, that is to say if the wage offer algorithm notices that you’re highly selective, it offers you a higher wage per mile than you would get if you were an ant. But as your selectivity goes down, the algorithm starts pricing your labor lower and lower. If you back off from that algorithm, the wage starts to climb up again until you’re back in. This is like a fisherman playing a fish on a hook. Reeling them out, reeling them in, reeling them out, reeling them in. In traditional labor markets, this is illegal. But when your boss is an app, it’s not.

And so this is the second realm… They’re unconstrained by regulation. So they’re not constrained by competition, they’re not constrained by regulation. And then there’s the third bucket of constraint, which is self-help. So historically, when browsers were a cesspool of popup ads, 20 of which would spawn every time you open a new window, and they would be one pixel square, and they would run away from your cursor, and they’d autoplay music, or they’d go full-screen, and show porn… I mean, these just awful, awful popup ads; we didn’t ban them. It’s just that browser vendors started to ship popup blockers that were on by default. It started with Opera, then Mozilla, and then because it was a competitive market, everybody started doing it. And so that was the end of it, right? Why pay for a pop-up ad if the pop-up ads never show up? So if you’ve been thinking about that Uber wage discrimination deal, it might have reminded you of something. Like, maybe it reminds you of exponential back-off in TCP/IP, or maybe it reminds you of how bids and puts are done by liquidity provision stock trading bots, where they notice that the price is going up, so they reduce the frequency with which they’re bidding until the price comes back down… So there’s no reason to imagine that you couldn’t like make an app that was like a meta Uber app for drivers, that noticed when the price was going down for labor, and coordinated among multiple drivers to not have that happen, to reject jobs until the price goes back up. Or you know, what you were just talking about - your app is giving you notifications you don’t want to see. There’s no reason why you couldn’t pop-up-block that. The reason that you can’t do it is not technical, it’s that section 1201 of the Digital Millennium Copyright Act. The same law that bans you from removing DRM from an Audible audiobook also makes it a felony to reverse engineer those apps, because they’re encrypted.

And so one in four web users has installed an ad blocker. It’s the largest consumer boycott in human history. Zero app users have installed an ad blocker, because the first step to installing that ad blocker is to reverse-engineer the app, which can land you in jail for five years. And so this is the third area of constraint that firms are removed from. Their regulatory capture allows them to exercise unlimited discretion in how they reconfigure these infinitely flexible digital tools, and it allows them to confiscate the discretion that we would have to reconfigure those tools, so they serve us instead of their shareholders.

[00:40:08.10] Even when nothing unlawful takes place, if bypassing DRM or violating Terms of Service, or Apple, the subcomponents on iPhones have tiny microscopic Apple logos engraved on them, so that when they’re refurbed, or harvested in the Far East, and then sent back across the US border to be used in refurbs, they argue that that’s a trademark tarnishment and they have them stopped at the border, so you can’t get the parts, so you can’t fix the phone… So even the kinds of repairs you can do on an iPhone without bypassing DRM, and risking criminal prosecution, importing the parts to do it from old phones is potentially a trademark violation.

So you have this incredible regulatory leeway in preventing business customers and end users from altering the business logic of the service to serve them. You also have the same leeway in undertaking those same alterations on your own behalf as the OEM. And no one competes with you, so even if you could beat those other two forces, there’d be nowhere to go. And the reason nobody competes with you is because of those two things. And so it’s all sewn up.

So how do we get everything back? Well, the first thing we’ve got to do is just antitrust. And that’s amazing. There’s a lot going on right now with antitrust. Google’s in court, Amazon’s in court, Apple’s got an investigation pending in the EU; Salesforce, they say they’re gonna go after the Microsoft Activision merger again in the US… You name it, this is a historically unprecedented time. It hasn’t looked like this since I was in diapers. You have to go back before the Carter administration to find this degree of antitrust. And not just here in the EU, in the UK, in Australia, in Canada, and even in China, where the Chinese cyberspace regulation actually bans companies from reconfiguring their services to block interoperability.

So you have this incredible moment where people are recognizing that antitrust matters, and it’s happening across lots of different markets, and lots of different agencies. The Consumer Financial Protection Bureau just promulgated a proposed regulation that will force your bank to interoperably port your data out, either to comparison shopping sites that will say “Oh yeah, this is your mortgage, this is your APR on your credit card. This is how often you get hit by overdraft charges. This is how much you’ve got in your savings, and what interest rate you’re getting. You should be at this other bank.” And then, one click, port all your data to the other bank, including all your transaction history, your pays, and everything. That’s coming out of that Consumer Financial Protection Bureau, not out of the consumer protection agencies, not out of the DOJ, not out of the Federal Communications Commission… It’s like the bank regulators is getting in on the act. So this is great. So this is step one, antitrust.

Step two, constrain the way that the firms twiddle the knobs on the backend. Make them respect and obey privacy, labor and consumer protection laws. And the book gets into, as you said before, shovel-ready proposals for this. Ways that you can actually hold them to certain codes of conduct, where breaches are easy to detect… Because a lot of the codes of conduct that people want to impose on tech are really hard to prove when there’s a violation. You have to stop people from harassing people on your platform. It’s like “Alright, now we have to agree on what harassment is. Then we have to take some instance of a user doing something bad to some other user, and agree on whether that was harassment. Then we have to depose your engineers and figure out whether you did enough to stop the harassment.” And now it’s like five years later, and there’s been 50,000 more, 50 million more harassment complaints. What’s the point? It’s just not going to work. Even if you really think harassment is a problem, as I do, this is just not a workable remedy.

[00:44:01.02] But the book goes into some other remedies… In Mastodon, there’s a one-click facility to export an XML file - or maybe it was a JSON file - that has all of the people you follow, all the people who follow you, as well as your blocks and mutes and whatever. With one click, you can export that, and with one click you can export it somewhere else, and everything just shifts over. The same way – if you know the RSS spec at all, there’s like a directive in RSS where you can say “This feed has moved”, and just like the next time the RSS reader hits it, it just redirects somewhere else. So if you’re using Feedly, or whatever, and they start charging you money for something that used to be free, or they start spying on your users, or they, I don’t know, start [unintelligible 00:44:39.04] spotted owls as an internal corporate initiative to find a youth serum, or something… You can just like upload the redirect directive and then all the people who subscribe to your RSS go wherever you’ve gone to; some WordPress feed, stuff like that.

Alya Abbott: And so the way those App Store policies work is that a single app can only get push notifications from a single server. It’s kind of like an anti-spam security measure on their end. And so for your self-hosted server to send notifications to the Zulip mobile apps, what you do is basically bounce that traffic through our server, and so that’s a service that a lot of folks who are self-hosting choose to pay for as part of our plans.

And then the other piece is just support. So if you want any kind of support with running your Zulip server - so there is community-based support in our development chat. So folks do come by and get some help there. But if you need SLAs, or if you need something more than just asking a question on chat and seeing if folks are around to reply, then we do have support offerings as well. So those are kind of the types of plans for self-hosted organizations.

Jon Calhoun: I know for me a lot of it comes down to notifications, and things like that. I know by default the Twitter app gives you all sorts of notifications for “We think you’d be interested in this tweet”, and things like that. And I had to shut all of that off, because like you said, it was kind of baity, or it was things that I didn’t care about… And when I narrowed it down to just the people I want to follow in the Go community and things like that, it’s really helped.

I think the minute it gets to the point where I can’t just get on and in a couple of minutes see Go-related stuff, I think is when I have an issue. Because if I have to go through a bunch of weeds to get to the actual thing I want, I’m basically just going to stop using it; it’s not necessarily going to upset me, it’s just going to be like “This isn’t worth getting on anymore.”

Mat Ryer: Yeah, that really resonates. I don’t know how unpopular that will be, honestly… Yeah, I feel like that’s true. There are times when it’s nice, but you just can tell it’s algorithms feeding you stuff. I get notifications for things which would outrage me if I read it and took it literally. So I’m getting notifications for things to outrage me now through that X app. I don’t get it through the others, but… Yeah, and I know why that’s happening. It’s just literally to drive all that attention. But I think so.

[01:23:08.02] But I do feel like we need to change. We need to grow up now out of this, into something else, and I think that’s what will inevitably happen. I don’t know what that looks like yet, but… Yeah, I know what you mean. I think breaks, especially; take breaks from that. If you find yourself waking up and the first thing you’re doing is scrolling, and just having like –

Justin Searls: So if anyone’s followed me or my website at justin.searls.co for a long time, you’ll know that this is a bugaboo of mine. I’m very, very focused on notifications, and controlling them, and everything… But right after my own personal productivity and my own personal distractibility, my second biggest concern is other people. I feel like everyone being so distracted all the time is resulting in worse outcomes throughout the industry, in all kinds of things. Like, has productivity, has people’s thoughtfulness, has creativity actually increased with all these awesome technical tools? Or are we just so spread thin and just distracted to all hell… But on that front, on the ability to actually focus, one of the things that I think is going to have the biggest positive impact in people’s lives… If you’ve ever used focus modes in iOS and across Apple’s platforms to set up – like ,I’m in my personal mode, and in my case I’ve got maybe five contacts who can message me, and I’ll see those notifications, but I won’t see the rest of the Messages app. I’ve got like only these handful of apps for which notifications can punch through… It’s kind of like a customizable or a context-aware Do Not Disturb Mode. Setting that up took me several hours, and a lots of trial and error, and explaining it to my wife – I’m currently in like month nine of trying to explain what it means when it’s on… But the new natural – or excuse me, the Apple Intelligence mode will have an automatic focus, like a Siri focus that you can just turn on called Reduce Distractions. And I suspect for most people, most of the time, that’s going to accomplish the same thing. The Uber marketing notifications will never get through. The social media stuff probably won’t. But if somebody sends you a text and it seems important, it’ll say “Possibly important” and that’ll pop through. I think that’s brilliant. That’ll help a lot of people hopefully finally succeed. Because who the hell is gonna set up focus modes manually?

Gerhard Lazu: So the code will differ. And the Slack notifications - if this application instance picks up a job, it will do whatever it’s configured to do, which will be the wrong thing.

Mike McQuaid: …random apps, random first-party Apple apps that will like spam you with like offers, or like sign up to Apple TV, or whatever. At this point, when I get an iPhone or a Mac, I’d really probably like to just be able to say “No apps do any notifications by default.” Even like badges. I will literally just turn that on for the apps I care about, because it’s just so bad for everything now… And all social media are now – I guess we were talking earlier about like being distracted and sucked into things… For me, any social media with like an Explore tab or anything like that, I just cannot have it on my phone. I do not have the mental willpower to now just sit and like –

Andrew Beyer: [51:49] It’s kind of a hard question to answer, because I don’t honestly really feel that way, especially on the Mac. I think that in this day and – honestly, macOS, at the time OS X 10 was one of the coolest innovations the Mac platform has ever had, and there’s a reason why we’re still on that foundation. You take something like a Homebrew package manager, and a terminal… And I don’t really feel like I need Linux. I use Apple platforms because I love the ecosystem; they do work really well together, whether it be receiving notifications on your Apple Watch, and those kind of things… It sounds silly, and if you’re fortunate enough to be able to afford kind of a more expensive ecosystem - like, that is one of the downsides, it is a little bit more expensive. But working on Apple products and using Apple products I think is very open and inclusive.

I don’t know how many developers know this, but when you go to WWDC (the Worldwide Developer Conference for Apple), they give developers time with Apple engineers. Three months ago, when they announced iOS web extensions, I had three separate sessions with engineers directly working on those APIs, and we were able to say “Hey, here’s what we need, here’s the problems we’re encountering, here’s what we’re working on.” And you do have a lot of input there.

Also, Apple is very open source and open in the community as well. Swift is open source; WebKit - you can go on there and just file any issue you want. I don’t really feel that it’s a hostile environment for developers or users. I know we hear the horror stories where some high school student reported a bug to Apple via radar and didn’t hear anything back, and it was this huge security issue… There’s a lot of horror stories, but to be honest, you go on Google’s bug tracker and file an issue, there’s a good chance you won’t get – I have issues that have been open for six years over there. It’s just the nature of the game and it’s part of the prioritization… But I would say it’s a great platform to work on and build for.

Andrew Beyer: It’s kind of a hard question to answer, because I don’t honestly really feel that way, especially on the Mac. I think that in this day and – honestly, macOS, at the time OS X 10 was one of the coolest innovations the Mac platform has ever had, and there’s a reason why we’re still on that foundation. You take something like a Homebrew package manager, and a terminal… And I don’t really feel like I need Linux. I use Apple platforms because I love the ecosystem; they do work really well together, whether it be receiving notifications on your Apple Watch, and those kind of things… It sounds silly, and if you’re fortunate enough to be able to afford kind of a more expensive ecosystem - like, that is one of the downsides, it is a little bit more expensive. But working on Apple products and using Apple products I think is very open and inclusive.

I don’t know how many developers know this, but when you go to WWDC (the Worldwide Developer Conference for Apple), they give developers time with Apple engineers. Three months ago, when they announced iOS web extensions, I had three separate sessions with engineers directly working on those APIs, and we were able to say “Hey, here’s what we need, here’s the problems we’re encountering, here’s what we’re working on.” And you do have a lot of input there.

[52:23] Also, Apple is very open source and open in the community as well. Swift is open source; WebKit - you can go on there and just file any issue you want. I don’t really feel that it’s a hostile environment for developers or users. I know we hear the horror stories where some high school student reported a bug to Apple via radar and didn’t hear anything back, and it was this huge security issue… There’s a lot of horror stories, but to be honest, you go on Google’s bug tracker and file an issue, there’s a good chance you won’t get – I have issues that have been open for six years over there. It’s just the nature of the game and it’s part of the prioritization… But I would say it’s a great platform to work on and build for.

Julia Grace: See, I think you’re highlighting on a really interesting question, which I see the parallels, and sometimes no internet is better than slow internet… Where you want a service to be really quick, you want to ensure that your notifications show up instantaneously… Imagine if you get a DM from your boss and you wanna know, you wanna be able to respond, if that’s your relationship with your boss… So we think about this a lot, and we think about it especially with respect to we’ve grown so fast, and now that over half the messages in Slack are sent outside of the U.S., we have to have an infrastructure that allows Slack to boot instantly everywhere around the world, meaning Houston, Omaha - since Adam, Jerod, I know you’re out there - but also in Japan, in Asia-Pacific.

So we run Slack in the cloud, and we’ve been cloud from day one, and as part of the infrastructure organization, we’ve had to build a lot of tooling to understand what our performance is around the world… And also, you were talking about notifications, and especially on mobile, we use the infrastructure - I believe it’s APNs, which is the infrastructure provided by Apple, to send notifications on iOS; there’s also an analogous system on Android. So one of the most difficult things is providing a service that is used 24 hours/day, 7 days/week, around the globe, that people need to do their jobs, that has to almost be more reliable than the internet backbone.

What I mean by that is there are parts of the world where the internet backbone is less reliable, especially in Asia-Pacific. Let’s say that you get a DM and it doesn’t come in fast enough. It seems delayed; Slack seems slow. You don’t care that there’s DNS issues that are happening in your part of the world; you need Slack to be fast, and that’s what you expect. So one of the awesome challenges here is figuring out how to provide that level of service when we don’t control the racked machines; we don’t have our own data centers, so how do we do that when we don’t have that low, low, low level of control? And the way that we do it and the way that we’re figuring it out - because again, the scale is just tremendous - is by building software. And that makes me incredibly excited. We are figuring out how to work with different vendors to build really resilient fault-tolerance software that can provide you that level of experience, when fundamentally the underlying infrastructure - the cloud providers that we run on, and then the internet backbone lines that they run on - do not provide the level of uptime that we need.