John-Daniel Trask: Yeah, so we actually didn’t start off there. We started off building some other software development tools… And you know, this was where things actually got murky, because my vision for my future was to own a software business. I didn’t actually know what it was gonna do. In a weird way, my goal, my purpose to be the owner of a software company had been with me for about 12-13 years at this point… And once I got there, it was kind of like “Hey, hang on a minute… Now what do we do?” All this time thinking about this, and I kind of got to this point…

So we actually muddled around for a few years, building different things… They were liked products; we could wipe our own face. The business was making money, which was good. But we also did some – we bootstrapped the business, so we did consulting work… We actually had Microsoft New Zealand come to the party on day one of founding our company and give us a quarter of a million dollar contract. And that was really our seed capital.

We built some software for Microsoft that demonstrated how teams should build modern web apps, and they used that as demoware around the world. Now, that actually came out of the relationship that Jeremy had built with Microsoft New Zealand. And this goes back to your point of who you associate with, where are you investing your time. He was putting a lot of his time into supporting the local software development community, running user groups and all of that… So he was well-known and well-liked, and generally appreciated in the local software community. Surprise… It worked out for us.

And to use that famous Steve Jobs quote, “You can’t connect the dots looking forward. You can only connect them looking back.” Jeremy wasn’t supporting the community because he wanted a deal out of it; he was doing it because that’s what he loved to do, and then this positive came. So he had actually put in not expecting anything out, and then got something out of it. And that was hugely valuable to us.

[27:49] Over those few years we also partnered up and built other companies. So we were fairly well-known as being a pretty good software development team, high-performance software, good outcomes… But we took equity in those companies. So we built New Zealand’s largest philanthropic website as the technical partner, where we owned a part of that; we built a business valuation SaaS company, we build an email mining software company… We did a few things like that on the side. That was all good.

In 2012 we decided we were gonna build the error tracking software, and that we were gonna call it Raygun. Now, our company at this point had actually been called Minescape… And I’ll tell you want - nerds naming companies, man… How did we come up with the name? I mentioned we had three founders at the beginning… And we basically all just came up with these names, put them in an Excel file, ordered them, and them we all scored them one to ten, and then we added those three rows together and sorted the sum, and then we just went with the top one, which was Minescape.

And I remember there was this one right down the list, Raygun… And I just remember thinking – well, it was actually Railgun, but anyway… It was from JB, and I just remember thinking “That’s the stupidest name for a company I’ve ever heard of.” [laughter]

Kelsey Hightower: Yeah, I think we always have this desire as humans to aggregate things out of the sake of simplicity. If we put everything in the same configuration management tool, then it will be simple. But typically, when you look at that configuration management tool, there’s all kind of if statements and weird logic and extensions that only the person who made that understands, and everyone just says “Hey, don’t touch it. Just use it as it is”, and now it’s very brittle.

I think the thing that does make things simple are things like “Here is the frontend. The frontend has these components, and it retrieves this data from these imports. It doesn’t retrieve this data from functions, it doesn’t retrieve its data from Kubernetes. It retrieves this data from end points.”

[32:16] So now if you’re a frontend developer, you can have endpoints be anything for your testing purposes. I could just run endpoints on my laptop and serve mock data, and the site will behave as designed. So now I say my deployment target for my frontend could by my local browser, or it could be a CDN that will then serve it to billions of web browsers. That’s simple to understand. So now you say “What tool will I use to complete that?” Well, it could be as simple as my IDE, where I hit Save, and it just goes to staging automatically, because I know what the contract will be. When it gets there, it’s going to use a certain set of endpoints, and those endpoints are required to have data.

So now let’s move to the endpoints. Any one other team can create endpoints. You make a promise, endpoint.hightowerlabs.com is supposed to serve this kind of data, and this structure. And if you do that, then my frontend app, my mobile app, my watch, my smart washing machine - all of them will know what to do with that data. And even though that sounds more complex, it’s easier for everyone else to understand that there’s going to be data returned from this endpoint. None of them need to know about Kubernetes, they just need to know what my contract is. You need these credentials to get this data, and I promise to keep it up 9.99% of the time. So now that we have these contracts, now I can go focus on my area and just assume you have your area.

So now let’s talk about deploy targets. Now, that API - we know we need DNS, and I could decide to point the DNS at Cloud Run, I could point the DNS to Kubernetes, independent. So I think these logical boundaries, these very clear contracts allow people to move very fast, because now we’ve reduced the amount of coordination that we have to do in order to figure out what to do next. So as a developer, without talking to anyone on the endpoint team, I can just call Curl and see what data comes back, and I know what to do going forward. So now if I wanna use different tooling for the Kubernetes component - and think about this… You don’t necessarily need to provision the cluster yourself. You could decide “I’m just gonna use GKE Autopilot, which will automatically scale the nodes across the regions based on what my application needs.” So my Terraform provisioning is “Enable Autopilot.” That’s it. No flags, no cluster side, no node pools.

And then maybe another tool takes off to manage things like deployments, deploying to that particular target. To me, that is easy and understand and explain to the relevant parties. And you’re right, if one person had to think about all of these things, we’d think they would have to actually learn a lot, but I don’t think that is the case. When I context-switch, when I go to work, I work on a Chromebook. When I’m doing personal projects, I work on a Mac. If I wanna play a certain video game, I do that on Windows.

Humans have no problem switching context when the rules are clear, once I make the switch. It’s only when we start to muddy the waters by mixing too many concerns. If the frontend team is using Terraform in a way that doesn’t actually make sense, then they get confused and now they’re spending so much time battling Terraform, instead of just saying “Save in the IDE, run some tests, if they pass, deploy the new set of static files to the CDN”, and be finished.

Max Ogden: Yeah. The human side of code that I learned through that process was… Previously, I think you nailed it in the question, actually. Previously, I was an outsider. I was a volunteer and I didn’t feel like I could actually influence the things the people that were working for the government could do, technically. I just assumed their process was set in stone, and they weren’t interested in me, as some random person who wasn’t official and wasn’t paid to help them. I just did what I could; I took their data, and tried to make useful on the outside.

I think the stroke of genius in the model of Code for America… I mean, they copied Teach for America, it’s very overtly modeled on that. I think the Teach for America model is you embed people. The Code for America model is about embedding. One of the really cool things about the Code for America program was you show up, and the first week is about understanding government culture, and how you can be an agent of change to show them alternative ways of doing things with technology. But an emphasis that it’s not technical, it’s a human problem. It’s a lot of social problems, and it’s a lot of incentive problems. We actually had a negotiation workshop, which was really, really useful. I still use the principles that I learned in that, every day. I had thought negotiation was about — if somebody is trying to detonate a bomb, you have to talk them down off of a ledge. It turns out that negotiation is if you’re talking to anyone, in your day-to-day life, and you’re trying to be nice to them, that’s what negotiation is about. It’s about having respect for people’s point of views, coming to a positive outcome.

[08:02] So the fact that Code for America didn’t have us do a bunch of technical things on our first week, but instead they had us talk about being change agents, being effective negotiators, I think speaks a lot to us about how they knew that it was essentially about embedding us inside of government, and having us inspire people with new ideas, and have an influx of crazy ideas that came out of it.

Going into it, I thought it was a technical thing. I thought I was going to be like, “Okay, I’ll go make a bunch of cool APIs, or build a bunch of cool apps.” By the end of it I realized that people inside of governments aren’t exposed to ideas like open source as much, because the hiring and procurement systems are essentially broken. They don’t have any way to compete with talent, for people that go and work at Google.

The Code for America model is you literally get people to quit their jobs at Google for a year, or go on a sabbatical, and then you get them to become government employees. I didn’t come from Google, I came from this smaller company, but the general idea is practicing people from the tech industry get to go and do a year of public service. What was cool about it was I actually became a government employee; I went through the ethics training at the City of Boston, I had a “@cityofboston.gov” email address, I mean, it was the whole shebang. I was an official employee in the Mayor’s office.

That was really empowering for me, because now I was on the other side of the wall, so to speak. When I was in Portland, I was just this random person that was volunteering; I didn’t feel like I had any power to actually change anything. Now suddenly I was a City of Boston employee, and I felt my opinions were valid on things, and I could set up meetings with CIOs, and talk to them about, “Hey, why are you procuring this horrible software, why aren’t you procuring open source software, for example?” That was super cool, I feel like it was a hack.

The Code for America hacked two things at once; they gave me a lot of confidence that my opinions did matter, because it made me feel like I was the expert coming in, and trying to help people understand they don’t have to buy horrible software, and hire people with horrible credentials; they can actually do things in a more progressive and modern, open source way. To them, it was exciting; they had somebody coming in and had a lot of excitement and enthusiasm. I definitely had a lot of people telling me they were surprised that they didn’t make me shave to work at Boston City Hall; I had a giant beard walking down Boston City Hall. It was definitely a bit of a culture change thing on purpose.

The point of it was you go into a city for a year, and you try to make some cool things. By the end of the year, you don’t just leave and the things go away. The idea is that by the end of it, you’ve given the city a different lens to view the process for developing software.

Actually, the hardest thing about it was that we had to come up with a way of contracting for support for the open source apps that we did, because they were deployed on Heroku and they didn’t know how to maintain Heroku services. It turned out that the biggest outcome of the entire thing was we had to draft a new procurement policy for the City of Boston that let them support open source software, have a support contract with an open source vendor. That was a totally new, groundbreaking thing for them because usually the support contracts are built into these huge multimillion-dollar contracts, but the idea that they could have a $5,000-open source support contract just so that if the app went down, they had somebody that they could call to help them. It was those small wins that were the long-lasting effects, whereas going into it, I didn’t know procurement was going to be the focus of all our efforts.

Sam Soffes: I started working on this thing called SSToolKit a long time ago, in like ’08 I think. It was just some things I wrote in Bible that I could reuse in my next app, because I was starting to do contract work. I put it on GitHub, because where else would I put it? It was free and open source… Maybe it was ‘09, I think that’s when I sent it to GitHub. Whatever. And I’ve just kind of always added stuff to it; I didn’t really expect anyone to use it or even look at it.

Then eventually a lot of people started looking at it, and replying to me and fixing things or asking me to fix things for them or whatever. Now it seems like whenever I go to meetups someone will come up to me that I don’t know and say like “Hey, thanks so much for that. I really like SSToolKit.” It’s like, “Oh, you’re welcome. I’m really happy to hear that you use something I’ve spent hundreds and hundreds of hours on. That’s pretty cool.”

That was kind of like the first thing I open sourced, and since then it’s like “Well, if there’s anything I can do that’s reasonable that someone else could use, why not give it to them and save them the trouble? And if they wanna help me make it better, then everyone wins.”

Tim Coulter: [35:49] Yeah, so right now the library that I’ve described so far is what we call Truffle Contract, and this is actually a library that’s existed for quite a while. This one turns everything into a promise, so you can use async/await with Promises, but it doesn’t use callbacks because what generally happens when writing a front-end for a smart contract application, or what you generally do is you say “Make this transaction, then do this transaction” after a specific request happened. There’s also other pieces of that, so not all interactions with the blockchain are right, not all of them are transactions. Sometimes you can call a function that will get executed, but it’s actually just there for reading data, so it doesn’t get recorded on the blockchain, it doesn’t cost you any Ether to do that… So a lot of the times it becomes like “Perform this transaction, read some data, perform another transaction…” and so this library provides it to you as sort of a promise chain - “I’m doing this, then I’m doing this, then I’m doing this…”, so you can have easy control flow.

Something that we released today actually is a library called Drizzle, which is the next evolution of this idea. The JavaScript world and the front-end world has been moving away from effectively home-grown transactions, direct interaction with the server, and instead you’re moving to a React and a Redux architecture. Well, wouldn’t it be great if you could take those Truffle artifacts that Truffle creates for you, throw them into a library and you’ve all of a sudden got a Redux store that is tailored to your smart contracts? That’s what Drizzle provides - if you like the React and Redux world, there’s almost no work involved to fit your contracts into your front-end application.

Jerod Santo: They said, “We gave you a two-year contract only, Apple.”

James Moore: So first up, we will need to communicate to the commission exactly what we’re not happy about with Apple’s proposed contract to allow other browser engines on. At the moment, there are a number of reasonable conditions, like “You have to regularly patch your browser, you have to follow certain security standards”, and all the browser vendors will be like “Yeah, thumbs up. We already do that.” But then there’s a series of unreasonable conditions after that. There are things like – the scope of contract is very broad. It says “You agree to follow all rules in Apple materials”, which basically, who knows what that means? Is that anything on apple.com/star?

Tammer Saleh: It’s pretty interesting stuff… If only it was adopted, right? But again, still, build packs have the same problem we just talked about - they’re great for the 80%, bad for the 20%. But this is exactly where I think something like Nix has a lot of potential. Like, Nix could have made Docker irrelevant, and could have abstracted away the things that we think of in a production Docker file. You could have just listed your dependencies in a manifest and your application would run within there. Maybe there would be a stronger contract about how your application runs, because build packs were that way; build packs were 12-factor, right? Could have had that same 12-factor contract, but allowing you to very easily specify all the dependencies that you might need. It could have been really interesting. But, I mean, in reality, nobody is doing anything other than Docker. Right? There’s no significant competing technology than crafting a Docker file by hand.

Matthew Boyle: Truthfully, I just look for any excuse to talk about gRPC, because I love it. For internal communication, I think gRPC is probably one of the greatest things that has been invented, if you will. I think from a DDD perspective, I’ve mentioned the idea of having sort of published language, which is – as my team looking out towards the world, this is what our contract looks like, this is what our… In the same way in a monolith, or in sort of any application, you can define a contract, right? Like, when you talk to me, this is what it’s gonna look like. This is the function signature I’m going to satisfy. We can do the same thing to other teams, too. And usually, we’ll define an open API spec, or maybe we’ll write in a wiki just what everything means, or maybe we’ll write a protobuf. And I guess my thoughts on gRPC is protobufs are very, very readable. They kind of give you forwards and backwards compatibility out of the box. And one of the really powerful things about gRPC, especially if you are working in a bigger company or a bigger team, is that it has support for generating code for lots and lots of different languages.

So as much as I wish the whole world would just use Go, sometimes for some reason people pick other languages… And so being able to support them and not be opinionated about how they build their systems is really, really powerful. And that’s something that starts with a protobuf, which is more about – most of our protobufs in Cloudflare are sort of 10-15 lines long, and they’ve got very clear headings about what this is, what the type of it is, it gives you that type safety… And from that I can generate code for so many languages.

So that sort of – domain-driven design has this concept of, as I mentioned, open host protocol, or open host service; basically, opening a subset of our system to the world. I think gRPC is a really great way of satisfying that particular piece. And if we’re building distributed systems, or building anything truthfully, we’re going to have to find a way to share it with people somehow, and protobuf and gRPC in general is a really great way to do that, I think.

Rosemary Wang: We are… I think it depends on the tool, though. Sort of a practice that you could approach is just managing it all as code, in which you express the configurations you need… Because the definition of infrastructure is actually much larger than it used to be very traditionally, right? We thought about it as a data center plugging in network switches, and the like. But in reality, infrastructure can include managed services now, as you pointed out. It can include fly.io, it includes any number of third party systems that you don’t necessarily directly manage, like HCP, but someone else might manage for you and you’re still using.

And at the end of the day, even if you’re using DigitalOcean, HCP, all of these managed services, you need to express that configuration somewhere. And most of these platforms have APIs now for you to configure whatever you need to use out of that platform. And fortunately, they’ve integrated with some kind of infrastructure as code tool somewhere. So we can pick TerraForm, just because we’re talking a lot about HashiCorp today, but a lot of this configuration can be managed as code. If you think about even Vault - what are the things that you need to set up the secrets for an application, too - you can manage that as code. Be declarative, be specific, be prescriptive, and then your application – you’re not breaking that contract that the software itself might express to depend on some of these infrastructure components and configurations. That way, you have it end-to-end expressed in one place, encoded in a single manner, and you can manage it without having to worry about breaking one thing because some schema change, and now you have to fix a downstream dependency, or something.

Tom Occhino: Yeah, that’s right. The views, in our applications, ended up being massive. And then the models that needed to power those views were massive. And you can break them up. You can say “Okay, now I have multiple models that feed up into this thing, and now I have multiple views…” But everybody was thinking about things from a sort of like, I don’t know, top-down, rather than sort of the component as the atom, inside-out sort of mentality. And that was the big shift, was “We’re going to define a seam, and I can do anything I want inside of this seam, but the contract with the rest of the application is intact.” And we could have done a better job of telling that story, but it worked out okay in the end.

Tom Occhino: Absolutely. I do not think that this should be a trade-off at all. And I’ll give one example of the opposite case than what you described. In order to enable a truly exceptional user experience, the developer experience was so bad that nobody wanted to contribute to the thing, and so it eventually just died. Facebook built an app called Paper for iOS, and it was around the time that the iPhone 4S was released. It was such an incredible app, ahead of its time. Every gesture was smooth, everything was 60 frames per second animation. It was built on a technology called Async Display Kit, but it was so manual. Everything was very hard to work with as a developer, and it was very easy to get things wrong. One sort of line out of place and you were segfaulting the app and hard-crashing, because it was doing direct memory access, direct access to underlying canvas painted views.

And so unequivocally, you could use this app and you’d be like “This is the best feeling app I’ve ever used. It’s an amazing user experience.” But because the developer experience was so – it was full of what we call footguns, or it didn’t enable developers to fall into the pit of success, developers didn’t want to work on it. And so that great user experience - there were two or three people who had it in their head, that could make changes, but ultimately the app kind of just didn’t scale, and we kind of shut it down eventually. We took some of the lessons and some of the tech and incorporated it into the main Facebook app, but Paper didn’t live on.

And then on the other hand I think every day we’re confronted with software that we use, that is clearly optimized for sort of time to market, and nobody really put any care or craft into it, and it’s just like “Look, I can tell you just kind of like put this out there, and it solves a purpose, it solves a need, but it’s not where I want to spend a bunch of my time. It’s not a delightful, exceptional user experience.”
[00:32:07.10] So my strong belief - and actually I think at one of the React Conf keynotes I gave a whole spiel about this… The only reason to invest in a great developer experience is in service of creating an exceptional user experience, and then being able to evolve your app and keep it relevant over time, while preserving and maintaining and enhancing the user experience.

I think that these two things go hand in hand, and you should be focused on - as a framework author, as a platform author, you should be focused on great developer APIs that by default create great user experiences. We’ve seen quite a bit of this in Swift and Swift UI. I think Apple’s doing a pretty good job here of making it so that “Hey, if you just write your code this way, look at the great user experience you get.” And you’re like “Oh my gosh, this code is easier to write.” And that’s what I love about React as well. Because we have this component as the isolation boundary, component as the sort of logical “This is my container”, I can optimize the insides of that component and the user experience associated with that component, and maintain the same contract with the rest of the application, and then somebody else can work on this piece and that piece. And even though we’re working on different parts, because we’ve maintained these contracts, we can have a really good-feeling application. I can continue to refine my one component and make it better and better and better, without degrading any of the rest of the app. So yeah, I think they go hand in hand.

Matias Pan: Perfect. So yeah, with observability already taken care of due to a very complete product, the other thing that we have is packaging. And like I said before, when you run lemmy service new, we create a Docker file, and all applications have to implement what we call a contract. It could very well be scripts, but since everybody’s using Java and Gradle, or any JVM-based language and Gradle, you have three commands that you have to implement as a developer, and that’s build, integration tests, and end-to-end tests, I think it is. So it’s those three tests that we usually do historically at the company.

[58:12] There’s a kind of weird distinction between integration tests and end-to-end tests that I’m not yet convinced… But yeah, we have to support the monolith and be sure to be backwards-compatible, so you have it there. And we use those three scripts to automate all the pipelines that are a part of that packaging. So packaging is not just about the actual things that you package, but it’s also how you package it. And we simply generate a GitHub workflow that uses a Docker file that has those Gradle commands, those Gradle build commands that we know will be, because developers will implement it… We use BuildKit to actually like build the image itself… I mean, we use Docker, but with a flag turned on, like “docker_buildkit=1”. So we don’t use BuildKit in the proper sense, with caching and all the nice things that it has.

So yeah, that’s how packaging gets taken care of - Docker for everything, auto-generated, if you want to change it, you can change it… You have it there, it’s yours to own… And that’s it. Yeah, that’s the packaging part.

Marty Schoch: [03:59] …in October, 2018 I left Couchbase. Some of your listeners may know I’ve been working there, and working on Bleve, a search library. You know, the time had come, and one of the exciting things about working on an open source library is that – you know, the project was started by Couchbase, but it gets adopted by these other companies… So what I decided to pursue was an opportunity to work with some of the other companies that were out there using Bleve in a different way. It’s always eye-opening when you get that chance to see your same codebase, but being used for some whole different application. So I was very fortunate…

I did some contract work with two different companies, and that ultimately led me to where I am now - I’ve just actually started a company called Bluge Labs. What we’re trying to do is get companies that are using Bleve on board to support Bleve, in sort of a new way. You could probably have a whole separate podcast on the economics of open source… So here we are, trying to support Bleve and open source search in Go in a slightly different way… And that’s just getting off the ground now, so you guys will all have to stay tuned for more information on that later, but… That’s been keeping me busy - figuring out how to take Bleve to the next level in terms of successful open source.

Gavin Wood: [07:53] Okay, so Ethereum is programmable money. Bitcoin is money, more or less, on the internet; “magic internet money”, as it’s sometimes called. Ethereum is programmable money. It’s doing the same thing, it’s still magic internet money, but this money can have particular software attached to it, particular conditions or particular logic. It can even storage attached to it, so the money can remember some sort of history, can remember what happened to itself, and it can do different things depending on that memory. So that’s really what we’re talking about with Ethereum.

Now, it could also be described as a decentralized application platform, it could be described as a smart contract platform, but at the end of the day, probably the thing that makes most sense to people is the notion that it’s money that can have program software installed actually in it.

Safia Abdalla: 75% on Accessibility. Some things didn’t have labels and color contrast. Progressive Web App is 55%… You just got grounded.

Nathan Sobo: I know. We had another silly thing were we didn’t staple some apple certificate that they returned to us, or something during our certificate signing process, which I probably set up and did wrong in the first year of working on Zed… And I think it was causing the app to need to like contact Apple servers or something after a system restart on startup. And it’s like, that makes our startup time look awful. But that’s not anything to do with – I don’t know, it’s just this silly little, tiny thing that can just destroy the experience. Anyway… But we fixed that too, so now [unintelligible 00:13:22.23] although we’ve still gotten some reports… So yeah, it’s tough. And it’s tough when it’s like – yeah, I mean, I’m used to it luckily from the Atom days. It’s running out there in the wild, a little far from your eyes and control.

Tryggvi Gylfason: I have a question for the chat here, and for anyone listening… This is an area for me of accessibility where I’m not sure where the problem should be solved. Is it in the app? Should each app adjust light and dark mode, and contrast, and inverted, and all that? Because a lot of the operating systems also have those accessibility features on phones, where you can invert the colors and you can do things… So I’m not sure how far to take this type of work when you’re building a web application or not.

Jerod Santo: Now, if you don’t see any chapters in your podcast app of choice, it’s possible that app doesn’t support the feature. In that case, maybe contact the developers and ask them to add it. Or, try out some other podcast apps that do support chapters. It’s a great feature, worth switching.

If your app does support chatpers, but you’re still not seeing them. There is a chance that we have a bug in our ID3v2 frame generation code. And since I didn’t write that code, I’m emotionally OK with that being a real possibility. In that case, please let us know in the comments.

OK, enough Changelog-gazing for now, let’s get into the news.

Chris Krycho: I think that actually gets at something really important that we haven’t touched on so far, which is that a concern for backward compatibility, which can make upgrading easier, in this context the fact that you can still run 16-bit DOS applications on Windows 11 - which is amazing in a lot of ways - also makes it much harder to move the platform forward in other ways.

Apple, by contrast - like, you want to run an application from the 16-bit DOS era? Ha-ha-ha! Maybe if you’re running it in an emulator written in JavaScript you can do that. Cool. But on the operating system, absolutely not. On the other hand, that’s part of what makes it really easier - I’m not going to say easy, because it’s an enormously hard task… But easier for Apple to do something like ship an architecture transition from x86 to ARM, and pull off the whole thing in a matter of two and a half years, three years, because they’re willing to just cut their losses for backwards compatibility sometimes. And we see the same thing for a higher level of the stack, that a framework that is willing to make breaking changes on the regular can move forward faster, but that’s more cost for the people upgrading.

By the same token, somebody who’s really committed to backward compatibility can bring along all of the people in their ecosystem very well, but it can often be much harder for them to get the benefits of trying new things. And I think here actually the example of React and Ember in the JS ecosystem is a really good example of this, especially – it’s less true now; React has gotten way more thoughtful about breaking changes. But in that kind of mid 2010s to 2018, 2019 time, React would ship breaking changes much more than Ember would. And React got to try a lot more things than Ember did.

On the other hand, I’ve talked to a lot of senior engineers at very large companies who got stuck on React 16 and never moved on. And that’s less true for Ember, but Ember got to try a lot fewer things along the way. There’s just a fundamental tension there, that then gets manifested in your versioning. And the versioning is often a symptom, rather than the cause in that case. It’s telling you a true story, but the story it’s telling you might be telling you something else about the dynamics. The fact that Windows can still run DOS 16-bit applications has other consequences for the kind of incoherence of the Windows ecosystem. But that’s a really big benefit if you’re an enterprise IT administrator who wants to not spend a lot of cost on upgrading applications that just work for you. Trade-offs, all the way down.

Preston Van Loon: I think the amazing part about this technology is that while there are no rules, you really get your opportunity to define those rules. So you’re able to develop what are called smart contracts, or these decentralized applications which - everyone can read them and understand what’s going on, what are the constraints of this protocol that you’ve launched on Ethereum. And if you’re talking about true decentralization, there are these concepts where you can have what’s called a DAO (decentralized autonomous organization), which is a group of anonymous people that have voting rights in the protocol and are able to make the upgrades and change the rules.

With Ethereum, or really any blockchain, it’s like a write-once-and-read-many-time kind of style database, and you’re not able to remove or delete or change anything unless that’s part of the code that you’ve written. And as a participant or user in your protocol, I can see that and I can base my risk assessment “Do I wanna be part of this application?” I can read the code and find out for myself.

Jerod Santo: I definitely like the conversation, because I knew you previously. I hadn’t followed you on Twitter, but I knew you as the Homebrew guy, and when I saw that tweet I thought - and I’ve never been interviewed by Google, so I don’t know what that process is like; I know they have brain teasers and whiteboarding and stuff like that, but I hadn’t thought about the interview process at large companies because I’ve never been part of a large company. And the conversation around it wasn’t just “Should Max have gotten hired/Should he not have gotten hired? Is Google evil? Is Max arrogant?” Of course, those things are always said, they’re on any popular conversation (unfortunately), but it was “Is interviewing broken? How can we do this better?”, those kinds of things.

I felt like the end of it, there were a lot of think pieces that were written. I think there were a lot of good things that came out of that, and so I would hope you’d do it again too, just so we can continue having these conversations. So that leads us to where you actually did land, which was Apple. Can you compare and contrast the hiring process between the two? Would that be profitable, or…?

Kevin Ball: It is interesting to look at the way big corporations do open source. Facebook seems to primarily focus on supporting their own open source projects, and they’ve released a ton to the community, which is phenomenal. But if you contrast to, for example, Google or Apple, both of those companies do release (and sometimes push) their own open source projects, but they also put a ton of time and energy in staffing, supporting other ecosystem projects.

Michael Grinich: It’s a really tough spot to be for companies. You work so hard to get product-market fit, through focusing on the product features and experiences itself - design, the onboarding, the sharing abilities, things like that… Maybe collaboration features, real-timeness… So whatever is kind of the unique secret sauce that brings people in. And then at some point, you have a big customer saying, “Hey, we want to use this everywhere in our organization. We just need you to build this thing that every single other enterprise app we have has this feature.” And it could be as simple as like single sign-on. That’s the one that a lot of people hear about first, right? Like, “Sign in through Okta. We can’t sign in through passwords to this thing, because it’s not secure.” It’s pretty tough in that moment to figure out where to put your focus. Because most companies - now, I’ve been there myself; if you’re the entrepreneur or the product leader, you’re just now at the moment where you can start doing the really interesting stuff. You can start really building the platform you want, or building the most powerful version of the product, and suddenly now you have to get pulled away to do this enterprise garbage that your team doesn’t want to do. You haven’t hired designers that were super-excited to build the audit log in your product… It’s a distraction, in a way, but it also is really, really necessary for company growth.

In the case of Dropbox, they eventually did it. Dropbox Enterprise today is an awesome product; it’s being used by thousands and thousands of companies. But in the early days, in a moment, and probably when Apple decided to use Box - because these contracts stick around on for a while…

Jerod Santo: No, I wasn’t timing you. Okay, I think that’s good. So my background is somewhat similar, in the sense that I began one place, and kind of moved to another place… But always was trying to be a generalist. And as I went freelance, or solo, or contract, I had lots of situations where people would come to me and say “I have this problem.” And the problem was usually some sort of technology. Like “I have a PHP app, and I need some PHP help”, right? Or “I have a Ruby on Rails app, and I need some Ruby on Rails help. Do you know any Rails devs?” I actually just got a LinkedIn message two days ago… You know, I’m in that latent space, and people still think that maybe I’m available for this kind of work, even though I haven’t taken a new contract since probably like 2017-2018.

And he says specifically like “Hey, I’ve got need for a React dev. Do you know any good React devs around? Because I’ve got a 30 to 60-hour project, looking for some help.” And these are people who - he just happens to know that I was in the business of, you know, contract for hire, and he’s looking for a pointer to somebody, but he doesn’t know how to categorize that person… Like, “I’m looking for a good developer, because I have a software problem”, that’s just not the way people think. They think in “What’s my codebase?” I guess that’s how they think, depending on who they are.

And so yeah, if you’re not a React dev, basically you’re disqualified from that. You know, I can go back to him and say “Hey, I know a guy name Adolfo. He’s really good at generalized things.” And if he trusts me a lot, he might be like “Sure, I’ll give him a shot.” But if he doesn’t trust me so much, he’s like “Does he know React?” you know… And so there’s your specialization.

So what you have found, and what I think is generally applicable, is when you’re putting your sign up, you kind of have to have some sort of specialized skill set, because people are going to be looking for that skill set, whether or not that’s in their best interest. That’s just what they’re going to do. Is that fair, do you think?

Justin Garrison: Yeah, this is my first time in a professional role doing Dev Rel, doing content creation and doing this style of work. I’ve always just been an engineer, and I love this job. I felt I was good at it. I enjoyed engaging with people and learning something deep technically, and then telling people how it works. And that cycle of back and forth, and then being part of a product that I know a lot of people used was also just wonderful. That was something that I joined Amazon for, and I really a lot of things at Amazon with how they were working on things, and that involvement to get some outside voices or non-engineer voices necessarily. I’m not writing the production code. I’m helping guide the product and then testing it and saying “Hey, this is how customers should or shouldn’t use this thing.”

And I’ve had experience across the board with – I helped launch App Runner, I did a lot of work with ECS, I had some talks I’ve done about Lambda… This is really broad, because no one uses any of these services in a vacuum. And I liked that breadth that I was able to just “Hey, I can pick out any of these things.”

This year - well, last year. It’s 2024. In 2023 Amazon really pushed for return to office… And I was hired as a remote employee before the pandemic. I started during the pandemic, but I had my contracts and negotiations and everything before, as a fully remote employee. I was remote at Disney before, so I was used to that. I needed a remote job. And in 2023 they were really saying “Hey, we’re gonna start returning to office.” I was like “I’m a Virtual Employee, I don’t have an office.” The closest office to me is maybe an hour away, maybe two hours away with LA traffic. I’ve gone there a few times to meet up with people and to have some meetings, but it’s not a regular occurrence. And I was always told that my role would stay remote, and I didn’t have to worry about it.

And then things started changing, more and more noise was happening during the summer; it was actually more people are gonna start coming back to an office part time. And I kept being told over and over again by management and my leaders “No, no, no, you’re fine. You’re a remote employee.” My entire team is not in a location. The DevRel team is all across the United States. And when I joined, we were all over the world. I was like “We don’t have a timezone, let alone a location.”

And so as things started progressing, it was becoming more and more clear that this was going to affect me at some point. And we filled out forms to get a remote exception, which was for a one-year thing; I had to renew that every year. And I got my approval for remote exception just days before I was told that our team was actually going to be disbanded, and our team would not exist anymore under the Kubernetes org, as part of the Kubernetes product team. And they wanted to get rid of this sort of DevRel product space under that product, under the service team. So that was like “Okay, what happens?” There was a handful of us on this team; what do we do? And they said “We want to give you time to find another job internally. And if you find a job, great. Go ahead and take it. You can shift internally.”

Amazon’s also been mostly on a hiring freeze for over a year… And so that was like “Well, there’s a lot that’s not going to be available”, and most of those other teams are also requiring me to go into an office. And so this wasn’t a matter of “Oh, let me just shift and start doing new work.”

[00:20:04.27] I had to find a team that was local in LA, because I couldn’t just go to any office. I had to go to my team office. “Return to team” is what they actually deemed it. And so if I wanted to go to the local office, I had to find out what teams worked from that office, and then see if they had openings, and then work with them. There were a lot of barriers, and I just kept getting more and more frustrated with some of that, “Hey, this isn’t actually just an easy switch.” So there’s not a lot of teams hiring. If they are, I’m probably going to be in a space I have to leave anyway.

And the more people I talked to, I’ve found that more and more people across different areas, different services and different divisions were hitting some of these same limitations and same frustrations, where they’re saying “Actually, I have to find something else or move.” And in many cases, it was just “Hey, well, how do you get out of this situation?” And the official email from Amazon was like “If you don’t return to an office, you will voluntarily resign.” I’m like, that’s not a thing. [laughs] That’s not how it usually works with this sort of employment contract. Because this is a contract. It’s like, “I give you some time and some value, and you give me money.” And that’s how any job works.

And when the rules changed, when the contract changed, it became more of a frustration. I’ve found that a lot of times people were just silent, and they just said “I don’t know what to do, because I can’t do anything.” So they would just sit around. I’m going to answer some questions as they come…

And I finished out my work for what I was already scheduled to do, and I asked for severance. And I said “Hey, this isn’t working. I’ve talked to a bunch of teams, I’ve looked around, but this situation is just not working.” And that was where that frustration was really coming from, where I was like “I can’t do anything, and I’m not going to find another job, and I’m not going to voluntarily resign, because you got rid of the job I loved.” It was like, in any other case, when someone gets rid of the position that you really enjoy doing, there’s some monetary or some situation where it’s “Hey, we’re gonna close out this contract.” I know, it’s an at-will sort of thing. You can be let go at any time. But also, I know there are some labor laws that protect some people in situations.

But I wanted to write that blog post mainly to give voice to all the people I talked to that were “I can’t say anything, and I can’t do anything, and I have no network of external people.” And a lot of them are fresh into technology from the past two, three years. They’re junior engineers that are just being forced out without any sort of compensation, or connections, or anything else, and I was like, I really wanted to give them a voice in a lot of ways, just by sharing my own experience and saying “This is what I’ve gone through, this is what I’ve seen, and I don’t think that’s right.”

Justin Searls: …or must it infect the rest of my code, and tests, and so forth? And what I end up doing with almost every one of these libraries that I pull in, and what I’m looking for is something that’s just like “Give me a function to call that solves the problem.” And then I can write my own wrapper, in my own code, that basically charts an interface between me and this dependency, creating a little bit of scar tissue between my app and it. And then that way, if down the road that version resolution problem ever comes up and now I can’t update my framework because this thing, they share a dependency, then I can go shopping to replace it, and I’ve got a shopping list right there in the form of that wrapper, because now I know what’s the contract between that and everything else. Whereas if I’m just using this fancy little wrapper DSL, and I’m sprinkling it all over my code, now the thought of switching to something else could be tremendous.

Philipp Heckel: And sort of expect it to be what FCM is. So I think most people that try to build their app on top of ntfy really want Firebase, and don’t know that Firebase exists. So a lot of people have reached out to me and want to contract me to help them integrate ntfy into things, and every now and then I do help people, but most of the time I say “What you really want is – I’d love to, for you to give me money to help you work on your app, but what you really want is this.” And it’s mostly Firebase that they want. They want to deliver their custom push notifications to their custom app then that’s not what it’s for. It can work that way, but really, what you want is this.