Jerod Santo: Right. Whenever you talk to or read from the real builders, the ones who are there for principles and for technology and not for gains, and ponzis, and like quick things - which many people just come for that - they like the down markets, because all of the hype and all of the noise and all the scammers kind of just go away, and they can just put their head down and build cool stuff… Which is what they were doing the whole time anyways. But all of this stuff surrounded them. And so they tend to like times like this, because they say “Okay, we can just quit explaining it to everybody, quit talking about it, and we can build, build, build”, and eventually, show us to the Promised Land.

It’s getting to the point now for me where it’s like – I remember when we first had Gavin Woods on the Changelog talking about Ethereum, and I asked him “Where’s the application?” And I’m still waiting for the killer app; and we had ICOs, and we had NF Ts, and neither of those had been killer enough. So it’s like, “When’s it gonna hit?” Is it gonna be the Twitter replacement? Well, it seems like we’re going federated for that, not full blockchain for that. All those people trying to do it, but… I don’t know. I’m still waiting for real-world, “How’s this gonna make my life better?”

Kevin Owocki: I think you can think of the Ethereum blockchain as a giant distributed database. What the blockchain does is it allows you to manage trust between parties across the internet. One of the reasons why that’s really powerful is we can store all the information about the bounty task on the blockchain, and that can be the source of truth for all parties about the state of the task.

So in contrast to a legacy financial system crowdsourcing platform like Upwork, for example, Gitcoin never actually is an intermediary that holds any of the funds between the bounty submitter and the bounty hunter. So when you post a bounty to Gitcoin, you’re actually submitting it to the Ethereum blockchain, and the funds that you’re associating with the issue live on the Ethereum blockchain, which is just a massive simplification over the credit card and legacy financial system where we would have to deal with translations between currencies, and having a legal structure associated with doing Escrow and stuff like that.

Basically, the task goes onto the Ethereum blockchain, and the way Metamask works is it’s just basically an Ethereum wallet that allows you to confirm that you meant to put that information on the blockchain, and the technology that powers Metamask is actually RSA public/private key encryption. So if you wanna do a transaction to the Ethereum blockchain, you’re signing it with your private key, and that’s how the rest of the computers on this giant distributed database know that the funds are authorized to come from you, and that the information actually came from you. So it all goes out there into the blockchain world, and then other people can pick up the funded work and turn it around by submitting their own transaction to the blockchain, which claims the funded work on the platform.

All this functionality lives on smart contracts on the Ethereum blockchain, and that’s a pretty powerful way of removing the need to have an intermediary, the need to do international translation of currency…

The other powerful use case that this enables is the ability to pay your bounty hunters with tokens. If you think about all these ICO’s who have raised 10-20 million dollars to fund their projects, they probably have some Ether on hand that they can use to incentivize actions of their community, but they also probably have a native Ethereum token that they can use to incentivize work on their repo.

[59:13] That’s powerful, because it allows them to tie their incentives as repo maintainers with the incentives of the person who turned around the work, because they both hold the same token that rises and falls in value associated with the success or failure of their project. So I think that that’s an interesting use case that could never even exist in the legacy financial world.

Stephan Ewen: Let’s say you’re recording the episodes and then every time an episode is done – no, let’s do an AI thing here, or so.

So you’re building your chat where you can chat with an episode, like “Okay, tell me where did they talk about this”, or “Tell me what episodes talked about these topics”, and so on. So what you’re doing whenever an episode is done - you’re feeding it first through a model that transcribes the audio, then you’re chunking it up, feed it through embeddings models stored maybe in a vector database, and then you have kind of a RAG style way of… You know, when a query comes, create the embedding, look up the similarity search in your vector database, feed it to the model to get the answer… For something like this, if you – let’s say you started just building the flow in let’s say a Node.js application, like in a simpler way; you just said “Okay, here’s the episode.” I have something, it gets uploaded… Let’s say you’re uploading it to an S3 bucket and there’s an event whenever something gets uploaded to this bucket; you have an event that represents this, and then it starts a Node.js script, or something like this. And this script is of the type that if it fails, somebody would have to restart it. Now let’s say you’re trying to implement that with Restate. I would say approach it the following way… The first thing is get a handle of Restate itself; there’s a cloud service that you can use on our site, which has a free tier. Either go there, or just use one of these ways to run it yourself on a single machine with an EBS volume.

[01:22:06.14] Then you have the server there. Then put your Node.js script maybe – you can actually put it on something like Lambda or ECS; just like use a serverless option to host this. And then use the Restate SDK to define the entrypoint and tell Restate “Okay, hey, here’s the service that you now should durably manage.” So Restate will then go there and discover this and understand “Okay, hey, there’s this (what what do we call it?) video transcriber, or video embedder service.” And then Restate knows about this. And then you would go to your Amazon console and say “Okay, for this type of event I want to create a webhook to Restate”, so that it makes an invocation to Restate and says “Okay, this thing has been updated.” You know, the kind of event that would previously call directly your Node.js process or script, you actually make it an HTTP call to Restate, and Restate would then call your process.

You’ve already gained one thing right away. You now basically have a reliable queue in front of it, right? Like, just that if you don’t do anything special. So when the webhook comes, it’s going to be acknowledged back, and Restate has this [unintelligible 01:23:15.22] of your process if it crashes, it will retry this… It will actually give you a nice observability; much more than you would get from your average message queue, about like individual retries, configuration about time-offs and back-offs and timelines and so on.

As a next step, you would actually then go into your script and say “Okay, let’s actually identify the steps where if something fails in this step, or after that step, I don’t want it to go back.” Like let’s say forking the process that does the transcribing, or like calling the LLM to create the embeddings. You introduce then the Restate context that you get by using the Restate SDK and just say, “Okay let me wrap these API calls just with restate.run.” That will capture the results of this durably, and basically turn – you’ve now turned it basically into a workflow.

Let’s say you want to do something like parallelize the different steps. Maybe just typing this one by one through this embeddings model is a little tricky thing… You want to fan out. You could then go and say “Let me try and do the exact same thing I do in a regular Node process and just make a bunch of function calls record, like remember the promises”, sort of a way to promise that all for those in the end join the results, and put those in the database. You can do exactly that in your code. Just, again, anchor this in the Restate context, so you get like this durable parallelization, durable stuff like scatter, gather, and so on. And so you would then incrementally sort rewrite your code to say “Okay, let’s actually make this step durable, let’s make that step durable, and that step durable.”

Say as a next thing maybe one of your folks wants to approve it before it really goes out. So you then possibly – you “Let’s do that in the simplest most possible way”, which is we create an awakeable or a durable promise in Restate and say “Okay, somebody needs to complete this actively”, like send an event, make an HTTP caller to complete this and say “Okay, this is approved, go through” or “No, this is not approved. Abort.” You can then use, for example – you could put the result, the transcription just in Restate, save it, and somebody could look at it from the UI and then say “Okay, yeah, I’m making an API call here to approve this”, and it continues. And so you can then incrementally rebuild your process into durable steps.

As the next thing you could then, for example, take it and migrate it from a long-running process to a Lambda function. Because one of the nice things you have with durable execution is when it’s waiting for something else to happen, it can actually just make this thing go away. Because with durable execution, it knows how to recover back to the place where it was by replaying the history of durable steps.

[01:26:14.28] So you could then say – you know, if you’re on vacation and you approve it a week later, you don’t have some process running and waiting for it. It’s just like, it’s going to go away, and when the approval finally comes, it’s going to come back, use the durable steps to replay back to the point, and then to the remaining steps. And so typically, folks would incrementally then rework their non-durable services; first connect them to Restate to basically get the equivalent of a durable queue, and then incrementally rework it and say “Okay, I want to add durable steps here, maybe parallelization, maybe a signal”, and…

Adam Stacoviak: It’s a term of endearment. I know it is. My ego’s not shot here, man. I know my kid loves me. Let’s go.

Jerod Santo: Exploring a stablecoin bank

Bridget Harris goes deep on the potential of crypto stablecoins to disrupt Visa and Mastercard’s duopoly:

Right now, Visa and Mastercard charge merchants egregious 2-3% swipe fees — which is typically their second highest cost after payroll. Sadly, smaller merchants are disproportionately hit by these swipe fees…. This is partly why Visa and Mastercard’s profit margins are each higher than 50%: small businesses have no choice but to accept Visa and Mastercard since they control 80% of the credit card market.

A stablecoin network could drop those swipe fees to essentially zero. Merchants hate swipe fees — rightfully so — and if they could opt for a lower-fee network that wouldn’t limit their TAM, they’d switch in a heartbeat.

I’ve long asked cryptocurrency bulls to point me to the killer apps. First it was ICOs, which didn’t really pan out (too many scammy founders). Then it was NFTs, which were a lot like the pogs of my youth (fun for a day or two). More recently, it’s been ~~prediction markets~~ gambling, which I admit is a legit app, just not one I’m interested in.

But, “knocking 3% off every transaction in the world”… That dog could hunt! Unfortunately, it’s also going to require government intervention, of which I’m a perma-bear.

Angelica Hill: Well, actually, most of us. I see you, Mat. Most of us don’t have big egos. [laughter]

Justin Garrison: [00:42:16.03] And what’s your ideal – I came to the livestream for how you’re running System Initiative on top of ECS… And one of the things that I was curious about and seeing that was like seeing the diagram of all the components. It’s complex. There’s a big diagram of a lot of stuff that’s all connected together in some way, shape or form. But there’s got to be a threshold there of like running a couple EC2 instances or an ASG with a load balancer. I probably don’t need System Initiative. Doing something that is fully System Initiative, everything, every component of it, to run it as a SaaS - it seems complex to do in one main diagram. And when I look back and think of the enterprise environments I’ve been in… I couldn’t imagine all of Disney Plus –

Adam Azzam: So you had said “Talk to me the experience of taking my Python code and converting it to a Prefect workflow.” For folks that are listening, it’s as simple as from Prefect import flow, and then @flow on top of your function. That’s all the work it takes to take a Python function and give it superpowers, basically. To add all the stuff that we’ve talked about. And that doesn’t take away any sort of attributes of that function. If I take my Hello World or my ETL flow that runs locally and I start adding Prefect things on top of it, it still runs locally. And so it’s not like – we don’t sort of require additional infrastructure just to have your stuff run. We detect whether or not you’re running it locally or not. And if you’re running it locally, we execute it as if it’s regular Python functions.

[00:32:03.29] Now, there’s kind of two stories of “Well, this thing is working on my machine. How do I get it to run somewhere else?” We have two ways of doing this - and I’m sorry for the vocab lesson - which is let’s say that… What’s the crawl/walk/run for anybody trying to get something to run remotely? It’s you start off on your machine, and then it’s like “Well, I’m gonna go get a server somewhere, and I’m just – if I can close my laptop and go to bed and this thing is still running on my server, that is what counts for me as running it remotely.” That’s the next step in the hierarchy of needs.

So for that, if it can run locally on your server - you go to EC2, you SSH in, you’ve got your console, you pull your GitHub repo, you pip-install your requirements… You’re still able to execute that flow on your machine, and now if you execute that function and you hit say like .serve, now in the same way that you would start a fast API application, it is now running, you can specify a schedule that it runs on… It exposes an HTTP endpoint that you can call out to if you want to invoke it on demand, and it listens to events.

So maybe you don’t want to just run it on a schedule. Maybe sometimes you just want to be able to hit a specific endpoint to manually invoke it, maybe from your Django app or something like this, or maybe you want to emit an event into the world that says “Look, when my user signs up, when my dataset is ready, I want to have this ETL flow go and operate on it. Because my data is not always ready at 6 a.m. Sometimes it’s not uploaded till 6:02.” So sometimes you want to have things be a bit more dynamic. That’s what happens in a [unintelligible 00:33:47.10]

So where are we in the hierarchy of needs? I had it on my laptop, but it stopped running when I shut it. I go and I get an EC2 instance, I have it running there… Now I can shut my laptop and it’s still running. And then now how do you do this at a massive scale? And how do you do this auto-scaling with respect to the amount of work that you have, right?

If your EC2 instance - you hit it 10,000 times, and all of those things, every invocation requires downloading some data, that machine’s gonna fail over quick. So what happens when you want 10,000 things to spin up just as many machines and you want to fan out? That would look like my flow.deploy.

So I’m here on my laptop, I’ve got my Hello World flow… I would literally just say “If name equals main”, like I’m writing any other script, flow.deploy, and the sort of autocomplete in your IDE is gonna tell you “Alright, what do you want to name this thing? Where do you want it to run? Do you want it to run on ECS? In Amazon? Do you want to run in Google Cloud Function? Where do you want it to run? Can you point us to – do you have this in a Docker container somewhere? If so, we’ll just like pull it from whatever Docker registry that you have. Or do you want to point us to a GitHub repository?”

And one of the nice things is when you build a tool like Prefect that treats failure as a first-class citizen, you see the failure modes of tens of thousands of people trying to get their code to run, and so this really allows us to build an ergonomic experience that’s like “Look, here’s the minimal stuff that we need to know in order to have a high-reliability guarantee that this is gonna work” when you shut your laptop, when you submit it to a Kubernetes cluster, what have you.

So the experience is really pretty dead simple. If it’s running locally on your machine, you can hit .serve, and now it starts a process on your machine. If you want it somewhere else, it’s .deploy, and we can guide you through exactly how to point it at remote infrastructure. Does that answer your question, Daniel? I hope I’m not glossing over anything.

David Beale: Honestly, though - it’s funny… Kustomer with a K, but they used Amazon ECS. So like –

Autumn Nash: Yeah. Well, I mean, not just that, but it can make an EC2, right? It can make all these different infrastructure pieces.

So how do you then keep them secure that something can’t go in and, I don’t know, make a hundred EC2s on one of your accounts, or…? You know what I mean? Like a bad actor isn’t doing –

Steven Wu: There’s kind of a couple things. So basically, we’re cloud-native. A lot of these older generation stuff, like Splunk, like Datadog, like stuff based around Logstash, etc. a lot of that tends to be – it’s very important to these companies, because of a function of the times that they have good on-prem solutions, and they kind of have these applications that are designed to be run on physical computers, and so on. So if you’re running a Splunk cluster, you have like a – maybe not a physical computer, but even on the cloud option it’s like a VPC somewhere, and that’s like a node, and that handles all of the capacities of the system, which includes concerns like storage, it concerns compute… And one of the big things is that for the most part your usage of those resources are not uniform over time, and they are not uniform in any given instant. For instance, you need storage all the time, but you pretty much don’t query most of the time. So if you kind of think about what the average query use case looks like, you spend like five minutes typing stuff into the search box every day, and then the other 23 hours and 55 minutes you’re not, right? So when you have physical computers that have some constant resource constraints, and some very spiky resource constraints, you have a lot of like weird utilization issues… And this leads to a lot of basically costs.

So in our system basically each of our things, each of the pieces of that system, each of those concerns are farmed out to the cloud service that is specialized towards handling that. So all of the index files go in object storage in S3, we run ECS clusters, or whatever… Well, not whatever; that’s what we do. I actually know that. For the actual indexing. And then we run basically serverless compute for doing the query, and each of those things are chosen specifically because that particular operation has a particular set of properties in terms of the volume, the spikiness in variance, the availability requirements and so on, that are very, very different. And I can kind of talk about any of those in detail; I don’t want to get lost too much in the specifics, but…

I think we’re kind of seeing a world now where in the current generation of these log search platforms people are more and more – you’ll see the term separating compute from storage more and more in people’s marketing material, because I think we are realizing now that even very, very large companies like Fortune 50 companies are moving all their stuff to the cloud, that you can now in fact build things with the assumption that pretty much even the biggest companies have enough cloud stuff for you to work with… Whereas I think even like five years ago you couldn’t. If you wanted to work with a big financial institution, you had to have some literal on-prem distribution where you gave them a computer. So I think it’s there’s a sea change that has kind of enabled this to be possible kind of in like the business landscape, and just in terms of people moving from on-prem to cloud generally.

Jerod Santo: I feel like that’s the mic drop. He just stroked our egos and answered your question all in one. Mic drop.

ChatGPT: That sounds like a big project. Here are a few things to keep in mind when migrating services from AWS ECS to a hybrid Kubernetes cluster. One, assessment and planning. Start by evaluating your current ECS configurations and services. This will help you understand the dependencies, data flows and necessary resources. Mapping out how these will translate into Kubernetes is crucial. Two, Kubernetes environment setup. Setup.

Brian Breiholz: Definitely. There are hooks, as I mentioned. There are engine uses, what’s called an ECS, Entity Component System; we probably don’t have the time to go into that, but it’s a way to manage your data and functionality. And for example, we provide hooks, like use component, where it’s,okay, you give a ref to your entity, you provide the component type, and you get a reactive hook. Or if the data updates too fast, like position or ammunition, for example, you get a callback function, so that you can manually write into a diff, or whatever, as your output.

So there are definitely hooks, which are similar to classical React; they are higher-order components, where in Cooper & Onita for example I had these big, red buttons that you could walk on, that would lift a platform. That was basically the mesh for the button, together with a higher order component for - I think I called it pressure plate, or something like that… Where it’s, okay, it will have this logic attached to it, you can move things up and down, but whatever gets rendered is up to you. And all the other platforms - like, there was a lift, or some rotating thing… All of them were higher order, in the sense that you could pass whatever gets rendered into there, and it will work out of the box.

Justin Garrison: It’s interesting too, because complexity at some point becomes the standard. And when you think of – I always think of cars. Cars are extremely complex systems, and they’re failing all of the time. You have multiple misfires in your engine all of the time, and you don’t ever know about it, but anyone can be taught to drive it. And the pedals and the steering wheel, and some blinkers, if anyone uses them, are pretty standard interfaces, right? And those are things that we can use reliably, and anyone can jump into any car, and it’s like “Oh, I know where most of this exists.” And something like Kubernetes is extremely complex, but it’s like a standard interface, too. Like, “Oh, I know how this works”, because I can jump from one Kubernetes cluster to another, and say like “,Oh I know how this complexity is going to work in some regard. There’s gonna be details and things that I don’t understand”, but that just kind of becomes the easy layer of it.

When I was at Disney Plus, we were building everything on ECS, and we could never find any ECS experts. Like, there was no one out there that knew ECS, unless you worked at Amazon. I’m like “We’re not hiring Amazon people, so how do we get ECS experts to come?” So we hired a bunch of Kubernetes experts. Everyone on the team was a Kubernetes expert, and we’re like “Hey, if you squint, this is kind of Kubernetes.” But there was all these edge cases and things that we had to build around, because it’s like “Oh, it’s not Kubernetes, and we know that we’re not driving a car, we’re riding a bicycle”, or something. It was a different paradigm of how we had to treat the system.

Adam Jacob: But when you think about it, the results – what drives good or bad outcomes in DevOps tends to be how good the experience is of trying to get those changes into the world, right? How tightly do we collaborate together? How cleanly can we work together? How quick are those feedback loops? Right now they’re incredibly slow. So think about how long it takes to know that the infrastructure change you made to Terraform was right or wrong. And in System Initiative, it would happen in real time. You would open a change set, you’d change the architecture, the qualifications would run, and they would tell you if it makes sense or it doesn’t make sense, and it happens immediately. That’s a dramatic transformation in the experience of what it means to do that stuff. So the outcomes that we produce are also therefore going to be better, because the collaborative nature of the work – like, one of the things that we saw when we built System Initiative with System Initiative was… Because we’re going to run this as a SaaS product, right? So watching our own team build System Initiative with System Initiative – they did it in real time, not by like being on Zoom, watching one person drive, like you would if you were pair programming, but instead, all three of them were logged into the same workspace at the same time, and they just split it up, and they were like “You go build the VPCs, I’ll go build the ECS cluster, you go build the IAM rules…” And they just all took different parts of this big graph, and they just went to work. And they did it all in real time together, and it was incredibly cool. We had them all share their individual screens. So as spectators – you can’t see this on the podcast, but our faces are like three-up, basically, in the thing that’s actually doing this. And we watch them three-up on each of their screens, and they’re collaborating together in the same change set, in real time, watching the infrastructure come in, things are going red, things are turning green, they’re talking more… It was incredible. I haven’t worked that way since I was building systems by hand for an ISP in the late ‘90s, with my best friend. And I was like “Oh…” That’s something we lost. People don’t even know we lost that, because you have to be old enough to predate the automation we gave you to really remember that there was a time that we did that work together, where that was the norm.

[01:24:05.27] Anyway, it’s beautiful… And I think that that’s the number one most disruptive thing System Initiative is going to do for people, is it’s going to let them see that stuff happen again, in a way that feels more real-time, that feels more iterative, but gives up none of the power or control that comes with the code, and to be able to do it together, which is what we know brings about the best outcomes. And so I think as you climb up the stack, when you start to get to “I’m an application developer”, you’re not going to want to see that the application’s full of stars. You don’t necessarily need to see all the infrastructure that builds your application. But it’d be great if you could dive down to it if you needed to. It’d be great if you could be like “Hey, the application’s not quite doing what I want it to do. Can you come take a look, engineer?” and summon that person to change set and show them directly the changes you’re trying to make. I think it’s very cool.

Justin Garrison: It was all ECS.

Kyle Wiens: I bet some people would love to buy that, and then have a product that would last longer. Yeah, nobody would. So what we need is a repair score next to the price. So when you go to buy a product, it should say “Hey, this is an eight out of 10. This is a three out of 10.” And then you know “Well, I don’t care if this is fixable.” Or maybe I really do care; I’m spending a lot, I’m investing in it, and I want it to last a long time. And so this is not a novel idea. France implemented this law about three years ago, and France rolled out a rule for cell phones and laptops and a bunch of other product categories. It says you have to have what they call the indice of repairability. It is a score card that factors in, there’s service manuals available, can you get parts quickly, how easy is it to take apart… And so if you go to apple.fr, and you’ll look at an iPhone, look at the price, the repair index is right next to the price on apple.fr. And if you go into the French equivalent of Best Buy, at retail, next to the price, it’s got the repair index next to it.

So we worked with the French government and helped them technically build and develop the scoring system. This is going to be integrated Europe-wide in coming years. It’s going to be built into the European eco index. So there’s gonna be a variety of information available when you go to buy an electronic products in Europe, and that will include repairability. So then the question is how do we do that in the good ol’ US of A? Because all of us are not shopping for electronics in France. And answer there is the Federal Trade Commission. So you know how you go and you buy a water heater, and there’s a yellow sticker on it that says how much energy it’s going to consume over its life?

Paul Vixie: Okay, so it used to be that if you were a CDN, like Akamai, a content delivery network, that you could simply operate the server, or whatever, Microsoft.com for example - you’d get a query, you’d answer it. Somebody would ask “Where’s www.microsoft.com”, and you’d look at the source of the query and say “Gee, we’ve got 35 copies of that content around the world. The one closest to you is that one.” And it would give you an IP address as part of the answer to your DNS question, that was the mirror that was closest to where you were coming from. This went away once OpenDNS and Google and everybody else started doing this, because the place where the DNS question will be coming to you from was not the end user. It would be – and it wasn’t their ISP, it wasn’t their house, it wasn’t anything that would help you predict where the web fetch was going to be coming from. And so a collection of companies who had monetized things to the point where they no longer work, proposed additional complexity, with vast privacy risks, and then deployed it, and it is the standard for the internet today.

So EDNS, which was my thing - so I’m blamed for this sometimes - EDNS client submit ECS, and it’s just a way to amend your query from your 8.8 server, or your OpenDNS server, your 1.1 or 9.9 server… It would amend your query by saying “And furthermore, the question I’m sending you is due to an end user who was on the following network. So if you’re planning on doing the CDN thing, you craft an answer for them based on that address. That’s the address to use. Don’t use my address, because I’m not where the web fetch is going to come from.” And boy, there were a lot of bugs. And there are still a lot of bugs.

[01:11:45.16] And so by sort of getting in there and saying “This is my leverage point. This is how I’m going to innovate. This is how I’m going to shim myself into the internet ecosystem, so that I can add value and get paid for it”, DNS works less and less well. And so when somebody shows me that and says “Oh Paul, you’re such an idiot. You created this terrible thing. It’s always DNS”, well, it’s not my DNS. The world has taken DNS for a ride, and there’s no guardrail where you’re driving it, and it doesn’t want to work the way that you want it to work. And I’m not surprised that you’re having the trouble you’re having. Sorry, long answer to short question.

Johnny Boursiquot: I definitely wanna touch on sort of – and I’m gonna paraphrase what you just said, Ian, in terms of sort of expectations. What is the expectation from a performance standpoint of this particular service? Because I think those expectations are sometimes and often in production environments from a resource standpoint translated into a certain CPU allocation, or a certain memory allocation. I had sort of a nasty surprise when I first started working with orchestration tooling, for the Dockers and these kinds of things, and realizing that “Oh, my program–”, the way I experience the output of my program is that it’d be working and doing things, and it would reach a certain threshold, and all of a sudden, it’s almost like it would just stop dead in the middle of trying to perform some sort of action. It would just stop. And I’d be trying to debug, why is it that this thing was running, and all of a sudden it just stopped? No stack trace, no error message… It just got killed off. And I realized that “Oh, crap, the orchestrator–”, whatever, whether you’re using ECS, or Kubernetes, or whatever it is, that environment allocated a certain CPU and memory resources for that particular service, and whenever I would go above that threshold for a certain amount of time, my process would get killed. I didn’t realize my services were just being killed off by that environment. And the moment I realized that, I’m like “Oh, snap.” To Ian’s point, you kind of have to know ahead of time, and work out with - whether it’s either working with the ops team, or maybe you are the ops team if you’re doing SRE stuff, and trying to understand “What is it that I need to accomplish ahead of time?” Which I think feeds into sort of knowing what your application or services is supposed to be doing with whatever data it’s supposed to be doing it… Which kind of leads to, if you’re going to have to deal with a certain volume of data, and you think you have a problem with how you’re handling it, what’s the next step? How do you go about trying to figure out where your problem is?

Justin Garrison: Yeah, this is my first time in a professional role doing Dev Rel, doing content creation and doing this style of work. I’ve always just been an engineer, and I love this job. I felt I was good at it. I enjoyed engaging with people and learning something deep technically, and then telling people how it works. And that cycle of back and forth, and then being part of a product that I know a lot of people used was also just wonderful. That was something that I joined Amazon for, and I really a lot of things at Amazon with how they were working on things, and that involvement to get some outside voices or non-engineer voices necessarily. I’m not writing the production code. I’m helping guide the product and then testing it and saying “Hey, this is how customers should or shouldn’t use this thing.”

And I’ve had experience across the board with – I helped launch App Runner, I did a lot of work with ECS, I had some talks I’ve done about Lambda… This is really broad, because no one uses any of these services in a vacuum. And I liked that breadth that I was able to just “Hey, I can pick out any of these things.”

This year - well, last year. It’s 2024. In 2023 Amazon really pushed for return to office… And I was hired as a remote employee before the pandemic. I started during the pandemic, but I had my contracts and negotiations and everything before, as a fully remote employee. I was remote at Disney before, so I was used to that. I needed a remote job. And in 2023 they were really saying “Hey, we’re gonna start returning to office.” I was like “I’m a Virtual Employee, I don’t have an office.” The closest office to me is maybe an hour away, maybe two hours away with LA traffic. I’ve gone there a few times to meet up with people and to have some meetings, but it’s not a regular occurrence. And I was always told that my role would stay remote, and I didn’t have to worry about it.

And then things started changing, more and more noise was happening during the summer; it was actually more people are gonna start coming back to an office part time. And I kept being told over and over again by management and my leaders “No, no, no, you’re fine. You’re a remote employee.” My entire team is not in a location. The DevRel team is all across the United States. And when I joined, we were all over the world. I was like “We don’t have a timezone, let alone a location.”

And so as things started progressing, it was becoming more and more clear that this was going to affect me at some point. And we filled out forms to get a remote exception, which was for a one-year thing; I had to renew that every year. And I got my approval for remote exception just days before I was told that our team was actually going to be disbanded, and our team would not exist anymore under the Kubernetes org, as part of the Kubernetes product team. And they wanted to get rid of this sort of DevRel product space under that product, under the service team. So that was like “Okay, what happens?” There was a handful of us on this team; what do we do? And they said “We want to give you time to find another job internally. And if you find a job, great. Go ahead and take it. You can shift internally.”

Amazon’s also been mostly on a hiring freeze for over a year… And so that was like “Well, there’s a lot that’s not going to be available”, and most of those other teams are also requiring me to go into an office. And so this wasn’t a matter of “Oh, let me just shift and start doing new work.”

[00:20:04.27] I had to find a team that was local in LA, because I couldn’t just go to any office. I had to go to my team office. “Return to team” is what they actually deemed it. And so if I wanted to go to the local office, I had to find out what teams worked from that office, and then see if they had openings, and then work with them. There were a lot of barriers, and I just kept getting more and more frustrated with some of that, “Hey, this isn’t actually just an easy switch.” So there’s not a lot of teams hiring. If they are, I’m probably going to be in a space I have to leave anyway.

And the more people I talked to, I’ve found that more and more people across different areas, different services and different divisions were hitting some of these same limitations and same frustrations, where they’re saying “Actually, I have to find something else or move.” And in many cases, it was just “Hey, well, how do you get out of this situation?” And the official email from Amazon was like “If you don’t return to an office, you will voluntarily resign.” I’m like, that’s not a thing. [laughs] That’s not how it usually works with this sort of employment contract. Because this is a contract. It’s like, “I give you some time and some value, and you give me money.” And that’s how any job works.

And when the rules changed, when the contract changed, it became more of a frustration. I’ve found that a lot of times people were just silent, and they just said “I don’t know what to do, because I can’t do anything.” So they would just sit around. I’m going to answer some questions as they come…

And I finished out my work for what I was already scheduled to do, and I asked for severance. And I said “Hey, this isn’t working. I’ve talked to a bunch of teams, I’ve looked around, but this situation is just not working.” And that was where that frustration was really coming from, where I was like “I can’t do anything, and I’m not going to find another job, and I’m not going to voluntarily resign, because you got rid of the job I loved.” It was like, in any other case, when someone gets rid of the position that you really enjoy doing, there’s some monetary or some situation where it’s “Hey, we’re gonna close out this contract.” I know, it’s an at-will sort of thing. You can be let go at any time. But also, I know there are some labor laws that protect some people in situations.

But I wanted to write that blog post mainly to give voice to all the people I talked to that were “I can’t say anything, and I can’t do anything, and I have no network of external people.” And a lot of them are fresh into technology from the past two, three years. They’re junior engineers that are just being forced out without any sort of compensation, or connections, or anything else, and I was like, I really wanted to give them a voice in a lot of ways, just by sharing my own experience and saying “This is what I’ve gone through, this is what I’ve seen, and I don’t think that’s right.”