Nishant Roy: Yeah, I think you’re right. I mean, it’s an impossible decision to make, right? Because on one hand, if you’re spending too much time on optimizing early on, you’re not going to make it as a startup, especially given this environment we’re heading into now… Speed of everything is of most importance. On the other hand, like you said, you end up with these legacy systems that don’t then support hypergrowth, or supporting new products.

I think one big thing that we’ve seen over the last probably five years to a decade or so is widespread adoption of video. So folks or companies who didn’t really optimize early on for, let’s say, content delivery for instance, suddenly you’re delivering video, which is obviously a lot more expensive, and people are having to reconsider how they built those systems, or in some cases even rebuilding those systems to operate at this new scale that customers now want.

For us, for my team in particular, the biggest scale challenge we were going through was the team growing. So like I said, my team specifically was ten people, now the entire ads infrastructure org is probably getting close to about 100. The ads team is probably three to four times that number. So the question we were asking ourselves is now that we’re going through this phase of growth, building a lot more products for a lot more users, for a lot more partners, how can we continue to keep that same level of velocity that our pinners are used to, our partners are used to, and us as engineers are used to, with this growing org, without causing more incidents or without having people stepping on each other’s toes? And I think that is, to some extent, the eternal quest; we’re never going to really hit a perfect happy – I don’t know what the right word is, but middleground between all three of those factors. But that is something we’re constantly evaluating and tuning for, is how do we build our systems in a way – either can we build better modularity, can we have more config-driven systems where folks can make changes without needing to understand how the rest of the system works?

For instance, for rolling out new experimental models, do I need to actually go and read exactly how those models are going to be chosen? Do I need to understand what features have been fed into my model? Do I need to understand how those scores have been used? Or can I just go in and make a simple JSON change or something, and say “For this subset of traffic, which may be country, or surface, or device, for iPhone users coming from Canada on the search page, I want you to use this model, with this percentage of traffic. And I don’t care how the rest of the system works.”

So those are the things that have been really on our minds constantly, and something we’re looking to continuously improving. In fact, the pre-submit test framework was one way that we did that, is anyone can go and make a config change to add a new metric and add a new slice, without affecting how any of the rest of the system works.

Our actual ads delivery system isn’t fully there yet. We’re continuing to improve this for developers, making it easier for them to make changes, a) without bringing down the system, b) without blocking other folks or needing to understand how your change interacts with someone else’s. But like I said, it’s the eternal quest, as the product offerings from the sales side and the product teams get more and more diverse, we realize that some parts of our system just weren’t built to support those sort of products in mind.

Something as simple as if we need to serve - these are some of the problems that we’ve solved perhaps, but if we need to serve both video and image ads for the same request, do we have a good way of doing that? Not a particularly hard thing if you’re building a system from scratch, obviously, but once you’ve built it with a certain assumption, just these basic things sort of start to fall apart, and then you need to go and look into “Should I just be redesigning this whole thing, or is there a quick and easy way for me to get this off the ground, and then go in and redo it to unblock the next big thing?”

[31:48] And again, what ends up typically happening - I think this is probably true for most big companies - is you obviously need to get that MVP out, so you do something a little hacky to start out with, and then you retroactively go in and ensure that a) it’s not gonna break anything, and b) how do we make this a more pleasant experience for developers and product managers alike? So that’s the wheel that’s always spinning, and we’re trying to stay ahead of it, but we’re usually playing catch-up.

Russ Maschmeyer: Yeah… I mean, we have a word that we use at Shopify, “toil”. This idea of work that kind of has to be done, but isn’t desirable work to do… And so we look for toil that merchants do. And so we spend an enormous amount of time sitting down with merchants, talking with them about how they use our platform, what they want more out of our platform, what they wish they could be doing with their business, what they are doing with their business… And from that, we’ve learned a ton about what are the ways that merchants would like to spend their time, and then what are the ways that they just kind of have to, because that’s the way the world works right now. So I think the opportunity for us is to find those moments, and to build tools, particularly magic tools into those spaces, that just sort of like make that go away. And when we do that, what we hope is that merchants will take that extra time that they have, that hour that they got back not spending on that one product description, or that one blog post, or that one email headline, like “Ah, should I use A or B? I don’t know…!” and just give them a really easy tool to generate that content, make it really high quality, give them the control to adjust if needed… And then publish it really quickly.

Dan Kohn: [44:05] Well, yeah. And for what it’s worth, even the content of the article didn’t have that as well, but obviously, the headline writer gets compensated by clicks, and that’s what – you know, maybe their A/B testing got more clicks.

Lucas Fernandes da Costa: So I agree with him. I do think you should have the direction in which you’re going. And now, how are you going to get there - things are going to get in the way; you’re going to have to be creative about which strategies you’re going to use to go into that direction and continue to make progress. Now, the problem is not with setting the trajectory, the problem is when you say, “We’re gonna go from point A to point B in X amount of time.” Because then, one of the dangers that you have is that you might be incentivizing the wrong thing at all costs.

Imagine you have a sales target, and your sales target is X, at date Y. Someone that gets to that target on that date, no matter how much they fluctuate, and how many people quit because of the attrition, and how ruthless that person that wants to achieve that goal is, that person is going to achieve that goal. Right? That’s the problem with having a long-term plan. And also, reality doesn’t care that much about what you think. So that’s one problem.

Now, if you’re measuring direction, if you have a predictable sales machine, if you want to use the Aaron Ross term, if you have like predictable revenue, and you’re going “Yeah, but maybe not at this rate”, maybe you’re not going to achieve this deadline at that particular date, but you’re going in the right direction, and you can make small adjustments to increase the slope, and make you go up - that’s better than having an erratic machine that hits the goal. So that’s what I like about setting a trajectory, and having metrics that actually measure consistently how you’re performing against that, rather than just saying whether you’ve reached a particular end state or not.

So people need to be very careful with their metrics. That’s another thing I say on that blog post, which is - if you’re just looking at metrics, and you’re managing exclusively by them, you’re gonna make wrong decisions, because… I think Deming has this quote where he says that the measurements of productivity in the United States don’t help increasing productivity in the United States. You need to look at the systemic causes of those problems. Metrics, they serve as an intervention point, as a trigger point. So they tell you “Something looks weird here. Maybe you should go and figure out what’s weird.” That’s how I see metrics, and that’s why I think it’s important to get metrics that reflect a direction, rather than setting a particular goal, at a particular time, and chasing it at all costs.

Josh Padnick: Well, the threat of a fork had to be credible in order for our request for them to revert, to be meaningful. So forking was our plan B, but we were prepared for Plan B. And then, when their deadline came and went, it’s all systems firing for monitoring the fork.

Jeremy Daly: Yeah, so it all depends, obviously, on what you’re building. But if I’m working on a new greenfield application, I’m going to ask myself the question “Can this be built in serverless?” If the answer is yes, then you build it in serverless. If the answer is no, then you ask yourself that question “Can I build it in serverless?” because you probably can.

It’s sort of a thing to me where I can’t see many applications that the majority of them couldn’t be built in serverless. I do think there are some limitations, again, especially with long-running tasks and things like that, but Serverless Inc. is launching v2 of their framework, which is gonna be cloud-agnostic, and one of the features they have there is you can actually launch your functions either as Lambda functions (which would be the traditional serverless), or you can launch them as Fargate functions or Fargate containers, so it would actually launch your function into a Fargate container, so you are using Fargate into a container so that you could run that as long as you wanted to. Basically, it would build the container and launch a little server for you and scale that. That’s kind of a new thing where serverless might be heading, where containers might be part of this.

But anyway, so if I’m building a new application though, I would pretty much look at it and say “What do I need to actually process? What’s the business logic that I have to write?” Because I think a lot of times people start planning an application they say “Okay, well what database should we use? What programming language should we write it in?” With serverless, I think you can just basically say “Okay, what do I wanna solve?” and then you can find a bunch of managed services and pieces that you can glue together, and you really don’t have to write that much code in order to get a working application.

You’re most likely gonna have a front-end to your application, whether that’s a React app, or Vue, or Angular, or whatever you’re using, then you start thinking about “Okay, how can I have serverless back my CDNs? How can I put stuff out in an S3 bucket, or on one of the other CDN providers, and say ‘That can be my single-page app.” Maybe that can go beyond a single-page app, because another component - I’m rambling here a bit, but talking about this gets me sort of excited, because I think this is definitely the future… If you look at something like Cloudflare workers, or Lambda Edge, which is sort of the global distributed CDN that will call serverless functions as different events happen…

[51:55] So you can call a serverless function when somebody tries to access a cached object somewhere, and that can change the headers, it can detect what region they are and route them differently. It can perform A/B routing, so that it goes different places; it can know that it’s a mobile app, or a mobile device that’s accessing it, so it’ll do something different there.

Not only that, it can actually wait for the response from the origin and then do something with the response from the origin to say “Okay, I’ve loaded an image, but now I wanna add these five or six headers to it, or I wanna change the caching behavior of it because it’s being accessed from a mobile device or being accessed from the EU, rather than being accessed from the United States”, or something like that. So you start layering in this now, where you have all of these back-end services that are glued together with serverless, and then you have all of these CDNs that are out there, that can host the front-end of it… And not only can they make API calls and do things like that, but they can also interact just as part of the execution of loading something there. So you can handle your SEO, you can – the possibilities are quite limitless when it comes to that stuff… And I’ll talk about authentication, but I’ll stop in case you have any questions in between.

Jerod Santo: Well, you only get one path through the maze, so like you said, there’s no counterfactual. We can play what ifs all day long… And it is interesting, and sometimes helpful to do that. I poke fun at Salesforce… Ultimately, I think they’ve been a great steward of the project, of the business… Yes, has the innovation slowed coming out of Heroku in the last couple of years compared to when you were there? I think absolutely, as a customer and just as an onlooker. But like yourself, I happily use Heroku to this day, when it fits my needs, which is in many cases. It’s a great platform, and it continues to be that apex of developer experience that other platforms try to aspire to. Could they have taken it further by now? Of course, we hope that it would… But it is, I guess, as they say, what it is.

I’m curious with regards to the most recent stuff though… So now you’re like a decade past; you’re doing your own thing. Emotionally, your mind space has moved on… You’re looking back. And we do see moves recently coming out of Salesforce with Heroku; the biggest one, taking away the free plan, and just this new direction which is very enterprisy, kind of counterculture to what Heroku began as, which was like the hackers’ platform, like the indies, and just the ability to play, and try something new, and start something from scratch, and have a runway before you have to get serious about it was always part of that Heroku aura for me. And so a lot of people are upset about this, thinking either a) we’re just mad because we had a free dyno and we don’t have it anymore, or b) this is kind of like the changing tide of something that we once loved, and it’s gonna start heading in a different direction. I’m curious, did you have an emotional reaction to that? Does it matter to you anymore? Do you feel like your legacy could be tarnished, or no? What are your thoughts around like - now you’re far gone, and they’re starting to change it in more substantial ways, that seem like they go against what Heroku was all about.

Mireille B. Reece, Psy.D: There were deadlines, so I was always pushing to get things done, and school wasn’t just my only thing I was doing… So to practice figuring out what things would work - and I’m pretty sure it was early on that I was like “Oh, screw it!” I just ended up walking away, and sometimes that would be - if it were late at night, I’d just wake up early and try to get whatever it is done. And I’ve always then gone “That’s my go-to move: walk away, go to bed, and revisit it in the morning”, because it would take me a third of the time to figure something out or get where I wanted to be in the morning, than it would if I tried to push harder, or tried harder.

Mireille B. Reece, Psy.D: Yeah. So P stands for Posture. How we sit… Again, there’s body language; if arms are crossed, that might come across as closed, but also, bear mind, it could just be that somebody is quite cold.

A stands for Affect. This is our scientific term for emotion.

T, Tone of voice. I talked about prosody, the manner in which I say things.

H, Hearing the whole person… So not just what they’re saying, but paying attention to the person in terms of not judging them, recognizing that they’re a human, they get to have their own challenges and struggles. Even if I don’t know what they are, it doesn’t mean they don’t have them.

And then finally, they Y, Your response. Emotions are contagious. Emotions, at the most fundamental level, our energy – which is interesting, because I think about anxiety, an incredibly contagious emotion, that’s hard to hold… So recognize that your response or your choice in responding can influence how that social interaction goes.

If I come across this cold, or condemning, or critical, or like we’ve talked about before, name-calling, that is not going to help foster more of that shared understanding. We all have these social groups, and I would think especially in the workplace, when people are working with teams, you have an idea of what that world is like, and the times, the deadlines, the expectations… So to recognize that they’re in it with you, so they kind of have a sense of what challenges you’re trying to navigate… And maybe even looking at empathy of like “Hold on… They’re human, so they’re still on my team.” I wanna start there, and recognizing everybody gets to hold that fundamental space.

Adam Stacoviak: Yeah. And then, honestly, I just was like – I like Raspberry Pi’s, they’re cool… NVIDIA and GPUs are all the rage… And I just heard a headline. Basically, if you’re a CPU maker, you’re getting into GPU making. If you’re a GPU maker, you’re getting into CPU making. NVIDIA has CPUs coming out, and Intel has GPUs coming out. So they’re flip-flopping. So what’s the other thing out there that’s maybe going to do this? Raspberry Pi. And an SBC style GPU, that attaches to these other smaller things would be totally cool.

Adam Stacoviak: I would love to see a comparison of Blue Sky, because the reason why I ask this is there’s a headline that you covered, I believe, in news… Blue Sky crosses the 15 million user mark, which is - it’s a lot of people. It’s not billions.

Adam Stacoviak: I mean, I don’t want to turn this conversation to the dark side, but the creation mechanism, to some degree, lives – I mean, I’ll go one step further and go back to that thing I mentioned earlier from Snacks, and I thought it was pretty interesting because there was one layer deeper on this… Which was the takeaway. The labels need social media fees to succeed. So we’re going back to the day of Napster, when they’re like “Damn you kids sharing files accidentally, or unpurposely, in this way, shape, or form that subjugates and subverts our ways of profiting.” And then you fast-forward 22 years-ish later and now there’s a headline that says, or at least a sub-headline, a takeaway headline that says “Labels need social media fees to succeed.” And they took a large hit. This is UMG, one of the most popular labels out there. They had Q2 results that were down. They would take a $10 million per quarter revenue hit. This is just paraphrasing some of the thing in there, but…

Kevin Ball: I’m gonna go with true. Go big or go home. This is a headline.

Adam Stacoviak: It’s definitely not a headline.

Adam Stacoviak: I saw a headline, I didn’t hear the details.

Adam Stacoviak: A headline.

Jerod Santo: Alright, let’s find out. Is that a headline or a headlie? [win alert] That is true. So all three of you successfully identified that as true. Good job. You all get a point for round three. Round four. “Elon Musk says X will fund legal bills if users are treated unfairly by their bosses.”

Adam Stacoviak: [52:07] That’s interesting to think about… Why now? Well, one thing Insider did say, a headline at least, that we can potentially agree on is “Apple sure kicked Meta’s butt today.” [laughter]

Adam Stacoviak: [55:48] Even open source program offices was less and less a thing. I think as open source program offices become more and more normal, or more and more needed, or legitimized in organizations as they grow, the need for a FOSS contributor fund inside that office grows as well. I mean, that’s just anecdotal feedback from myself; it’s why I see kind of happening through the tea leaves, not necessarily a headline I can point you to… But that’s how I see it happening. Like, these have become more and more popular. And it begins with someone who cares. And that’s you, Duane; you cared enough to use your leverage then.

Daniel Whitenack: Yeah. And I can say a few things that I’ve seen… Even in the past week or so, I saw some demos of different things people were building. There was a hackathon that I participated in remotely in San Francisco from Latent Space Demo Days, and Jeremy Fisher built this automated D&D referee, that was – I think it’s Dungeons and Dragons Infinity… And it essentially lets someone play Dungeons and Dragons infinitely. It never generates the same text, so you can play it forever, and there’s like an image component, there’s a text component…

I think this represents very much what people are finding so appealing about these things. In one respect, one aspect of what they’re finding appealing is the sort of endless creativity it seems, both on the image generation side and on the text generation side. But that’s sort of like - Dungeons and Dragons app, people might find that very engaging, but then you think about, okay, well, if I can do that, what does that mean for advertising and copywriting? Well, I can write up prompt to generate a blurb for an ad, and then I can write a prompt to summarize that blurb into a headline for my ad, and then I can use that headline in another prompt to generate an image for my ad. Right?

So people are building kind of this chain functionality that really does powerful, useful things, but there’s this other side of it where every once in a while you kind of get these scenarios that are either very disturbing, or you get output that that isn’t desirable, or people illustrate the sort of biases of these things… What have you seen on the more - I don’t want to say sad side, but on the unwanted behavior side. What is the behavior – so if we like this generative, creative behavior, and the utility that that can provide, what’s the other side of it? What’s the unwanted behavior that you’ve seen?

Jerod Santo: Alright, we are back for the award-winning podcast to play its potentially award-winning game, Headlines. This is a game in which I have scoured the interwebs and I have found headlines. I have also fabricated some headlies. It is your guys’ job to figure out which is which. So here’s how the game works. I will read a headline, at which point each person has the opportunity to stake a claim - is that true or false? Is it a real headline, or is it a false headline? If you guessed right away, you can potentially win two points. Or you can wait for more information. I also have the first paragraph from the news stories that I will read. You can hear the first paragraph, which will give you much more information, and you can then decide whether it’s true or false, at which point you score one point if you’re correct, and zero if you’re incorrect.

For longtime listeners who are wondering why I’m not punitive anymore, and subtracting points if you get it wrong - because my math skills are terrible, and I got confused last time, and it was rather embarrassing. So we’re gonna go simple, positive scores. We’re going to keep it positive today, and we are going to see who’s the best at detecting BS.

Thomas Steiner: The way working with a clipboard works is very interesting actually, because if you think about clipboard interactions, you can go to the macOS Finder and copy a file and paste it. So then you paste a file. You can go into Photoshop and select everything and copy it, and then you have an image on your clipboard, and you can paste that image file, in the sense of you paste the image data.

If you think “I’m going to a website and selecting everything”, and then you paste it into let’s say Word, and you will see that you have pasted sort of HTML that then magically got translated into whatever word things should be the best representation of this HTML content… So preserving, for example, the semantic information that something is a headline, headline one or so, would be translated into heading level one in Word. But then if you press this magic keyboard shortcut, which is Command+Shift+V on a Mac, you will paste text only. So you’re sort of losing this information, and just pasting the text.

[47:59] This is pretty powerful as a concept, because with a clipboard API, you can do the same thing now. You can put different representations of the same content onto the clipboard. So if you’re building an application that edits SVG files, for example, you can put the image data on the clipboard, but at the same time, you can also put the source code, because it’s XML-based, so it’s all readable; you can put the source code on the clipboard, and then depending on the application context, if you paste into VS Code, you can paste the source code, and if you paste into something Adobe Illustrator, you can paste the image data; you can do the same thing there on the clipboard. And the way it works is super, super-simple. If you look at the code, it’s just essentially creating different blocks of different MIME types, and putting them on to the clipboard.

So that’s my first API that I really, really, really love. The second one is File Handling. So it allows registered applications and progressive web applications, applications, web applications, whatever, to register themselves as a file handler… Which means if you have an example application that deals with .example files, you can say, “Hey, my PWA can become the file handler for .example files.” Which means if people have a .example file in their file explorer, Windows Explorer, or macOS finder, or whatever, they can just double click, and the PWA will open. Very, very powerful concept. And what I like most about it is it feels so natural. So once you’ve gotten used to it, you don’t think anymore, “Is it a .exe file and Word opens? Or is it a .example file and the example app opens?” And if it’s well made, you can, in many cases, not even spot a difference between a native application and a web application. Sort of what we said in the beginning, and we’re sort of back to there… If you build something that is well integrated into the operating system, people think of your applications like apps, and that’s what you want to be at. So that’s a point you want to be at.

People don’t think, or should not have to think about “How is something implemented?” Especially not users. They should just be using your app and be like “Oh, wow, this is so cool for (I don’t know) editing my video files, or adding my .example files”, or whatever.

So I think this is where you want to get at. People should be using your app without really thinking it’s a web app. It’s just something that fulfills their use cases, covers their use cases well.

Jerod Santo: Alight, it is time for Headlies, the award-worthy segment, which did have a theme song, but Kball laughed at it so much that I had to delete the file and never play it again… So no jingle for Headlies, but it is a game - and this is the first time we’re going to have the game with multiple players. It’s always been a one-player game. I played it against Nick, and then I played it against Kball… Here we have three participants.

So here’s how it works… I have scoured the web for various headlines. I have also fabricated some headlines. Each headline also has a description, like the first paragraph from the story… And so I will read a headline, at which point each of you will have an opportunity to guess whether it’s true or false. Okay? If you guess immediately, right after the headline, you get two points on the table. So you can either win or lose two points. You can then opt to hear the description, the first paragraph from the story, and after that description is read, now you can guess true or false, and you get one point or lose one point. So you have more opportunity to score points right after the headline, but you have less information, and then afterwards, less points, but more information.

Each of you will guess separately. So you’re playing against each other, and we tally the points at the end of all the headlines. Whoever has the most points, wins.

Feross Aboukhadijeh: I’ll just second what Tobie said… I think we’ve been super-lucky with the types of attacks that we’ve seen so far. It could be much worse than what we saw in this particular attack. I think we’re starting to see these kinds of supply chain attacks happen on a much more regular basis. It’s sort of picked up in 2021, and it hasn’t really let go since. I see a headline about something like this every two weeks, it seems like; sometimes even more frequently. And if you look at the kind of packages that are being affected, they’re all these pretty headline-grabbing numbers in terms of downloads.

So the two packages in this incident were faker.js, which has ten million downloads a month, and color.js, which has one hundred million downloads a month. But we’ve also seen attacks against UAParser.js and COA, as well as RC… And all of those packages had at least 30 million downloads a month each. When malware makes it into one of these packages, through whatever mechanism - you know, in this case it happened to be the maintainer themselves who added the malware. In the case of the three I just mentioned, UAParser.js, COA and RC - those ones were actually a third-party who got access to the modules and added malware to them.

In all these cases I feel like the attacks have been pretty naive and caught pretty quickly. We’re not really looking at super-sophisticated actors here. In the case of faker and color it was very obvious that something had happened, because the packages were outputting gibberish code, infinite loops, sort of outputting the Unicode text – it was very obvious that something had happened to the package.

And then in the case of the three I mentioned earlier - all of those were pretty obvious as well. There were crypto miners added to the modules, so when you updated, your CPU usage would just go straight to 100% and it would be mining crypto for the attacker. So this isn’t like a super-sophisticated thing; it gets caught within a couple of days. Someone notices, the malware gets removed from npm, and then we write a bunch of new stories about it and people talk about it.

There have been hints at much more scary attacks. If you go back to 2018 with event-stream. The event-stream incident was much more sophisticated. A bit of obfuscated code added to one of Dominic Tarr’s packages when he gave up the package to another maintainer to take over after he was done working on it… And in that case, that was an attack that targeted one very specific company. And it was a no-op for everybody else. And so that went undetected for much longer. I just think it’s interesting that we haven’t seen anything that sophisticated since 2018, which makes me think either people just have forgotten that those things exist, or maybe we’re just not finding them, which is the much scarier thought.

Mat Ryer: Thank you for taking the heat off me, and placing it rightly onto Batman, and that not-a-number thing in JavaScript. They both deserve [unintelligible 00:35:00.17] Okay, let’s talk about ints. There’s a proposal to change an int to be arbitrary precision, which… When you think about that - this is number 19623… That as a headline doesn’t make much sense, but Daniel, perhaps you could explain this one to us.

Adam Stacoviak: Yeah. Or being censored… Because it’s not like we say anything that needs to be censored, but that’s the other issue; on either side of the fence you sit upon, there’s an opportunity for control, I suppose, of the content. I had even heard about Spotify, when Joe Rogan brought over some of his stuff… Some of the shows didn’t make it over, the most controversial. I didn’t read more into it, but that’s a headline I saw… So I assume “Hey, I’m a headline-driven person. That’s true.”

Adam Stacoviak: Do you actually say anything, like a headline is a suggestion from them, and it is in your hands at the end? What they write may actually become the headline, but it’s actually just a suggestion, and it’s more like–

Adam Stacoviak: And by controversial, it’s like, controversial to you. It’ll just fuel your fire, stoke your fire even further, and create or inherit a bias… Which, you know, we all can read about AI training biases, and so-and-so abandoning it, Amazon banned some sort of AI bias thing, recently I saw a headline…