Jerod Santo: I appreciate you joining us. So I came across your work, as one does, on the internet, by following a link from somebody else, in some other place… And I landed on Grid World. And I immediately thought “We’ve got to get this guy on JS Party, because this is so cool.” And let’s talk about it. But before we start there, in reading this Grid World piece, which is kind of a - what’s the word? Not literature… An essay? It’s a –

Nick Nisi: I know… I’m not sure if they had grid support. I don’t think they did. But that’s the level of progression I got to, was there. And now I’m trying to learn grid on top of that. But right, now everything looks like a flex to me, and I could just do that… And I feel the same way right now with container queries. I see a lot of excitement around that, but then I look and I’m like, “Well, I can’t use it because Firefox, so I’m gonna ignore this for a year, or whenever it comes out for that, and then I’ll look into it again.”

Divya Sasidharan: I think subgrid is super-cool. It’s really neat, and it shipped in December, so it’s pretty recent… But I think [unintelligible 00:49:19.12] only available in Firefox at the moment, so hopefully other browsers implement it. Because I’ve had the problem of trying to hack a grid within a grid, which is really annoying; it doesn’t work.

William Falcon: Yeah, so today Grid doesn’t support deployments. The thing that we like to focus on is making sure that we really nail certain experiences before moving on to other things. So we will support deployment at some point, probably very soon. But the thing is like, I don’t think that we’re fully optimal on the training side yet. I think we wanna provide a really world-class experience there.

So for our users today - you can now access artifacts, you can get model checkpoints and all that stuff… So the deployment – most users have a deployment system in-house already, so they can just take the artifacts and do their thing. So we’re not blocking any of that. And all of these things are URL-based, and if it’s Lightning - that’s very easy to do.

Now, we’re gonna make it a lot easier, for sure - kind of the way that we do things - but today we are laser-focused on training. But I will say, I think working with Grid at this stage is great, because I think companies will be able to help us influence that roadmap, and help us build something that they really care about as well… Because as soon as we start getting to deployment, we’re gonna do it our way, and we have a very special way of doing things, so we hope that we have the feedback from the community and users to make sure that we’re doing it in a really useful way.

Kevin Ball: One of the distinctions that I’ve heard that I really liked was that Flexbox, from the word “flex” - it’s about distributing things in space in a way that you want them to be controlled by what’s in the content, or what’s in the elements itself, let the elements sort of figure out together how to distribute themselves… Whereas Grid is much more command and control; so if you’re trying to say “I want these things to be in columns”, or “I want these things to take exactly this space”, Grid is much more intended for that. And you can kind of, as you say, shoehorn Flexbox into more exact things, but that’s not what it’s intended for really, and that’s not what it’s best at.

Christopher Hiller: There’s really not a whole lot I can say, except that I was working on an app, I was just kind of playing around, trying to get something working with Node SerialPort in Electron… And I wanted to try a framework and didn’t wanna get into a huge toolchain, like with Create React App, or – Vue’s toolchain is sizeable… All these boilerplates drive me nuts, but that’s another thing. So I didn’t wanna use the boilerplate. I found something very simple with Svelte - it’s basically you have like a roll-up config, and you run the thing and it compiles the thing… And you look at your code, it is – it’s kind of like Vue, where you have this (I think it’s a) .svelte file, which is kind of analogous to a .vue file… Where you have your script, you have your styles, you have your markup in there… It is so straightforward and simple and elegant, and just like “Wow! How easy can this be?”

My brain melted a little bit, just because it was far and away the most straightforward and easy to understand framework that I’ve used in a long, long time. I think definitely people should take a closer look at Svelte, from that end of things, the architecture of the project.

[27:58] What’s different about Svelte, as far as I understand, is essentially it has no runtime; it’s just a compiler. And you write this code, that follows these conventions, and it poops out a bundle, and markup, and whatever it needs to do, and it runs. And the bundle sizes are very small. There’s not a lot of startup overhead, because it doesn’t have to bootstrap all this stuff… It’s just kind of incredible, and a great idea, and I feel like frameworks are going to start moving in that direction, where they get rid of this whole overhead of loading a runtime in your browser, or what have you, and it’s just simply compiling down to minimal vanilla JavaScript.

But because of that architecture they could optimize for - well, what’s the simplest straightforward thing we can do? Let’s look at what’s been successful in the past. People talk about how easy Vue is to understand for somebody who’s really new to JavaScript or web development. They went with that idea, I think, and just kind of “Let’s make this even easier, and just simple and straightforward.

I was really impressed with Svelte. I don’t really build too many web apps. If I do, they are of the hobby variety… But I’m definitely going to take a look at it, and I’m gonna keep working with it on this particular project, because… Yeah, it’s awesome.

Mikeal Rogers: I read this interesting book - I’m halfway through it - about habits and about how to form habits and how to get rid of bad habits… And as you’re reading it, it’s so obvious; all of the things that they’re saying are so obvious. But also, whenever I set out to do something new, I never do any of this stuff, and usually fail at keeping up with that.

But trying to structure things so that you have to do something, or you can’t do something that you don’t want to be doing… Like, if you don’t have junk food in your house, you will not eat it… [laughs] That is a very basic thing that you should probably do, if that is a habit that you wanna have.

Another one that I really like is called habit stacking. You take habits that you already do regularly and you start layering other things on top of them, so that they end up being the thing that pre-empts you to do the next thing. That’s been really cool.

Brett Cannon: Yeah, there’s a slight story behind that. Back when I was in undergrad I wrote a program for myself to measure the amount of time I did data entry for my father’s business. I was on Windows at the time, and there wasn’t a function called Time.Strptime, which I needed to parse dates to get back the time tuple representation. I figured out a way to do it, and I gave it to what’s called the Python Cookbook, because it just seemed like something other people; I was just trying to be helpful.

[08:18] Subsequently, I had to figure out how to make it work, because – well, I didn’t really have to… Basically, it bothered me that you had to input all the locale information. I figured out how to do it over the subsequent months, and actually, as a graduation gift for my undergrad, the week following I solved it and wrote it all out, and I asked Alex Martelli, the editor of the Python Cookbook, which had published my original recipe, “How do I get this into Python? I think it might help others.” He said, “Well, email this mailing list, pythondev, and they’ll explain it.” I did, and they were very helpful, and just walked me through it on how to get a patch sent in to the mailing list, to get that all merged in, and I did.

This happened to coincide with a year off I was taking between my undergrad and going to grad school, because I have a bachelor’s in philosophy, but I wanted to go to grad school in computer science… So I was taking more or less a gap year to build a programming portfolio to prove to grad programs that “Hey, I do know how to program, regardless of what my degree is in.” I realized that open source provided that potential opportunity by being able to say “Look, I’ve contributed to these projects.”

I happened to get involved with Python early - it was actually in June of 2002 - and I quickly really latched onto the group and how they function, the people and all that… And I ended up basically spending a good chunk of that year off just participating more and more. I started to write the Python Dev Summaries, which were a bi-weekly/semi-monthly summary of what was going on on the development list, because obviously, email volume, and a lot of people couldn’t keep up. That led to me noticing the little issues that needed to be fixed, so I was able to then jump in, learn what was going on - because I had the time - submit patches, get those merged… And I just kept doing it to the point where I said, “Hey, can fix this. I just need someone to merge it”, and then Guido just went “Oh… Well, can’t you just merge it yourself?” I was like, “No, I don’t have commit rights.” He was like, “Oh, okay. Someone will give them to you… You just merge them yourself.” And magically, I just had commit rights.

Gergely Orosz: I’m keeping tabs on it as much as I can… I’m not hearing too much. What I am hearing is Facebook has started to hire a lot. They’re the only company who’s really ticked up their hiring. Other companies are also – hiring just seems to be quietly coming back. I think I talked with some people Google where they’re getting headcount. Amazon hiring is back, even though Amazon also fired a lot of people with the return to office…

It feels to me that there’s no massive hiring, but there’s now – I’m not hearing places that don’t do backfills. And these companies, typically large tech companies, in a normal year, they would lose 10% of their headcount, or 8%, in terms of people leaving and backfilling. It seems this is a bit lower right now… But still, for these companies to hire even 5% of a 100,000 person company, or let’s say there’s 50,000 people, software engineers working at a company like Microsoft - I’m just guessing - 5% of that would be… It’s in the thousands. So I hear a lot of backfilling and open positions for more experienced engineers, to areas that seem to be hiring a lot less. Engineering leadership, engineering management… Most companies are reducing management layers, so there’s less space for senior managers, directors, and I’m hearing a lot of directors, people who were managing managers, sometimes going back to managing teams, so taking a step down… And new grads; new grad hiring seems to be very much down at a lot of these companies.

I’ll have [unintelligible 00:43:32.24] hiring will be back in big tech. Last year, at least in Silicon Valley, most big tech did not hire interns. Roblox did, for example. I spoke to a friend at Roblox, and this friend told me something really interesting. First of all, they were pretty surprised because they got all the graphs to choose from. Usually Facebook takes the best grads, and Google, and they’re kind of left fighting for the rest… So this year they could just choose – sorry, last year in 2022 they could choose from all of them.

And then this manager told me something interesting. They hired these grads in the spring, they were working with them, and he asked one of them “Hey, how are you doing? What are your friends up to?”, and this intern said “Well, a lot of my friends who didn’t get the internships they were hoping for, they didn’t get good offers, some of them just went back to grad school for another two years, and they’re hoping the market will get better.” And there’s been a few people, apparently - this is Silicon Valley - who have just changed from software. They just took to a different place.

And another really interesting thing that I’ve heard, that I think is very specific to the US, is - one of my friends who’s a manager at one of these late-stage startups, her husband is a university professor somewhere in California. And he told me that during the lockdowns, there’s been a class that was a pass/fail. Instead of getting grades, they were just pass/fail. And what he noticed is these classes are a lot worse in terms of capability. So now he’s taking some of these classes that last year there was only pass/fail, no grades, and apparently he needs to dumb down the assignments, because they just don’t have the skills. And she’s also hiring some of these grads, and she’s saying that this whole pass/fail year, which is like one or two years, is just not where it is before or after. So she was like “Oh, I think we’ll have a problem. These graduates may be – for no fault of their own, they’re not going to be as competitive.” She said they can probably catch up, but this is just so interesting on how something seemingly so small - it’s COVID, let’s make it a bit easier, pass/fail… And on the workforce, managers notice, professors notice. They’re like “They’re not working as hard. They’re kind of more [unintelligible 00:45:37.26]

David Sweet: Yeah, so I was in finance for a long time. My next step after that was to go and try to build my own trading strategy… And it was at a time when day trading and intraday trading was something lots of people were doing by hand very successfully. It was right after the Nasdaq/dotcom boom. So I partnered up with a buddy from my research group in grad school, and we just started building.

We went on the web and went to the websites for the exchanges and just figured out how to do this, and built a small, but working trading strategy. It was super-exciting, in part because we got to build our own strategy, but in part also because we were building our own company at the same time, for the two of us.

It was interesting that it was a totally different kind of experience from any one I’d had up to that point. Up to that point I’d always – and I think a lot of people have this experience going through school… You rely on your practice to give you confidence that you can do a particular thing that you’ve learned. When you go build a business, you have to kind of build this meta confidence that you can figure out how to do new things… Which is nerve-wrecking, but super-exhilarating. It was a lot of fun.

Evan Sparks: Yeah, it definitely was a very common career path, and it’s funny - probably a few years later everybody then went into ad tech, or something like that, and now it’s probably autonomous vehicles, or something. There’s always an interesting corner and a hot area to be doing this stuff, which is one of the things that I find super-fascinating.

So after a few years of that quant finance thing, I found that other people really liked looking at P/E ratios all day, and that wasn’t for me. I was much more interested in the technology problems we were solving. I ended up going to work for a startup in the NLP space called Recorded Future. We were taking the web and throwing it through this massive NLP engine, and building structured data products based on it… And trying to figure out how we sell that structured data to places like trading firms, but also the federal government, and so on. Ultimately, that company found a good niche in threat intelligence, basically trying to build predictive indicators of where cyber attacks are gonna happen, and so on, but again, with this same data-driven machine learning technology.

In many ways, the roles were pretty similar… One being in financial services, but the other being in this totally different startup environment. But always building models and driving forward data products. In both cases though I found I was spending much more time building and maintaining my own infrastructure than I was worrying about the modeling problems. And it was really the case in those days - this is around 2010-2012 - as Hadoop was becoming popular and so on, where as soon as I was tasked with analyzing a dataset that didn’t fit in the memory on my laptop, my world just collapsed. You were forced to figure out how to write MapReduce jobs and so on. I took that as kind of a good signal to go back and invent the world that I wanted to live in in grad school. So I had the good fortune to join the AMPLab at UC Berkeley right around the time that Apache Spark was born, and my co-founder at Determined AI,

Ameet Talwalkar and I got to work on it right away, building out the machine learning ecosystem around Spark. So we were among the designers and initial contributors to MLlib, which is the standard library for machine learning in that ecosystem.

The rest of my Ph.D. was really focused on “How do we give people tools to build, and machine learning applications and optimize them in a large scale, distributed fashion?”

Heather Meeker: If you’ll forgive a small foray into the technology, when you build a program it’s not just one big, amorphous blob, at least not usually. You build it in pieces and you kind of stitch it together. That process, at least in a language like C or C++. It’s called linking - you take all the little blobs and you stitch them together, and that’s called linking. There are at least two ways to do that: one is dynamic and the other is static.

The static way is where you tell the program that’s building your program, “Just put it all together and smash it into one binary.” That’s static linking. Once it’s put together it doesn’t change, in the sense that all the blobs are sticking together in exactly the same way, and that fact doesn’t change.

Dynamic linking is something a little more sophisticated where one of your blobs, say, might do something that you don’t wanna be doing all the time, like printing a file. So you tell the builder that is building your program, “Build the program, but leave the printer blob on the side. When you need it, go get it, and execute it, and then flush it out of memory.” It’s a way of conserving memory at an expense of operating speed. Now, you can tell I’m an old programmer because I don’t even know if programmers think as much about that anymore, but we used to be always trading speed against memory usage. It’s one of the reasons why you have dynamic linking. You don’t wanna be stuffing your memory full of code that you’re never using, or rarely using. It’s much more efficient to say, “Okay, when you need that thing, go and get it, execute it and get rid of it.” That’s dynamic linking.

To make a long story short, GPL is for whole programs, and if you put the program together as one big blob or a bunch of blobs, it doesn’t really matter from the point of view of GPL compliance, whereas a license like LGPL allows you to use a dynamically-linked library and bifurcate the licensing. [27:58] A proprietary program can use an LGPL library as a dynamically-linked library. I am greatly simplifying the rules there, but that’s a sort of basic compliance rule.

So that’s the difference… It’s a difference about how you engineer the build of the program, and LGPL in particular is directed towards libraries that are primarily used as dynamically-linked libraries.

Derek Collison: Yeah. NATS was kicked off as part of a system that I was building at the time at VMware called Cloud Foundry, which was my spin on Heroku for the enterprise back in 2010, I guess. And at the time, I had come off of time at Google, I spent about eight years there, but prior to that was at a company called TIBCO, which before that was called Technicron. And we had been designing lots of messaging systems that then were used to scale up distributed systems for customers and clients. So I just had a natural way of building systems, like command and control, telemetry, eventing, using these types of constructs. But TIBCO was closed source, and so we couldn’t use that.

So we reached out to Alexis Richardson, who was running a company that was kind of in charge of RabbitMQ, which at the time was kind of the major open source player from a messaging construct. And RabbitMQ is still part of, I think, VMware/Broadcom now, part of Cloud Foundry or whatever that has turned into be. But at the time, it was more of an enterprise type of a technology. What I mean by that is that if you asked it to do something, it would bend over backwards trying to do it for you, even at the detriment of everyone else. And we had some challenges there, which led to the creation of NATS, which was kind of like a dial tone, and think of it like the electricity in your house, or your condo, or whatever. I can plug in a really bad blender and I might blow out something in my own place, but hopefully it doesn’t take out the electric grid for the whole neighborhood. And so NATS was born on a very simple principle of “Just do a couple of things very, very well.” And yeah, it was the proverbial weekend project. So I left on a Friday, and Monday we plugged it in.

[08:35] And believe it or not, applications that were written for that initial version, the protocol that speaks between the clients and servers still run today. So you can just fire up something from the Cloud Foundry days and just run it against a modern NATS server and it just works. So we’ve tried very, very hard - and we’re not perfect by any stretch, but we tried really, really hard to make sure we were always backward-compatible with the protocol. Because we knew the protocol, even though it was designed very, very quickly, in only a couple of days, that was kind of the really big thing from a technology standpoint there.

Mike Lewis: The reality is that when you are solutioning, it is often a barrier for the client to visualize a solution, because we tend to map sort of a biomorphic representation of the process onto the problem and the solution. And we’re trying to imagine a robot doing a human’s job, and just instantly you give it arms and legs and a head, right? So for instance, if I were to say “Daniel, how long do you think it’ll be before robots replace cashiers at the grocery store?” So we have Kroger in Cincinnati. So at Kroger, how long? And instantly, where does your mind go? It’s kind of like “Well, I’m just trying to imagine, okay, the Tesla Optimus bot”, or one of these, just trying to fumble around with your delicate groceries, and not crushing it, and scanning it… And it just seems so hard to think through and pull off. And you would just map that so far out into the future. And what you don’t know is that my little brother, who works with the robots for Kroger, look nothing like what has popped into your head. They’re like these little square garbage can-looking things that run around on tracks; it’s like they really operate on the z-axis, so it’s like they’re stacked on top of each other, and these grid systems, and they’re dropping products from one to another… And in the end, in the bottom, a bag of someone’s order comes out and it loads right onto a truck, and a human doesn’t touch it.

And this is an example of how it’s just really hard, when you’re staring at a problem, trying to think of a solution, it’s really hard to think way upstream. But the reality is the solutions tend to live up there. We start so late. We often start at letter Z, when like we should be starting with “What’s an alphabet?”

Chris Shank: [00:09:49.23] The component paradigm has gotten us really far. It really enabled us to – like, if you go back to 2012, 2015, whenever that idea started and started to spread, it helped us build more complicated user interfaces on the web. There’s no doubt about that. And I think because of that, we’ve hit this wall… And part of that wall is the limiting factor of a lot of these web applications that we see today, I’d argue are not complexity at the rendering level. They’re complexity at the behavioral level. One of the things that I think we’re really lacking is a set of “Here’s all the different types of web applications you can make.” The ones that I’m thinking about are partly what I worked on; we’re streaming real-time market data, we’re contextualizing new stories that are also streaming in real time with that market data… There’s an endless amount of APIs and data that you want to integrate. Outside of FinTech you have real-time spatial canvases, multiplayer experiences… The type of applications that we’re trying to build today are a lot more complicated than they were 10 years ago.

I’ve had anecdotally experiences of “What the hell is happening in this application? Why is this data coming in wrong? Why are our streaming services being canceled and restarted?” And then there’s also this other aspect of - the UI is staying the same, and all of the bugs and the additions that we’re adding to this application are behavioral. It’s just “Oh, we’ve got to multiply X, convert this currency”, or whatever. And that doesn’t happen in the UI, or how we render things. It’s happening in the behavior, 20 yards away. And I think we really lack primitives and approaches to understanding applications better. We really struggle to see how “Oh, a user interacted with something, and there was a click event. What is the application doing?” And part of that stems from this tension of co-location and components sort of being really encapsulated things. Though Reason React aggressively wants you to co-locate things is “Oh, it helps you think reason about the application.” It’s “This button or this table is just its own thing”, and you just need to think about state inside of that thing. And it’s like, well, what happens when this table is talking with the drawer, and affecting other parts of the application? The behavior that we want to co-locate is getting hoisted up and up and up. And then it’s like, well, when now we don’t have co-location, we’re just passing all this data down. And so when you want to say “Oh, this cell in a grid was clicked”, it’s not just the side effects and the thing that the user interface needs to do in those cases is not encapsulated to the table.

Adam Stacoviak: And so GranTurismo, the way the game came about was to be as close to a video game simulator of real car racing. And the way that they photographed, or – I don’t know, they had like this laser grid to capture all the cars, and the sound. This is, I’m assuming, what you’ve done, Adam, which is you’ve gone into the immersive details as a creative director to say “How can we truly capture the essence? Not just what we think is the essence of this theater, but truly the essence?” The feel of the seats, which you can’t feel in Vision Pro yet.

Adam Stacoviak: Yeah, for sure. Can we talk about some subtle human leveling up that’s practical for ChatGPT? I mean, I know it’s not Cody. Do you mind riffing a little bit? So last night my wife and I, we were hanging up pictures of our beautiful children. We took pictures of them when they were less than one week old, and then we have pictures of them in the same kind of frame at like their current ages, and one’s seven and one’s three. So it doesn’t really matter about the age. They’re just not one week old anymore. So you have this sort of brand new version of them, and then like current version, to some degree. And it’s four pictures, because we have two sons, and we want to hang them on the wall. And my wife was trying to do the math - and we can obviously do math; it’s not much. It’s like an eight foot wide wall; we want to put them in a [unintelligible 00:15:29.05] with even spacing, all that good stuff. I’m like “Babe, we should ask–” And like, I’m more versed in this than she is; not so much that she doesn’t use it often, she just doesn’t think to. And I think that might be the case of why it’s not being more widely used, is they don’t think to use this. And I’m like “I don’t want to use it. It’s a word calculator.” Or “I want to use it, it’s a word calculator.” I don’t want to think about this problem myself. I don’t want to do the math. I could just tell it the problem; my space, my requirements, and it will just – it will tell me probably too much, but it will give me a pretty accurate answer. I’m like “Let’s just try it”, and she’s like “Okay.” And so I tell it “Hey, ChatGPT, I have this eight foot, five inch wall in width, and I want to have these pictures laid out in a grid. They’re 26 inches squared, 26 wide, 26 tall, and I want to have them evenly distributed on this wall in a four grid.”

It gave me the exact answer, told me exactly where to put them at. We did it in five minutes, rather than like doing the math, and making a template, and writing all these things on the wall. It was so easy, because it gave us the exact right answer. That’s cool.

Nick Nisi: [27:51] Yeah, no, that’s great. And totally right. And I think that there’s definitely a use case where you could bring this in and experiment with it. And it’s relatively low-effort, I guess, for lack of a better word. You can get going pretty easy. And could you kind of start small, and scale, and maybe see – I could immediately see me wanting to use this on something like a data grid or something, where I just have a ton of React nodes that get rendered, and you’re trying to eke out as much performance on that as you possibly can. So could you use this as a “I’m just gonna start there and see. I’m trying to render a million rows or something, and see how it goes from there.”

Kevin Ball: And it says “Baseline - widely supported.” And actually, I do think it’s kind of nice to have that front and center, at the top… Especially if I’m like – I see that now before I’m learning about it, so if I were to come in to check out a new feature, and I’m excited about it, I read about it on a blog post and I’m like “Alright, I’m gonna go check this out on MDN.” And before, I’m reading through it, and I’m like “Yeah, this looks cool. I’m gonna try it. I’m gonna try it. Okay.” And then I get down to the bottom and it’s like “Oh, sorry, this isn’t supported anywhere except Chrome, or anywhere except Firefox”, or whatever. And I’m like “Oh, well, I just spent all this time learning about it, but I can’t actually use it for anything interesting, because nobody supports it.” And here it is, right at the top… And they could have done that by putting the compatibility grid up there… But this is very tight, it doesn’t take away as much. But it maybe catches me before I’ve gotten through that investment cycle.

Lars Wikman: Yeah. That’s generally what I aim for. Some people really, really get excited about trying to solve problems at scale. I really, really don’t like what I see of systems at scale. All in all, it tends to be sort of a big challenge of making layers upon layers upon layers of people, and tech, and bureaucracy interact in a somewhat useful way. And there are certain things you cannot do at a small scale.

For example, the post office system, or the power grid - there has to be large-scale coordination in place, and then there also needs to be a lot of smaller systems that play nicely within that larger one. But I’m not interested in solving like a 100-engineer or a 200-engineer problem, in general. I like small teams, I like small organizations, and I trust smaller organizations more. There’s a lot of idealism in what I do, and I also optimize for my own enjoyment, which is why I’m not at like a FAANG, or whatever. I don’t think I could be bothered to pass those interviews anyway, but… I’m looking for things that I think can work at a particular scale.

And sometimes small teams can run large things. For example, WhatsApp is a pretty good example of that. Now, I bet they had a lot of orchestration going on, because they had to, because they were at an immense scale. But they also did a lot of things that are not commonly done. For example, [unintelligible 00:29:29.23] code updates all the time. So I think there are atypical ways of doing almost anything, and you can make it work, and you can probably make it efficient. And I don’t think you can find sort of a competitive advantage compared to other more general-purpose organizations. Choosing Kubernetes today is probably not a competitive advantage, because it’s so common. Doing well with Kubernetes, and sort of having a good org, and a good team, and all of that - that’s a competitive advantage compared to companies that are doing Kubernetes poorly. But almost everyone is doing Kubernetes, so I guess there’s no advantage to be found there.

[30:12] It’s a little bit like – I consider Elixir a competitive advantage for many companies, or a potential, at least, competitive advantage, compared to, for example, all the companies that run Java. You cannot win a competitive advantage by choosing Java, because that is not an outlier; it has no opinion, it is the most general choice. So there’s not a lot of advantage you can glean there. But if you go sort of off the beaten path a bit, either because you go sort of “Oh, we’re going to own all these details ourselves”, or “We’re just not going to bother doing half the work that everyone else considers critical.” There’s advantages to be found there.

For example, Apple likes to ship half-finished features and services… “No, no. We just removed a lot of buttons, and it’s so simple, and so straightforward.” Yes, but you could also add some options, so it’s more flexible. But they don’t. And I think that’s part of their plan for sort of shipping more things, even though they’re an incredibly large organization.

If you make decisions, and you sort of take chances and go in particular directions, I think that’s where you can find interesting things. I don’t have what I consider a complete plan for my operations. I don’t have all the tools figured out that I would like to. I’ve got some recommendations for some nice tools for sort of picking up WebHooks and just running commands off of it… It was like “Oh, this is written in Go. It’s going to be one binary. I could set it up with systemd, and it would run there.” That could probably be what picks up my final artifact from CICD and puts it on there, and then I have some scripts then to manage the deployment. But I don’t have a final idea that I’m like “This. This is how it has to be done.” And right now, often it’s done manually, for my personal needs, because that’s good enough. I script the most annoying parts.

Kevin Powell: If you like video-based content, which I know some people really like, I do think I have a lot, and I cover a lot of material on my YouTube channel, so it’s a good place to start. My written content is very lacking at this point. I started writing and I never really kept up with it. So if you’re more into written content, there’s a lot – I mean, between CSS-Tricks, Smashing Magazine, Josh Comeau has amazing content out there… So his blog posts are always really high-quality. There’s Stephanie Eckles, who has moderncss.dev, I believe it is, as well as SmolCSS. So moderncss.dev is more like let’s dive into like topics, and it’s a bit more in-depth on how we can accomplish certain things using modern CSS techniques, things like clamp, min/max functions, the grid auto-flow, and stuff like that… And then SmolCSS.dev (I think), and that’s nice, because it’s just basically like “Here’s a pattern that you can try and use, and here’s the code for it.” So it’s just like little snippets that you can sort of steal. So that could be really useful as well.

[48:01] I do have courses if people don’t mind investing into things like that… But no, there’s so much content that’s out there these days that I guess it depends what you’re trying to learn, where you’re trying to get it from, and what the goals are, and everything there… But yeah.

Jason Carter: Yeah. It’s interesting, because you have to think about it differently if you’re a platform provider of the technology. So if you’re Zoom, you control the entire experience, right? You have a dedicated mobile app development team, dedicated desktop team etc. If you’re a company like Vex, you can sort of provide components for people that, “Hey, if you use this on your application, then you will get a really great preview page, and the devices are going to work really well no matter what platform you’re on.” And that’s very challenging.

[34:24] I’m always trying to join our kind of internal demo from any device that I can, just to see, “Hey, how does it work on this?” I just got the folding Samsung smartphone, and I’ve been having a lot of fun playing with that… And that trips up the site quite a lot. People aren’t designing for that sort of a system.

So the way that I see that it could work is providing super-great tools, whether that’s SDKs that work on lots of different platforms, with just sort of like training wheels and safety built-in, where “Hey, if you want to get started and build a better application, just clone this open source repo that has examples of how it works.” You could just run that yourself, if you want; toss in an API key, and you’re off to the races. Or “Hey, here’s a really good audio picker. And we know that it’s cross-platform, we know that it works well… Just take that if all you need is an audio picker”, versus “Hey, here’s a video grid that works super-well across all sorts of systems.” And it’s kind of a moving target in the browser as well, because there are constant updates and changes to WebRTC, right? Safari did not support it for quite some time, until, if I recall correctly, Apple added FaceTime to the web. And then all of a sudden, they fixed up their WebRTC support so they could support their own product.

So long story short, it’s challenging, but I think the best way you can do it is make it so that it’s much simpler to provide those experiences, so that if someone is kind of using Vex as we hope they would, with our components, then they’ll have a good time, and hopefully, that spreads out to more and more applications.

Brendan Burns: Yeah, I think one of the things – Ganesh mentioned the upstream team, which is another team in my organization that focuses on engagement with the Upstream open source project… And I think in order to do a good job of both understanding how releases happen, and also potentially influence how releases happen, we have to be engaged. And we’ve had members of my team be the release leads for the open source project; not for AKS, but for the whole Kubernetes open source project. It’s a totally thankless job effectively, of like herding all of the cats of this giant project into a release… But that means that we have an intimate understanding of not just what each release looks like, but also how the broader release is evolving. And recently there was a slowdown from four releases a year to three releases a year… Effectively a reaction to the broader community saying like, “Oh my gosh, we cannot keep up with this pace of change.”

I think the developer community as well, the internal Kubernetes developer community as well sort saying “We need to slow down. We can’t just keep jamming more and more code into this thing.” But I think the real difference that I see in releasing Kubernetes versus releasing it for AKS is exactly what Ganesh is talking about, which is… You know, for AKS a lot of what “at scale” means, or at hyperscale means, is incredibly diverse customer workloads… From large-scale machine learning batch jobs, all the way through to real-time serving telephony, even like teams calls. And the upgrade has to work for every single one of them. The upgraded Kubernetes has to work for every single one of them. And it’s not even just about the workload, sometimes it’s also about like what API features did they decide to use?

[42:08] And one thing we learned early on in the Kubernetes project is no matter how much you call it beta, if it’s stuck around for two or three years, you may as well call it GA, because people will have treated it like it’s GA, and you will have set the expectation, because it hasn’t changed… And the minute you change it, it causes amazing ripple effects. And frankly, you can’t – once you have a certain number of users, you don’t have the option of saying like, “Well, but we said it was beta, and you’re all broken. Good luck.” That doesn’t fly in AKS really, at a certain scale, because it’s the principle of least surprise, I guess, at some level. Like, if you haven’t touched it in two years, people are going to assume that it’s stable, because it was stable.

So I think that’s the real distinction that is important for all of the Kubernetes providers, especially for Azure, because that’s the one I worry about is “How do we get that rock-solid reliability so that when the person presses the button, or when the Event Grid that Ganesh was talking about triggers, and someone automatically upgrades, it works?” And then tracking also. We keep track of the SLO for that upgrade, to make sure that we actually are validating it, and that we are achieving it. And sometimes that involves actually going back into the release and finding fixes, and Ganesh mentioned, carrying patches to help while you’re upstreaming those patches, and things like that… As well as, of course, something that Ganesh didn’t mention, which is making sure that also we handle CVEs, and we get notifications as a provider actually in front of the CVE release, because we’re on the embargo list… And so we can ensure that our customers are patched and secure on day zero of a vulnerability, and that they can either choose to upgrade, or in some cases, they’ll receive an automatic upgrade, kind of depending on the severity of the security issue.

Zac Smith: Yeah, so the TL;DR is that chips are getting hotter. Why are they getting hotter? Mainly, we’re getting dense, the nanometers are getting smaller on the fad processes. That’s how you kind of stuff more transistors in. In order to then do that, you need to push way more power through these things, and we’ve created innovative ways, like what Lisa and team have done at AMD about chiplets, and having lower yield requirements, and putting multiple chips on a single die. But in the end, we’re just running into a physics barrier here. You add it by adding more layers onto it. So suddenly, you’ve got multiple layers in the thin fad or whatever they call it. Even with memory and NVMe. So everything is having denser transistors, with more power going through them, and you have this kind of movement towards the way, as you kind of get rid of the nanometers, your only way to make things go faster and more efficient is to push more power through them.

So that’s one in the general-purpose, large-scale silicon trends that we’re dealing with. And the second thing is we have way more sophisticated purpose-built technology at this point, like GPUs, or accelerators. We have things that are very, very specific at doing one thing very well, and you then keep them busy, so you just use a lot more heat. There’s an electricity problem that we have there, and certainly, as we shift to a more renewable energy footprint, instead of just buying credits and offsets actually generating things like green hydrogen, so you can offset demand and use it, exposing – there was a great panel with the Intel team last week or the week before about how to expose to the world of software reliable metrics on “Well, that would not be a good time for you to reindex all your data stores. Maybe you should do it at noon, in our Texas data center, instead of at 2 AM in our Frankfurt data center, where we don’t have any renewable energy.” We don’t have a way to even express that in our industry, a standardized way, let alone to do something about it. We desperately need that…

[32:20] But anyways, getting back to it - accelerators and purpose-built technology are getting hotter… So you have this electricity thing, more juice into the rack, and denser, effectively… And then you have the other problem, which is cooling. We’re kind of getting to the upper barriers of two things. Number one, we’re getting to the upper barriers of how we can air-cool this stuff. A lot of the times – and you can see simulations, that about 20%-30% of the energy in a data center is just fans. If you ever walked into a data center, they’re very loud. They’re loud because there’s all these little tiny, 20-milimeter fans running at the back of every server, just sucking the air through, just to create airflow on individual computers, to pull it over those chips and those heat sinks.

So in big data centers you’ve got 20%-30% of the energy just using fans to pull air around… And then we’re getting to this density level where you just can’t cool it if there’s not enough air flow to be able to do that… And especially in a mixed data center. In a hyperscale data center is where you can build around one specific thing, you can kind of purpose-build some of the stuff around it, you can (as I like to say) build your data center around your computers… You can’t do that at some place like Equinix, where every enterprise service provider has different things. I also kind of believe that we’re gonna have a future of compute that’s more heterogeneous, versus homogenous… So we’re gonna have a few of a lot of things, versus a lot of one thing. So I kind of think that we have to solve this in a more scalpel-driven manner.

So moving the liquid – I’m not gonna go into all the things, but just think of it like your car radiator or air conditioning. Pulling a liquid that turns into a gas over the hot part, the chip, the plate, whatever, and then being able to do that does a few things. Number one, it can be way more efficient. You can stop all those fans, you can stop pushing air around which doesn’t go in the right place, at the right time, and start to put the right cooling at the right place.

The other thing you can do is create a much, much higher differential between the intake and the output. What that allows is - you’ve probably heard of things like heat pumps. You can actually turn that back into energy. So you’ve got a natural thing called a giant turbine called “thousands of computers creating heat.” That sounds kind of like a power plant to me, right? Right now we literally just exhaust that, we’re just trying to get rid of it. But if you can create a differential and actually capture (I’m gonna call it) hot enough liquid, you can actually turn that back down to energy, or sell it to the grid for municipal purposes, or whatnot. You can use that energy if you can capture it.

And then the most important part of that process is actually today most of our data centers and most of the data centers in the world use evaporative cooling, and that takes millions of gallons of water per day to evaporate this heat. And that is simply not sustainable. So we need to move into a closed system, where we can keep the water and the liquid and not evaporate it all along with it.

So there’s these momentous challenges and opportunities… I think it’s – like what I’ve touched on earlier, related to some of the business model changes are gonna be necessary to that… But as we – like, for example at Equinix we have a goal of reaching Carbon-neutral by 2030 using science targets… We have to explore all of these options with not only ourselves, but our ecosystem partners - the silicon partners, the OEMs, our customers etc.

[35:44] I think one of the biggest challenges we have right now is that in an enterprise data center, with this diversity of technology that’s going on, everything from Dell servers to NVIDIA DGXes, to boxes that you brought in from your – you know, “This is a ten-year-old server I’ve got… Let me bring it into the collo.” Still useful, and actually that’s probably one of the best things you could do, is continue to use that server, so we don’t have to make a new one.

Derek Parker: So in the case of a probe attached to a function or something like that, it’s blocking. For that moment of time, the execution of the program that invoked or triggered the probe will be stopped, so that the eBPF program can do any kind of inspection that it needs… Which is also why, for the context of tracing - this is what got me interested in eBPF, was the combination of the two last subjects we were talking about, which is the overhead of system calls, and just the overhead of tracing in general.

A similar thing would happen – so the reason why I got interested in it was because I wanted to make Delve’s tracing backend more performant, have less overhead, so that maybe you could use it in a production context if you wanted to.

I did a tweet a while back, when I first started working on it, of like the overhead added. I had a program that ran in some odd number of microseconds… And then with eBPF-based tracing on top of it, it went from maybe like 20 microseconds to 300-400 microseconds, or something like that… Which sounds like a significant amount of overhead, but we’re talking microseconds.

And then I timed it using the ptrace-based tracing that Delve had been using prior, and it bumped up to like 2.3 seconds. So you go from microseconds to actual seconds, which is like – you can’t have that kind of overhead if you’re tracing something in production. So that was kind of what got me interested in it, having these really small, targeted programs that can be called as a result of something, that don’t do context-switching between kernel and user space, and stop the program for as little time as possible, and really get kind of performant, detailed, but also ad-hoc tracing.

[16:01] The other difficult thing was making it ad-hoc. So a lot of times when people write eBPF programs, they’re very, very targeted. You already know when you’re writing the function what corresponding function it’s gonna be attached to; what kernel function, or whatever… You generally already know; so you kind of know the arguments to expect, and things like that. But in the case of Delve, and what I was doing with eBPF, I was kind of abusing it a little bit to try to do just – I wanna attach this probe to a completely arbitrary function, where I don’t know how many arguments or return arguments it has. I know nothing about it, but I wanna get all the information out of it. How do I do that?

So that opened up a lot of questions of how to write a generic eBPF program, and how to communicate between that program and Delve, communicate between the kernel space and the user space, in a way that also won’t introduce back the slowness that I was trying to get rid of.

Aaron Parecki: And that was actually one of the big conscious decisions I made when I started tracking my location, which was I didn’t want the fact that I was tracking my location to change where I was going. So at the beginning, think back 2008 - smartphones were brand new, the iPhone was only a couple years old… So that was not really a normal thing that most people had access to at that point. So the big worry was like “Oh, well if you’re tracking where you’re going, aren’t you gonna be concerned about somebody finding out, or concerned about whatever?” So I just tried to make sure that the act of tracking my location was not changing where I was going, as in I wasn’t avoiding places, or I wasn’t even going down other streets in order to complete a city grid, or things like that… Because I wanted it to be passive collection, just about what I do, not trying to treat it as a challenge to visit every street, or something.