Zeno Rocha: [41:55] What is cool about them, and not cool too, is they have to keep improving their game, otherwise their products get obsolete. And especially now with AI, you can generate so many different emails, and they’re highly personalized… And it gets even more tricky. But I think the beauty is – or the challenge for them is “Okay, how do we evolve?” And in the beginning, you’re totally right, Adam, there was a lot of emphasis on the IP level. So then you would have like an IP, and if the reputation of that IP is good, then just let all those emails go through. And then email providers came up and they were like “Okay, now I have this big IP pool, and I just shove people there”, and then the good actors balance the bad actors. So then these inbox providers, they’re like “Oh, okay, so now we have to go up a different abstraction layer and look at the domain more than just the IP.”

And they have different techniques. They look at how fast you send emails, and that’s something that dictates “Are we going to throttle the emails or not?” They look at the engagement early on for like emails that are coming to the inbox, and based on that, they dictate the inbox placement. “Are we going to keep it on the primary box, the promotional tab, the spam folder?” And those things are constantly evolving. But the thing I don’t like is like – I wish they would evolve as fast as the web, for example. Because I feel like the web was super-slow, maybe in like the 2000s, and then 2010 HTML5 comes in, and CSS 3, and ECMAScript 6, and it’s like “Oh, wow there’s so much movement.” And now we don’t care so much about how this website looks on Opera versus Firefox versus i6. It’s just like the same website. Very little things like that are different in Safari than Chrome…

Mat Ryer: It probably could. You could certainly get a slapped, red back. Have you ever jumped into a pool and just landed on your belly on top of the water?

Salvatore Sanfilippo: It’s a lot of people, and the development part is almost completely in Israel. A lot of remote work is inside Israel, in many places of Israel, in the North, in the small cities, small villages… And there are truly talented people there. Incredible persons. Also very loyal persons to the company, that understand the project and understand it is very important. Also, they are people that kind of fight for the AGPL thing, because they thought it was a good thing. So people interested in the project. And I believe it is a very strong pool of talented people in Tel Aviv.

Also, another thing that happened that I believe is extremely cool is that we are acquiring new people that are like that. What happened is that one of my colleagues, Oran, which is truly a genius of programming, together with Yossi - they are two really superstars that nobody knows. For example, Yossi was the one that wrote the abstraction layer in order to make Redis SSL as a module. So you load a module and it is SSL, otherwise you don’t have the complexity, and stuff like that. For example, this Chinese person starting to do pull requests only on comments. This comment, this – and another guy, which is a Portuguese, for example, also doing initially simple pull requests. But he recognized that they were smart, and started to help them to create more complex pull requests. And since Oran works a lot, since he’s responsible of all the developments of Redis on Flash, which is, you know, the way that we have to offload part of the data from RAM to Flash; it’s a complicated fork of Redis. You know, he had to do a true investment, and now these two persons are very strong core contributors of Redis, making complicated stuff.

[00:52:00.19] Also, another guy, which is the one that implemented the expires in the hash type elements, single elements - he’s very talented, and he optimizes this thing to a so low level. And for example, one thing that I still love about Redis, and I think that we have an edge here compared to, for example, Valkey, is the design idea. In Valkey there are a bunch of people and they do a lot of agreement design, which is a way of working that I don’t believe will be optimal. And why I hope that Valkey is going to be great, because in some way this was – you know, the thing about the BSD is that whatever happens to the company, to myself, the code in some way can go forward, because it takes other streets, like Amazon, or Google forking, [unintelligible 00:53:00.11] However, from the point of view of the software idea, I don’t think that Valkey can do a better job than us. And one thing that I would love is we go back in to AGPL to be also OSI-approved it again, and then compete on the quality of the ideas, and the quality of the developments… You know, because Redis is going to diverge a lot.

For example, one of the leaders of the project said, after seeing in my blog post the vector sets thing, “We will never do something like that”, because they don’t trust in this line of, you know, AI is this terrible hype. For example, this is one way to approach the problem. I am, for example, an AI enthusiast, and this already creates a different take on the project.

Gerhard Lazu: Well, I’m working towards that… But I can show you one more thing, which was not planned… I know we’re a bit over time, but if you want to see one more thing, I’ll show you my M25. Every year I basically take one of these machines online… And I have like a 24, 25… So this year, this is the machine that came online. It’s running TrueNAS; as you can see, it’s an i9-990K. It has 128 gigs of RAM. So it’s like fully maxed out, basically… And this is how… I’ll just move that screen a little bit. You can see all the storage, it has two pools… It has an SSD pool and an HDD pool, so spinning disks, and some slower SSDs. They’re the Evo, I think, 870. And it’s something that you need to have in place to be able to have decent storage, between Linux, and Windows, and Mac, and everything to just work. So that was one of the projects, and I didn’t have time to talk about it. Maybe next time. I know that you are a TrueNAS user, Adam… ZFS, and like all that stuff.

Ashley Jeffs: If you’re like an actual user, there’s a lot there that’s obviously very, very positive. It’s just that the immediate aftermath was difficult, because we didn’t have anything to show at that point. We were just telling people “This is an acquisition.” So there’s no benefits to show them at that stage other than “Hey, isn’t it great that there’s shareholder value?” And we’re telling them that all these things are changing, “We’ve done a rug pool on all these licenses, get over it!” It’s just difficult, but it was fine.

I did a live stream, either the day of or the day after, just to like talk people through what was actually changing to the repo. And to be honest, most people are either neutral - they don’t really care; as long as the software still exists, they don’t really care about any of this stuff, because they just use it for work, because they’re a normal person who has a job. Or there were people who were happy about the fact that there’s a serious mission behind this now, this sustaining, in theory, rather than just me doing it… Like, people used to complain about the bus factor before the acquisition, which is perfectly valid. So there’s that now they no longer have to worry about.

[01:16:11.02] I mean, I wasn’t disappointed, but I obviously understood that there was a bit of a backlash. To be honest, I kind of enjoyed the Hacker News drama, because I don’t get opportunities to be on Hacker News much… So it was nice to just be in there. “Oh, it’s Ash here. I’m the maintainer, and this is my thoughts… And I know I’m going to go straight to the top of the comments, because I’m the person being talked about.” It’s like free attention. Like, you get attention and you’re like “I don’t have to do anything.”

Jerod Santo: And privacy too, which those two are, you know, kind of eternally linked, aren’t they now? Because being able to throw a smart rock into a river and monitor things is awesome. But being able to throw it into my pool and monitor my swimming, maybe not so much. Just for instance, you know?

Kent Quirk: And who you can hire. Your available talent pool, which includes who and who your talent pool knows, because that, especially early in the life of a startup… You know, your collection of people that you hang with makes a difference. And so yeah, Python might well be the right choice for you, because those are the people you’ve been working with and growing up with, and so you deal with those problems later. You’ve got to survive first.

So yeah, I think there are languages you can choose that will make your life easier, there are languages you can choose that will make your life harder… It also depends on what industry you work in… If you’re working in finance, you’re going to be able to choose a different language than if you’re working in software as a service, than if you’re working in games. So yeah, all of those things…

Nick Sweeting: Yeah. So no matter what, I’m going to have to be both. There has to be a nonprofit component, there has to be a for-profit component. It’s going to be a sort of peer corporate structure relationship, similar to any company that does massive content re-hosting, like Archive.org, like OpenAI, like Mozilla, like MAPS… Basically, you have a nonprofit and you have LLC’s underneath it that do anything relating to money. The content is only ever hosted by the nonprofit, which is not earning revenue for it, but you can sell software that people use, that contributes to that pool of content.

And so the financial motivation to – basically, the financial motivations are kept separate. You’re not incentivized to profit off of the copyrighted material, which I think is important… Because as this eventually grows beyond just me, I don’t want to have sort of corporate structuring that is pushing it in the direction of destroying copyright.

Juan Andrés Guerrero-Saade: Yeah. I mean, it’s super-unpopular. It’s not that I agree with it, but I’m saying - like, if you just steal a couple of ideas, and then the laziest fricking programming language that most people use to just kind of script their way out of a paper bag, suddenly has a couple of like the nice features of Go, and you go “I’ll just stick around here until I’m ready to wade into the pool of being an actual good developer. I’m gonna just be cool over here”, right?

Jerod Santo: Well, we saw a lot of support behind OpenTofu in light of the HashiCorp relicense of Terraform. I’m not sure if that support has continued. We don’t really swim in that pool very often. And so is OpenTofu actually getting a foothold, or it’s going to become the new Terraform, or is it not? We’ll probably find out at this year’s KubeCon or something, but… That’s one where there was a community fork that came out immediately… Versus what happened with Elasticsearch, which was Amazon created the fork, OpenSearch.

Johnny Boursiquot: Yeah. Just to quickly recap, basically trying to get teams to not only focus just on their service and just their stuff, because their stuff is part of a bigger pool of stuffs, that is perhaps causing the reason for the page. So if we only have just teams caring about just their dashboard and nothing else, I think that’s a missed opportunity.

Jerod Santo: There’s an analogue to this in economics. Benedict Evans talks about the process of bundling and unbundling, and he says in any given industry you’re either in a bundling process or an unbundling process… And it’s cyclical, right? So an example of that is television, where we’re all cable TV, everything was bundled as one, and then we broke out of that individual, on-demand, subscribe to this, that and the other thing, and how we’re like in a rebundling that’s happening; you can see it with YouTube TV, and different aggregators trying to pool together content. And that sounds very inefficient… Your heptagon sounds inefficient, because it’s like, “Well, we’re going around in this circle.”

But what I’ve heard pointed out is that progress often looks like a circle when you look at it on its head, like in a two-dimensional plane… But then when you look at it in three-dimensional, it’s more like a helix, where it is moving in a circular way, but it’s getting better as it goes… And I think with software, that’s a lot of what we’re seeing - these iterations, and a lot of times returning to the old idea, but you’re returning to it with new eyes. You’re returning with new tools. And so you are building up, but you’re not like building blocks on top of each other; you’re kind of like circling the wagon, but you’re going up… You know, it’s like a helix rising… Which is slower than we would want it to, but it’s still progress, right?

Adam Stacoviak: The fund could easily be replaced by a pool of people that are just simply angel investors, though. And it doesn’t have to literally be a fund…

Ben Burtenshaw: So when we create a dataset, either manually, or with AI feedback or AI generation, we have all the information there to understand the problem. We have a set of guidelines, we have a set of labels, definitions for those labels, with documents, and definitions to those documents. We give those to a manual annotator, or we’ll go out and collect those documents, and w give those documents to the manual annotator. And we’re trying to describe that problem so that the person understands it to create the data.

We can essentially take all of the same resources and give those to an LLM, and get the LLM to perform the same steps. So there’s two parts to that. There’s a generative part where the LLM can generate documents… So let’s say we’ve got 100 documents in our dataset, but we want 10,000. We can say “Generate a document like this one, but”, and add variation on top of that. And we can fan out our dataset, our documents from 100 to 10,000. We could then take those same documents, or a pool of documents from elsewhere, and we could get feedback on that. So that could be qualitative feedback. “Tell me which of these documents are relevant to this task.” “Tell me which of these documents are of a high quality, are concise, are detailed”, these kind of attributes. So we could filter down our large dataset or our generated dataset to the best documents.

We could also add labels. So we could say, “Tell me which of these documents relates to my business use case or not”, these kind of things. Apply topics to these documents… And then we can, in doing so, create a classification dataset from those labels. Or we could, in one example, take a set of documents and use a generative model to generate questions, or queries about those documents. And we could use that to create a Q&A dataset, or a retrieval dataset, where we generate search queries based on documents.

Nick Nisi: They had like a pool party thing… After the water park closed to the public, they reopened it just for that conference, which was awesome.

Scott Prutton: So yeah, we have we have a thing called Veritech. It creates cyclones. Cyclones are basically a little server that runs inside of a Firecracker VM, that runs the untrusted code, and then returns the response back to Veritech. So John and I built like a VM instance that we spin up on demand. We have a whole pool of them listening, waiting for work, and they’re single use. So you send code to it, it runs the code, and then it tears itself down. So we can run like thousands of these in like a second, more or less, get a lot more throughput. It is what Lambda uses under the covers, but we have a lot more control over it. We don’t have to worry about timeouts, and cold starts, and those kinds of things. It’s pretty cool.

Justin Garrison: One thing I’ve found fascinating about some of the talks you’ve given and information is the fact that Meta is still notably an on-prem company. You have your own data centers, you have your own regions, you have machines… And it doesn’t seem like you try to hide that from people. You don’t try to abstract it away. At least I haven’t ever seen a reference to like “It’s our internal cloud.” No, it’s like a pool of machines, and people run stuff on the machines. And the software and the applications running on top of it are very much like a – this is just like a Systemd unit; you’re just running it containerized.

What other types of services do you have internally that people need? I mean, I saw references to things like sharding for like “Hey, we need just fast disk places, and we need some storage and databases externally.” But what are the pieces that you find that are like common infrastructure for people to use?

Adam Jacob: I mean, there’s certainly no one left to hang out with at the end… And certainly they’re the people who are most likely to be able to talk to you about how it is. Like, they’re the ones you can most talk to about “Oh, that’s awful. I’m so sorry that happened to you. I know what that feels like.” They’re also your peers. They’re the only other people who know what it’s like. So when you think about destroying that pool of people… Like, of course I wanna help people up in that way, because I need those people. Someday I’m gonna need them. I’m gonna need to call them up and be like “I had to do this hard thing…” And they’ll be like “Oh, yeah, I understand.” They’ll be there for you in a way that they won’t, obviously, if what you did was punch them.

Jerod Santo: Yeah. I remember I was at a hotel one time and I was hanging out in the pool, or maybe the hot tub, and there’s a sign that said “This pool is not for defecation purposes.” Yeah, which was a very strange sign. And that might not be verbatim, and I can’t remember if it was defecation, or – it was very formal though, so it probably did say that. And I thought “Yeah, somebody pooped in this pool at one point.” And there was an incident where they said “We’ve gotta put a sign up, because –”

Jerod Santo: I’m not gonna wade into that pool. We’re gonna let that Adam have the final say on that. Alright, Nick, anything else you want to say about this or any topic before we let you go?

Tejas Kumar: Yeah, that’s a great question. You’re unlikely to crash things, especially like – so I’m working on a 2021 MacBook Pro with an M1 chip. It’s pretty old. It’s three years old. It’s a very old device. And it works just fine. It’s pretty much impossible to crash, unless you really get around it.

For example, the macOS kernel is really extremely world-class at making sure you don’t crash the system. There’s plenty of safeguards in place. So what will typically happen is your application itself will freeze, so not your entire system. And at some point it will say “Hey, this thing’s taking too long. Do you want to force it to quit?” and the kernel will just pull the kill switch. Very cool.

There’s a way around this using an environment variable. So PyTorch is the thing that’s causing the memory problems. PyTorch is an open source library from Meta that helps with machine learning. And so PyTorch allows you to set an environment variable called mps high watermark ratio. And this is at what point do you throw an out of memory exception. Because high watermark literally means you’re about to reach the watermark, the level where if this is a tide pool, you’re going to start losing water; you’re going to overflow, literally. I love the language we have in computer science. Overflow, watermark etc.

So you set this, it’s a threshold before overflow, at which point PyTorch will just kill the process. You can set that to zero. And then what will happen is you just completely bypass – so you’re like “You know what? If we have an overflow, we have an overflow. I’ll just like hard reboot.” And so you set that to zero, and then you’ll crash your system. Because all your GPU, all your CPU is going to be consumed, and you’re not going to have free resources to respond to the Caps Lock key and turn the light green. You’re not gonna have resources to –

Jesús Espino: Okay, next step - one of the things that I was intrigued about Go, because it’s one of the key things, is goroutines. It’s something that everybody wants in Go, everybody loves in Go… And I did want to understand better what a goroutine is, and how it works… And I started digging into the source code and I started investigating how the goroutines work under the hood, from the perspective of the goroutine, of the lifecycle of the goroutine. And actually, for that I prepared another talk that is called “The secret life of a goroutine.” And that in that talk, what I try to do is follow the process of when a goroutine is created, how it’s created, and what are the steps that the goroutine goes through, and how it changes from one state to another, and how the reasoning around that changes.

And one of the things that I discovered while I was doing that was it’s something that I already have kind of the intuition around that, or the understanding about that, that is goroutines are cooperative. But whenever I started seeing the code, it was more obvious and evident that it’s a cooperative approach.

In threads, in operating system threads, you have the operating system deciding when to cut something and assign another thread to the processor. In Go, the goroutine itself is responsible of saying “Hey, I’m going to stop myself, call the scheduler to select another task, or another goroutine, and the goroutine is going to start running.” Not only that, I just stop myself because I’m waiting for something, and the one that is going to wake me up is another goroutine. For example, whenever you send something to a channel, and it’s another goroutine waiting, the goroutine that is sending to that channel is going to wake up the other goroutine. So they are collaborating together to do all the processing.

There’s some special goroutines like the system monitor goroutine, that is doing certain things, or the net pool that also - it’s kind of monitoring certain things for getting back that goroutines whenever they finished… But in general, the goroutines are waking up each other or are sleeping by themselves and calling the scheduler to select another goroutine. So that cooperative nature of the goroutines is something that I’ve found very interesting, and it was an a-ha moment for me also. Yeah.

Chris Benson: I think there’s certainly a chance at it, and I would argue – it’s the same argument I’ve made in previous shows where we talked on similar topics, is that we’re seeing… As the AI industry is has been maturing these years at an incredibly rapid pace, but we’re still seeing many of the things occurring that we saw when the software world was really maturing over several decades. And the place where open source has really, really worked are in common touchpoints where all organizations or many organizations need a common thing. And they might build something differentiated on top of that, for their revenue, to drive profitability… But there’s so much that is underneath that point of differentiation that they and many other organizations can get the benefit out of a lot of effort, a lot of work; a lot of times they’ll have paid employees do it…

So there’s a point where working together and doing open stuff makes sense for business, and it drives profitability. It may not be your single point of differentiation, but if it’s anything under that, why not? Why not share the costs, and pool expertise for the best possible foundations? And so what I’m hoping is that we continue to see that play out in the AI space. We’re seeing – if you look at Hugging Face, we’ve already talked about the fact that a couple of months ago they announced that they were hosting a million models; those are all open source. Really, really impressive. And so I think that there is a good chance that a vibrant, open community around AI can and will continue, and it will have a lot of corporate players involved in it. So I’m very optimistic in that way.

Break: [00:13:19.26]

Adam Stacoviak: I’d like to see better security tooling, that aids a developer. Does that shift left. Versus being like “Nah, third party. Not cool. Rug pool, not cool.” Super-not cool. I would like to have more security tooling in there that’s for developers, and developer-focused, and doesn’t become this - and I hate to use the word “like” in there… Doesn’t become this signal versus noise issue where you’re just getting alerted to things [Dependabot!] that don’t matter. Right? We talked about this recently on securing GitHub, with Jacob DePriest. And so I think my stance is really like “This sucks.” That linking out to third-party CDNs - that’s dead to me, based on this for sure… Because you can’t trust the domain ownership anymore, right?

Gerhard Lazu: I really like how we are improving Jerod’s pool. This is amazing.