Mat Ryer: “I don’t mind Ash… I don’t even mind Bash.. I like caching and cash, and I’ll clash with the dash in a flash, man… I’m not going to say gash… I don’t even mind hash… I’m out on the lash, man… I’ve got a rash. Oh my gosh, man, I’m going to smash your face in if you backslash me. That’s trash. Backslashes are trash. A backslash is a trash… Yeah, a backslash is a trash… And don’t say forward slash… Just say… Slash… Just say slash. Just say slash… No need to say the forward bit… Just say slash…”

Andreas Møller: But when we look at these big, massive games we play, they’ve been built this way. Like, there is a lot of code involved, there’s a lot of really talented engineers working there, but a lot of the logic is built using these visual tools. And so the basic idea was “What if we build web apps the same way?”

So the best way of describing it on a technical level is probably that it is, in effect, a programming language with a built-in framework, but the biggest difference between – like, if you’re familiar with Elm, that’s actually a really good analogy as a programming language.

So Elm was this programming language with a built-in framework. The Elm architecture was the inspiration for Redux, when that came up… But it was a whole language built just around the idea of building frontend apps. And Nordcraft is actually the same, but instead of the code part – normally, when you have a programming language, you have the code, that gets passed into an AST, which is this abstract representation of your program, and then that gets compiled to either code, or JavaScript, or whatever it is the engine you need to run it in. And all we did is we sort of cut off the code part, and we were saying “What if we just design the big syntax tree of what our program is going to be, and then we store that in a database? And then we build an editor for manipulating that tree.”

And what that means is a lot of the things – and that editor is where normally a visual program is like a big board of like a million nodes. We have some of that too, but what we’ve done is saying “Different problems have different needs in terms of how you build a tooling for it.” So if you’re building UI, obviously the editor is going to look like a UI editor. You’re going to drag HTML elements, and you’re going to click them, you’re going to apply styling. And because we wanted to build for professionals, very early on we set on this path of saying “We’re not going to try and reinvent new abstractions here”, because that’s really hard, and it usually takes a long time. And there’s a reason why we haven’t seen new abstractions since probably React, or Angular time. All the new frameworks are the same abstraction.

[32:15] So we’re not going to try and invent a new abstraction. We are just going to create a very nice UI for working with the web platform. HTML, CSS, everything you set in our style panel maps to a CSS property. And we even in our tool tips show you what the CSS property is, and what the CSS is going to be rendered as. Because we want you to have full control.

So the HTML you see inside Nordcraft’s visual editor is going to be the HTML we render for every component, on every page. The CSS you see is the CSS we render. We do hash some class names, but that’s about – other than that, you basically see exactly what it is.

And then different problems have different solutions. We have a workflow editor for like “When I click a button, what do I actually want to happen?” That becomes almost a flowchart. And it turns out that – and we sort of separated, we made a distinction between what we call actions, which is I do something that’s like I call an API, I update a variable, I trigger an event… Those kinds of things. Things that we in functional programming say have side effects. And pure functions. So pure computation, we call that formula, anything that has a side effect [unintelligible 00:33:35.15] And those are actually two different editors. And that allows us to sort of visually separate the two, which turns out is really nice, because you get the high level structure of what is it this thing does, and then you can dive into the details much easier. So it becomes a naturally nice way to organize your code, of saying “Take all the computations and actually hide them initially, and just show me what are the steps. And then if I want to dive into why you did that step or why you did that other step, then I can go and do that.” And I think it’s not far from how people often organize code anyway. But each of those kind of have their own UI. They’re a separate problem. The editor for doing pure formulas, pure computation looks very different than the editor for doing a flowchart. And completely different from any of the UI stuff.

Jerod Santo: Yeah… Like a giant hash table in the sky, that also has some very gnarly semantics that you can use.

Joran Dirk Greef: Yeah. And the second thought was this question, “How can you possibly be as safe as 30-year-old software that was created Windows 95? How could you possibly be as safe?” And I think the answer to that is actually a question. In the same way that “How could you be 1,000 times faster than something that is 30 years old?”, the question is “What has changed in the world in the last 30 years?” So much has changed from a performance perspective… And then when we look to safety, and especially mission-critical safety, so much has changed.

So 30 years ago, consensus didn’t really exist in industry. Brian Oki had pioneered consensus a year before Paxos View-stamped Replication in ‘88. That was his thesis at MIT, with Barbara Liskov. The consensus did exist… You know, how can you replicate data across data centers for durability, to actually survive loss of a machine or disk? But that wasn’t really an industry 30 years ago, so you don’t really have first-class replication. Yes, you do have it these days, but it wasn’t designed in from the start. And I think there’s more examples around testing. So deterministic simulation testing, what FoundationDB did - and we actually got it from the people at Dropbox, not Foundation. James Cowling and them at Dropbox were also doing DST.

The idea is like you design your whole software system that it executes deterministically. So you don’t use – if you use a hash table, the hash table, given the same inputs, always gets the same layout. You don’t use randomness in the logical parts that users would see. you can think of it basically like fuzzing your software or property-based testing. Given the same inputs, you get the same outputs, and you design all your software like this. Then you can test it. But if the test fails, when you replay it, you’ll get the same result. And so distributed systems were so hard to build 30 years ago, so not many people did. And then when they started really building them, they actually started with eventual consistency, because people were still figuring out how to do strict serializability. So there was a lot of fashion around eventual consistency, which has gone away, I think… But back then, to build a distributed system you just kind of assumed it’ll just be eventual.

So how to build distributed systems was quite hard. And it was hard because it was hard to test them, because the failure of one system over there causes the failure of another system here. And you can never – when you find a rare bug, you can never replay it, and you need so many machines… And these bugs – you know, before TigerBeetle, I worked on systems that were distributed, like full duplex file sync, hence my interest in Dropbox. But those systems were incredibly hard to test. There would be bugs, you know they’re there, they’re taking a year to find and fix. And then you realize it’s a buffer overflow in the libuv. That was some of my first C code that I ever wrote, years before TigerBeetle. It was fixing a buffer overflow in libuv. It was the Windows event notification, some interaction with multi-byte UTF-8, and different normalizations of NFC, NFD. That was this distributed system’s bug. And it also needed long file paths. So it took a year – like, we knew it was there…

Ilya Grigorik: Right. The CSP policy is “require SRI for” scripts. So you’re saying “All of my script resources must have a hash”, and then you can configure that to be a report-only policy, such that it would still execute if the script is missing the hash, but you would get the violation fired in the background. And the browser has its own logic for prioritizing batching delivery and doing all of that, to get you the report.

Salvatore Sanfilippo: It’s a lot of people, and the development part is almost completely in Israel. A lot of remote work is inside Israel, in many places of Israel, in the North, in the small cities, small villages… And there are truly talented people there. Incredible persons. Also very loyal persons to the company, that understand the project and understand it is very important. Also, they are people that kind of fight for the AGPL thing, because they thought it was a good thing. So people interested in the project. And I believe it is a very strong pool of talented people in Tel Aviv.

Also, another thing that happened that I believe is extremely cool is that we are acquiring new people that are like that. What happened is that one of my colleagues, Oran, which is truly a genius of programming, together with Yossi - they are two really superstars that nobody knows. For example, Yossi was the one that wrote the abstraction layer in order to make Redis SSL as a module. So you load a module and it is SSL, otherwise you don’t have the complexity, and stuff like that. For example, this Chinese person starting to do pull requests only on comments. This comment, this – and another guy, which is a Portuguese, for example, also doing initially simple pull requests. But he recognized that they were smart, and started to help them to create more complex pull requests. And since Oran works a lot, since he’s responsible of all the developments of Redis on Flash, which is, you know, the way that we have to offload part of the data from RAM to Flash; it’s a complicated fork of Redis. You know, he had to do a true investment, and now these two persons are very strong core contributors of Redis, making complicated stuff.

[00:52:00.19] Also, another guy, which is the one that implemented the expires in the hash type elements, single elements - he’s very talented, and he optimizes this thing to a so low level. And for example, one thing that I still love about Redis, and I think that we have an edge here compared to, for example, Valkey, is the design idea. In Valkey there are a bunch of people and they do a lot of agreement design, which is a way of working that I don’t believe will be optimal. And why I hope that Valkey is going to be great, because in some way this was – you know, the thing about the BSD is that whatever happens to the company, to myself, the code in some way can go forward, because it takes other streets, like Amazon, or Google forking, [unintelligible 00:53:00.11] However, from the point of view of the software idea, I don’t think that Valkey can do a better job than us. And one thing that I would love is we go back in to AGPL to be also OSI-approved it again, and then compete on the quality of the ideas, and the quality of the developments… You know, because Redis is going to diverge a lot.

For example, one of the leaders of the project said, after seeing in my blog post the vector sets thing, “We will never do something like that”, because they don’t trust in this line of, you know, AI is this terrible hype. For example, this is one way to approach the problem. I am, for example, an AI enthusiast, and this already creates a different take on the project.

Jimmy Miller: The other thing that would make this a lot easier is if you kind of got rid of this text-based approach to programming. Maybe you go for something like – I don’t know if you’re all familiar with Unison. So Unison is a language that is being produced right now. They have like a Unison cloud, and all of this. And the idea behind it is your codebase itself is a big, immutable database. So when you make a function, you don’t store it as – say you have map. You don’t store it as the function map, as like “This is the name and this is the text I have.” You store the hashed contents of that function, and map is just metadata on top of that hash of the AST that you wrote.

And so your code is all completely immutable, and it’s all content-addressable… And so what you could now do is you could have very easily “Hey, I want to write a note on this map function.” Because it’s this hash. “Here’s my note attached to this hash. And what they do when you refactor, and maybe someone publishes a new version, is instead of having to be like “Here’s this version of a library” and I just get all the latest, you actually see “Oh, they changed map from hash one to hash two. Do I want to accept the new map thing? Oh, okay. Everything in my codebase that’s pointing to hash one, now I change it to point to hash two.”

[01:30:11.20] And if you have this kind of setup, it would be very easy to attach these kinds of notes as metadata. So maybe we’re just like – these are all the kinds of things that these more futuristic ideas of programming could unlock for us very trivially, and we’re just trying to shove it all onto text, and that’s why we’re running up against these problems.

Jerod Santo: So let’s go back to MagicLinks and talk about OTP, because this is kind of – to me, it seems like maybe an evolution of MagicLinks and an improvement. So the idea here is that I’m still going to send you something that you can then confirm that you have, but instead of just making it a link, which in our case it’s like a long… It’s not like an md5sum, it’s like a hash value that you would not be able to just rattle off; it’s shorter, and time-based, and usually it’s six numbers that are provided… And so the one-time passcode is sent to the email, or whatever way you can send them. I’m sure you can push notify it, or whatever. And there’s a click provided, so you can still just click on it and just embed it in the URL in that case… Or you can just read these six characters and type it back out… And that really solves one particular bummer about MagicLinks, is the shareability aspect, and the switching contexts aspect, which a lot of people run into. It’s like “Hey, I’m on my phone. I sent myself a MagicLink, and I don’t have that email app on my phone.” There’s all these different weird things. Or it opens in an app-specific browser inside of my email client, and so it logs me in inside of Gmail app, but I go back to my other app and I’m not signed in… Well, with these one-time passcodes, you can solve that by just either copy-pasting the six digits, or just remembering them for 10 seconds and typing them on the other side. So that seems like a nice evolution.

Marc Boorshtein: Exactly. No hash though. No, we got the password, we just handed it back off to [unintelligible 00:12:55.28]

Jerod Santo: No toast… I had some hash browns.

Qi Xiao: [00:35:49.23] Yeah, I didn’t go back to that thousand-line Bash script, because somebody else took over the maintenance of that script. But I did rewrite – so Elvish has scripts; the project REPL has a bunch of scripts inside it. And over the years, I have rewritten all the Bash script in Elvish, and it’s always an improvement… Like the string manipulation thing you were just mentioning. So lBash does have string manipulation functionalities, but they usually have pretty obscure syntax. There’s a built-in way in Bash to remove a prefix from a string, and the way you do it is, say, if your string is $s, what you do is $, and then brace s, and then I think it’s hash, and then hash prefix, and then closing brace. So it uses this very symbolic syntax; it makes extensive use of punctuations. Which I guess if you do this on a daily basis, you will remember it and you will like how concise it is. But if you only do this once per week, you probably are not going to remember it, and you will keep doubting yourself, like “Is this the right punctuation?” But in Elvish, because it’s designed like a programming language, you just go to the Str module, and there’s a function called str trim left. There’s str trim left and str trim right. Oh, sorry, there’s str trim left and str trim prefix. So those are the things you would use, because it’s slightly longer than just writing a hash sign, but it is much more readable. And if you’re a Go programmer, these are implemented using the same functions inside Go’s Strings –

Ismael Valenzuela: Yeah, as Greg mentioned before and explained really well, that this is an industry that is always like chasing the new shiny. What’s the new thing that can solve all of my problems? And there is no such a thing. It’s a lot more complex than that. And every time that we try to find that single tool, that silver bullet, we often fail, because of a lack of an understanding of how all these things need to come together.

So we’re in the middle of that hype, and now the tool is, of course, AI. And I would say even more specifically, LLMs, generative AI, because we know, and you guys on this show know well, that when we talk about AI, it’s not one thing. It’s a lot of different things. For example, at BlackBerry we have been using for many years, coming from the Silence engine, from the Silence days, a predictive AI engine. We know we’re talking about predictive machine learning, essentially.

And I remember – well, I wasn’t at Silence at that time, but some of my colleagues that were told me that they were at BlackHat, I think probably 2016, or something like that. They were talking at BlackHat about this, and a lot of people were like “Oh, that’s not possible. You’re selling smoke. That’s not the way you detect malware.” Fast-forward to today and everybody understands that you cannot fight malware with signatures.

I mean, in our report - and we produce these reports on a quarterly basis - we talked about the latest increase in the last quarter. We’re talking about a 53% increase in unique pieces of malware. I don’t know if the audience is familiar with the concept of a hash, or a fingerprint. You take a binary blob of data and you create a fingerprint or a hash of that, and that says “Okay, that’s unique.” Different hashes, different files. So we’re talking about over 11,000 pieces of unique malware per quarter that we have seen with our telemetry. How in the world are you going to create a database or maintain a database –

Gerhard Lazu: No, that’s a good one. That’s exactly what… We’re like trying to hash it out, so that we share the understanding that each of us holds, so that we can talk about like what would be – because we talked about this in the past, and I really liked Jerod’s question… He was asking – we were talking about like kaizening, and we were talking about all this change, but “Are we actually improving?” And that’s why when I try to think about this - and I was thinking about “Okay, what would the improvement look like?” And can we – I mean, we can measure it, and we can check “Have we delivered on this?” And until like the last deploy that went out, I was fairly happy with the time/duration that these deploys were taking… But based on the one which I have right in front of me, the build going from one minute and a bit to almost three minutes - I think that variability is something that I would like to understand first, before optimizing the boot time.

Jerod Santo: Four key comma value. And that key starts with one. Well, are you iterating over a hash there?

Jerod Santo: Well, moving on… Let’s start right where Adam left off. Not with the moisturizer, but with a dish made from a mixture of leftover food, especially meat and vegetables. That’s also known as a hodgepodge, or a hash, or a ragout, or a gallimaufry. That is correct, Adam.

Ryan Worl: Yeah, so Kafka has – let’s start with topics. Topics are basically just a name for mapping consumers and producers together. They agree on the name of a topic for where they’re going to send the data to, and where they’re going to consume the data from… And within a topic, there are partitions, and a partition is basically just a shard to make that topic scalable. There are a lot of different ways to decide which shard you’re going to write the data to, but let’s just say for now you do it by hashing the key of the message, and then routing it to the shard based on the hash of that key. So if you have the record with the same key, you’ll wind up going to that same broker every time, that owns that partition.

So that’s how it works in the open source product. The brokers own some set of partitions from a leadership perspective, and then there’s also replicas of that, that are just copying the data… And it’s just other brokers that are the replicas for those partitions.

So the broker will write that data that it receives from a client, a producer client down to the local disk, and replicate it out to the followers, and then a consumer can come along, and read either from a replica or the leader the data that producer wrote. But they’re all coordinating on essentially - one of those brokers owns the partition specifically that I’m interested in and reading. So that’s how it works in the open source product. And in WarpStream we’ve decoupled the idea of ownership of a partition from the broker itself.

We have a metadata store that runs inside our control plane, that has a mapping of “Here are all the files in object storage, and within those files, the data for this partition for this offset is here.” It’s in some section of a file in object storage.

So any of our agents, which are like the stateless broker that speaks the Kafka protocol to your clients - any one of those agents can consult the metadata store and ask “I want to read this topic partition at offset X. Where do I have to go in object storage?”, and potentially multiple places object storage. “Where do I have to go in object storage to read that data?” But because the metadata store inside the control plane is handling the ordering aspect of it, essentially, you get the same guarantees as Kafka, in terms of “I have this message with this key, that’s routed to this topic partition, and I want them to stay in the same order, because I’m writing them in a specific order.” That ordering part is enforced by the metadata store inside the control plane, but the data plane part of actually moving all of those messages around is only inside the agents and object storage. So it lets you do that thing that I was saying before, where if you want to scale up and down, it’s very easy to do that, because you don’t have to rebalance those partitions, which take up space on the local disk, amongst the brokers in order to facilitate that.

Simon Wijckmans: Yeah. So I tried this in my career, and it was incredibly noisy, and it made for alert fatigue, and it wasn’t helpful. So the angle we’ve taken is quite different, where we basically try to understand what types of data are being accessed, or what types of actions are taking place… And when there’s a substantial change in that, then we will start alerting.

If we must alert every time a script slightly changes - well, that is incredibly noisy, because a lot of these scripts are obfuscated. And if they make a small change to a timestamp, or to anything in there - it could be as simple as a version number. Because of the obfuscation working a bit like a hash, it becomes completely different. So using abstract syntax trees and all types of other tricks we pull on our site to actually understand on a higher level how the script performs, what it does, what it accesses, and then alerting when that substantially changes - that is the better approach.

The way we also work is – of course, we are always working to improve this. We actually want to make this a product that blocks on your behalf if it must be blocked. And we do not rely on threat feed intel. Of course, we use it, so that if there’s a low-hanging fruit, we’ve got it. But that’s not our core thing. We built our own detection engines, that look at changes over time. And because we’ve also got that free tier, and we’ll have a bunch of people’s blogs where so-called there were no scripts, but there were then three scripts, we can of course see these scripts on many different websites. So before it becomes a problem for one of our big, high-security customers, we probably would have already seen it on some other websites as well.

So that free tier is not just an amazing “Use our thing and hopefully someday it’ll pay for us.” No, it is also actively making the product better for everybody, safer for everybody. And so by using our thing, you’re basically making your own life on the web a little safer, if of course people use our product. And that’s a pretty powerful position to be in. So we don’t really want to go down the heavy alerting route, because it just doesn’t really help. It goes in a massive queue of alerts people receive every day, and then it really only causes trouble afterwards where it’s “Hey, why didn’t you respond to this alert?” Well, because I get 500 of them a week. I cannot deal with all of these alerts. It just creates more pressure on the engineer that’s ultimately going to be held responsible for when it goes wrong. It doesn’t help. We have to be very diligent about the alerts that we send, and when we send them.

Jordan Eldredge: Yeah, so we started this partnership with the Internet Archive. So I was trying to find different Winamp skins to test my WebAmp project, and I was “Oh, I’ve got to find all these different Winamp skins to find all the edge cases.” Because I’m basically reverse-engineering this file format. And it’s not documented, so the examples in the wild are the spec. And that led to me collecting a few thousand, and then I was “Oh, these are falling offline.” And websites that used to host them are – no one’s paying attention to them… And so I was “We should try to grab up as many as we can.” And someone put me in contact with Jason Scott at the Internet Archive. And so we started a collection there, and they actually did an integration where the WebAmp player actually loads up on the Internet Archive itself. So you could actually listen to all their audio files in WebAmp.

But then that announcement created this flywheel where people were getting in touch and they’re “Oh, I have a collection of these.” And that’s what sparked the Winamp Skin Museum. And yeah, we just crossed 100,000 unique Winamp skins. Now, that’s unique based on their MD5 hash… So there are some re-encodings, or version 1.1.whatever… But yeah, it is a pretty – I think a much larger number than I would have expected when I first started out on this journey.

Kris Brandow: Yeah, I mean, I think that that’s perfectly fine, of saying “The specification as date x.” You can get snapshots of the HTML specification, for instance, that are tied to a specific date. So you can go implement something in your browser and you’re like “Well, this was what it was at this specific date.” But also, it really depends on the type of contract that you’re doing. HTML works in the way it does because of, you know, you have these browser vendors, and there’s kind of like nobody here that can be like “No, you must do it this way.” So it’s all a shared agreement. Which is not dissimilar from how APIs you consume from third parties work. It’s just different, because there’s only one implementation instead of multiple implementations. So now you have to extract that same sort of promise in a slightly different way, but I don’t necessarily think we need the version numbers to give us that. Or even even dates, really. It doesn’t matter.

[00:48:16.12] At the end of the day, what you want is sign something that says, or have an agreement that says “This is what I expect from you. And if you want to change it, then we need to have a conversation”, or someone needs to have a conversation. Or in the case of a consultant or a contractor to a company, if the contract changes on their end, you can bring me back, but you can’t come after me and say I didn’t do the right thing. I did the right thing as of this. So if you want to judge my work, you’ve got to judge it based on this specific thing. Which once again, whatever pointer you use to reference that thing, whether it’s a version number or a date or hash or what have you, as long as the thing itself is stable and stuck in time, that’s what you care about. That’s the agreement you want to be focused on.

And I think at the end of the day, version numbers, like… I don’t know, even with all these version-numbered APIs, can you actually – with Open API you can. But with other things, can you go download the actual contract that’s being agreed to? If they go back and they change all of their documentation and just completely [unintelligible 00:49:14.28] an entire API endpoint, how are you going to be like “No, it said this before.” I mean, I guess you could go use the Wayback machine if they’re big enough, but that just seems like a very precarious situation to put yourself in, instead of having the “No, here’s a contract that I had of some sort.”

And I think things like Open API can be those contracts. I think that’s why we have these at the end of the day, is because they function in that way. I think we just kind of muck them up a bit with version numbers.

Ian Lopshire: But it’s not like you could hash your C code, and make sure that what’s in that tarball is what’s in your C code, because that’s just not how it works. It’s a precompiled piece.