Adam Stacoviak: Okay, so wsl.exe in this thing. And now you’re in a shell that has Rsync. And so you’re running Windows, you’re in a terminal, and you’re running Rsync… So now I can take some files on this Windows machine, because I’ve whatever through compression to put it into long-term storage, or whatever… And I do that, and I run that command which is a Bash script, probably, on that same Ubuntu image… And then I would Rsync it over to the NAS, on a Windows machine. That’s badass. That’s awesome, dude. Does that not get you excited?

Andreas Møller: So because most of authentication is a backend issue, we don’t handle most of it, because we don’t have a backend, we just work with whatever your backend is… Whether you coded your own, or whether you use like Supabase directly as a backend, or whatever you’re going with. There’s also some – like, Xano is one of that more low code/no code style backend; it works really well with that as well. So from our point of view, again, we’re frontend, so we’re just doing HTTP requests to a server. So what we’ve done around authentication is say “What matters for us is how we store the token, in essence.” Whatever strategy you have for authentication is possible.

What we sort of nudge people towards is storing your token in an HTTP-only cookie. Because that’s generally the most secure way of handling it. And what we’ve done is we have an API proxy. So whenever you have an API from the frontend, if you run it through our proxy, we can actually authenticate that request. So let’s say you need to send an authorization header with your user token to an API - that’s the most common approach - well, you can actually set up “Well, that’s what I want to send”, and you don’t actually have the token on the client, because that’s stored in an HTTP-only cookie from when you authenticate it, but we actually put that token in before that request goes to the server. And that way, even if you have some script injection attacks, etc. they can’t actually steal your token. That’s not going to fit for everyone. Sometimes if you want to do – especially if you’re doing like real time WebSockets and stuff, that’s not always an option, and you do need to have the token on the client. So whatever your authentication strategy is, is possible. And it sort of speaks to the general approach…

I sometimes say “We didn’t really invent that much.” And by that, I mean almost everything works the way you would expect it if you stopped looking at it as a visual tool, and started looking at it as a JavaScript framework. So when people ask “How do you do authentication in React?”, it’s sort of almost a half abstract question, because it’s like “Well, however you like.” And similar, “How do you store things on the client?” Well, there’s local storage, there’s session storage, there’s IndexedDB, there’s all these options… Because it’s a web browser, right? And we are a framework, and we just build visual tools on top of that. But we didn’t invent whole new abstracts. The web’s pretty good. The HTML, CSS - it’s the best we’ve come up with yet, I think… So we’re not reinventing it.

Obviously, it’s not running JavaScript, but it’s very much stealing everything it can in terms of function names and making sure that everything is very familiar to someone coming from that world. And again, all the browser APIs, they’re the same.

Derek Collison: I haven’t. We built a Swift client. I wasn’t involved in it. Again, I dropped the ball on my learn a new language every year. I actually did program a little bit Swift… I think the interesting part about where Zig is is that we can control direct IO, meaning we could go through io_uring on Linux, if it’s there, applicable, or whatever… And it allows us also to control more precise layouts of the data structures, and of course memory.

So anyone who’s been programming in Go for a long time and has a tremendous amount of investment in a big codebase, you start to see things pop up. It’s like “Oh, avoid JSON at all costs, because the built-in JSON marshallers [unintelligible 00:17:59.17] lots of memory.” The network IO and the disk IO subsystems in Go are very different. And the disk IO subsystems are pretty much a native thread that kicks back into your goroutine that says “Okay, it completed. The write completed”, or whatever. And I think what we need is a lot more fine control over those things.

So what I’ve been looking at is the adoption or the ability to introduce io_uring in a meaningful state. And I think Zig’s done a pretty good job there. Maybe not within the language per se, but with folks that use it. Tiger Beetle and Bun come to mind. And so that one’s kind of a big one. The data structure layout, the garbage collection and taking control over disk IO… Which - JetStream, we’ve got folks that are coming in and say “Oh yeah, I need to do a million stored messages per second, replicated three ways, and then an async mirror over in a different region and a different cloud provider” type stuff. So we can abuse the disk, the storage subsystems pretty well, and we might be getting close to kind of everything we can accomplish with Go… I’m not saying that that’s definitive, but it’s starting to feel that way, as we’re starting to do - not unnatural acts, but we’re starting to do a lot of things inside of the server to get around how Go actually marries concurrency with disk IO.

Gerhard Lazu: No, no, no. I think this is good. This is exactly – I mean, these are the big questions, because honestly, there is no right answer. So a progressive rollout is the most cautious one, especially if you don’t know how the new system is going to behave. In our case, we’re spending a lot of time to double check that the correctness of the system is right, and that the system behaves correctly when it comes to all the other… So it’s one component, the CDN, but it integrates with S3, it integrates with a bunch of other… It integrates with S3 for stats. It integrates with Honeycomb for all the telemetry, for all the traces, for all the events. It integrates with R2, the different R2 backends for the actual storage of certain components. So there’s a lot of – we’re just basically replacing a central piece, and everything around it still remains; the integration has to be right.

So yes, we could do a gradual rollout in that maybe from a DNS perspective we say “25% of queries return this backend, or this origin.” Sorry, in this case, let me just not compound the word origin… 25% of the requests go to Pipedream, and 75% go to Fastly. And how do they behave? But at that point, we are maintaining two systems, which is okay, but it cannot be a long-term solution. So we want to shorten the window in which we run both systems at once, and that both are active. Because we could very easily switch, for example, to Pipedeam, make sure that everything runs correctly. Let’s say that we detect that “Hey, for some reason something isn’t behaving correctly.” We still have the old system. We just point the DNS back and everything continues as it was, which is why two of everything. That’s another principle that we have.

So at this point, we have two CDNs. We have two applications, which are completely isolated. Now, they are running on Fly; the runtime is the same, but if one was to go down, the other one wouldn’t know about it. So we’ve designed this in a way that is very cheap to fail. The new stuff, if it fails, will have impacted maybe a few minutes’ worth of traffic, and fail catastrophically… Which is why we’re running all these benchmarks, running all these correctness, to make sure that the chances of that happening are low. No guarantee in anything, but they’re low.

And going back and forth is super-easy, because we run both things at the same time. The problem of running both Fastly and the new one is that we may see inconsistent data that gets written out. I’ll go to great lengths - I mean the logs, I mean the events; I’ll go to great lengths to ensure that’s not the case. But if there are little discrepancies, we may end up with different data, and it may take a while to find that out, especially on the metrics side.

Stephan Ewen: So the Restate server, which is where the low-latency consensus log lives, and the thing that basically becomes the reverse proxy for your service - that thing has not really any requirement if you wanted to get started. It’s a self-contained binary, it embeds its own distributed log, a RocksDB storage engine, its own consensus engine… The only thing, if you want to run it as a single node, is you need to give it a persistent disk. It’s almost that – let’s say, if you would want to run SQLite or Postgres, it’s a little bit like let’s go back to the good old days where you download one binary, I just start it and it’s actually running. It’s actually good, there’s nothing else you need to do. But then at the same point, it’s also able to go from that single process that you start with a single binary to actually cluster up and build a distributed cluster… And there’s a very interesting architecture in there. We build it basically for the cloud data age, where you would say any system that you run at scale should not really store its own data, but it should just make use of object stores for as much as it can… Because S3 and these systems - they’re these bottomless, insanely durable and insanely cheap storage systems. So make use of that as much as you can to put a large chunk of your data.

So that means while you may be working with the data on your individual nodes, you’re not really required to safeguard it on the nodes, because you can recover it from S3 or an object store. So what Restate then does is it actually implements its log in such a way that it only uses this to really give you the very low latencies for the durable steps, and then in the background it incrementally moves data to S3, which makes the individual nodes fairly lightweight to operate.

So to go back to your question, like “What are really the requirements when you want to run it?”, if you want to run it on a single node, non, or a persistent volume if you actually want to run it like in production - if you want to run it in a distributed setup, give it an S3 bucket. Those are the requirements. If you want to use it from your code, the requirement in your code is to use the SDK, and to basically create a Restate entry point that Restate can connect to and where it can use a sort of durable invocation protocol that understands how to decode that. This sort of entry point is mimicking the popular frameworks; it’s relatively close to Express.js if you’re talking in the JavaScript world, and the Java world looks more like Spring Boot and so on.

[01:13:42.14] And then within an individual durable function, service handler, you need to use the Restate context to say “Okay, I want to run this step and record it as durable. I want to create this as a durable promise for a persistent callback”, or so. But otherwise, the structure of your code is very much the same as it used to be. So it’s supposed to be as little invasive or to get as little in the way of how you used to do things as it can… Just sort of changing the paradigm as in – because it has this fine, good durability for these operations, you can get rid of a lot of the sort of unhappy path code. There’s still cases you need to treat, but mostly you still need to treat sort of like persistent errors that come from the application, and where you’d say “Okay, you’re making a call to an API where you’re not authorized. There’s not really a way you can recover from this. You’re just trying to do something you’re not supposed to.” And “Handle this.” But don’t worry about handling process failures, network failures, rate limits that bounce you back. Don’t worry about many classes of race conditions, about the state being maintained in the database versus the logic that interacts with it in a function that where you don’t know really “Did this go through or not?” If you put the state at the Restate handler, it’s just going to be consistent for you. All of those things by keeping the structure of the code like those two, what you used to write.

Jerod Santo: Grepping logs remains terrible

algernon first stated his public opinion that grepping logs is terrible a decade ago. The good news is he’s back, the bad news is his opinion hasn’t changed. After a quick comparison of running alike queries against 4.4 GB of JSON logs on his modern M2 SSD desktop and loading the same data into VictoriaLogs on an old Mac Mini, he finds:

What we saw here is that a computer with much more RAM, and computing power, and faster storage got absolutely obliterated by an overloaded Mac Mini that is ten years its senior. And not only in speed, but resource use too: I haven’t shown it, but working with raw text used a lot more CPU time, disk IO, and memory; meanwhile the Mac Mini barely blinked.

And not because it was all cached in memory! These are all cold queries. Cached queries are much, much faster. But how can it be that such an underpowered device runs circles around a powerful desktop?

Purpose-built software, dear reader. Purpose built software.

The moral of the story: databases are good!

Don’t grep your logs. Shove them in a log-ready database and query that. Not only will it require considerably less disk space, it will require less computing power, less RAM, and as a result, less time to perform the queries. And if that’s not enough, you can give the logs structure! You no longer need to remember the order of fields in a httpd’s Common Log Format, you can query against a named field instead. You can run aggregations, computations, build stats, correlate, join, and do all kinds of other database-y stuff. Not only does it perform better, you can do more, and do so more conveniently.

Joran Dirk Greef: So you can actually have a business and sell something that’s gonna make something cheaper for startups. And similarly for enterprise, you can have a business and sell something that is going to provide the value they need, which is - now, they might have SRE teams, but they need the infrastructure to support massive scale, like Petabytes. How do you connect TigerBeetle to object storage S3. Like, OLTP data lake; not only OLAP data lake, but let’s just connect the OLTP direct to S3.

And so this comes to your question about the tug of war, and licensing, and all of this. And I think the big mistake that we can make - and I used to make this until it became clear for me that third moment - was that an open source license affects the implementation, not the interface. But when it comes to competition in the business world, that doesn’t happen at the implementation. Typically, it happens at the interface. So if you think of like some of the fiercest competition, when things were really on a knife’s edge for the web, it was the interface, not the browser implementations. It was the interface that the war was fought. Mozilla fought that war, and we needed other browsers to fight it, because the interface was being embraced, extended and extinguished. Triple E. And then you think of like Android, and Java, and again, it wasn’t about the Android implementation, it was the interface. And that was that Oracle/Google lawsuit, you know? And then again, you think of like “Well, Confluent –” Kafka’s Apache 2.0 open source.

Then Red Panda came along - and I’m a huge fan of Red Panda, because very similar design decisions around being direct with the hardware, static communication… All of this very, very similar. We came from a similar time period, but in that time period the things were changing, how you built things. But Red Panda came along and they saw the open source implementation of Kafka and said, “Well, thank you, but we don’t want it. But that interface, that is great. That’s where our business will be also. That interface, thank you very much.” And so they built a massive high-performance implementation.

And then WarpStream came along and they said, “Well, Red Panda, you are business source license, not open source source available. Confluent or Kafka, you are Apache 2.0 open source.” But that’s all implementation. I’m gonna do my own implementation, thank you very much, of object storage. But the interface - great. Okay, now we’re all competing. And so I think the myth is that source available was kind of the thing that always – I always feel that something inside of me dies when I see a great company relicense, or when I see a young startup follow that lead. Because to me, source available says that “We think it’s gonna stop competition.” It doesn’t. You may as well be on the beach, building little moats and sandcastles, but innovation technology is like a wave. It’ll just find a way around you. It’ll WarpStream around you. You can’t legislate competition away. And we shouldn’t be trying to build companies where we think the success of the company is us creating a monopoly. The world’s too big. There’s too much for you to be – you don’t need monopolies to do really great. And that doesn’t build trust, to say to your customers “You can only buy from me.”

So I think people think it stops competition, and they think it helps them sell, and it actually defeats both of those, because you get complacent, and you actually fail to build trust. So you burn trust when you relicense, and if you start source available, you’re gonna be doing diligence with enterprising and say “Sorry, you’re not open source.” It’s license confusion, you know. And maybe some people get it, but there’s a little 1% headwind. And it’s actually - it’s a category error, because you’re spending so much effort chatting to people about implementation licensing… And the rest of the world is competing on interface. And TigerBeetle’s interface is quite simple. Very simple.

[01:24:30.17] So we could – I don’t know, whatever license we apply, it doesn’t matter. Debit/credit is where we compete. There are companies that offer debit/credit as an API and it’s very similar to TigerBeetle. But we compete on trust. We didn’t just take a general purpose database and slap on debit/credit. We went deep; we really cared and we built the whole thing.

And people pay us. So before we were a company, we had an energy company reach out and we landed quite a good contract, very quickly. And I think it came down to trust. So open source builds trust. So open source is great for business. It’s also orthogonal.

I think the other thing is if you – so there’s a lot of things in TigerBeetle that are like the… I had done many experiments of my own, my passion projects. They’re all in TigerBeetle. Many parts of the design of TigerBeetle come from these various experiments that I did. And so I was never going to put that all into a project if it’s not open source, because it’s just too valuable. I want to always be able to play with it no matter what happens to it. And I think we all feel like that, like our critical infrastructure just has to be open source. So yeah, I think that’s kind of how I think of it.

Amal Hussein: You got it. And what’s really cool about this is – we are a trust platform. We don’t look at your data, we actually – we use this tokenization workflow, which is we kind of bypass… We use our clients to connect directly to storage buckets. And so we generate a token, and we store a reference to that token in [unintelligible 00:34:45.29] But none of your data passes through our layer. And so hence the control plane/data plane that we set up.

Gerhard Lazu: Yeah. For video storage, it’s something that you would need. If you were to do editing… And especially if you edit it for multiple machines, you need like a fast network, a couple of other things. But…

Techno Tim: Yeah. Heat, noise, power… I could do away with most of my fans. And they’re loud, in general; they make noises. My flash SSDs make zero noise. They give off almost zero heat. They take up a quarter of the space, you know? And so because of that, my NAS is 4U, because it needs to fit these drives. I don’t know. I wish we were just all SSD, all flash storage.

Glauber Costa: But it’s not super-fast for analytical stuff. So the first thing that it unlocks is just this, queries that are much more complex, that you can run in dashboards, and things like that. And also, for the serverless environments, Turso being one example… And for clarification, I’m going to say - our product will be renamed to Turso Cloud, and we have all the intention in the world to rename Limbo to just Turso. Limbo is just not a great name. Just a parenthesis here. The story, it’s all in the blog posts, but we never expected this level of support, so we just came up with a made up name for it, that means nothing. But now we will address that.

But our product, Turso Cloud, allows you to do serverless SQLite on the cloud. You have a bunch of HTTP requests there in the middle. It allows you, for example, to host your data on S3. Because if you have your data hosted on S3, your query now is not necessarily super-fast, because you may be hitting a page that is not local. So it allows you, for example, to run SQLite with partial storage, like with most of your data on S3, and some of the data locally. It allows us to do all sorts – it allows us to run on the browser, because the browser, as you know, is a very sync-unfriendly environment. If you block everything, the page just doesn’t load. So in the browser you have to be async.

So there’s a lot of things that it unlocks. In fact, the one thing that – Limbo doesn’t even support transactions. It’s very, very early, which for a lot of people who came to contribute, they actually saw as an advantage, because you come with this energy and there’s so much to do… But we’re not even yet at the point where we support transactions. And the one thing that is already different and is resonating a lot is the fact that it is async.