Kelsey Hightower: Yeah, I think we always have this desire as humans to aggregate things out of the sake of simplicity. If we put everything in the same configuration management tool, then it will be simple. But typically, when you look at that configuration management tool, there’s all kind of if statements and weird logic and extensions that only the person who made that understands, and everyone just says “Hey, don’t touch it. Just use it as it is”, and now it’s very brittle.

I think the thing that does make things simple are things like “Here is the frontend. The frontend has these components, and it retrieves this data from these imports. It doesn’t retrieve this data from functions, it doesn’t retrieve its data from Kubernetes. It retrieves this data from end points.”

[32:16] So now if you’re a frontend developer, you can have endpoints be anything for your testing purposes. I could just run endpoints on my laptop and serve mock data, and the site will behave as designed. So now I say my deployment target for my frontend could by my local browser, or it could be a CDN that will then serve it to billions of web browsers. That’s simple to understand. So now you say “What tool will I use to complete that?” Well, it could be as simple as my IDE, where I hit Save, and it just goes to staging automatically, because I know what the contract will be. When it gets there, it’s going to use a certain set of endpoints, and those endpoints are required to have data.

So now let’s move to the endpoints. Any one other team can create endpoints. You make a promise, endpoint.hightowerlabs.com is supposed to serve this kind of data, and this structure. And if you do that, then my frontend app, my mobile app, my watch, my smart washing machine - all of them will know what to do with that data. And even though that sounds more complex, it’s easier for everyone else to understand that there’s going to be data returned from this endpoint. None of them need to know about Kubernetes, they just need to know what my contract is. You need these credentials to get this data, and I promise to keep it up 9.99% of the time. So now that we have these contracts, now I can go focus on my area and just assume you have your area.

So now let’s talk about deploy targets. Now, that API - we know we need DNS, and I could decide to point the DNS at Cloud Run, I could point the DNS to Kubernetes, independent. So I think these logical boundaries, these very clear contracts allow people to move very fast, because now we’ve reduced the amount of coordination that we have to do in order to figure out what to do next. So as a developer, without talking to anyone on the endpoint team, I can just call Curl and see what data comes back, and I know what to do going forward. So now if I wanna use different tooling for the Kubernetes component - and think about this… You don’t necessarily need to provision the cluster yourself. You could decide “I’m just gonna use GKE Autopilot, which will automatically scale the nodes across the regions based on what my application needs.” So my Terraform provisioning is “Enable Autopilot.” That’s it. No flags, no cluster side, no node pools.

And then maybe another tool takes off to manage things like deployments, deploying to that particular target. To me, that is easy and understand and explain to the relevant parties. And you’re right, if one person had to think about all of these things, we’d think they would have to actually learn a lot, but I don’t think that is the case. When I context-switch, when I go to work, I work on a Chromebook. When I’m doing personal projects, I work on a Mac. If I wanna play a certain video game, I do that on Windows.

Humans have no problem switching context when the rules are clear, once I make the switch. It’s only when we start to muddy the waters by mixing too many concerns. If the frontend team is using Terraform in a way that doesn’t actually make sense, then they get confused and now they’re spending so much time battling Terraform, instead of just saying “Save in the IDE, run some tests, if they pass, deploy the new set of static files to the CDN”, and be finished.

John-Daniel Trask: Yeah, so we actually didn’t start off there. We started off building some other software development tools… And you know, this was where things actually got murky, because my vision for my future was to own a software business. I didn’t actually know what it was gonna do. In a weird way, my goal, my purpose to be the owner of a software company had been with me for about 12-13 years at this point… And once I got there, it was kind of like “Hey, hang on a minute… Now what do we do?” All this time thinking about this, and I kind of got to this point…

So we actually muddled around for a few years, building different things… They were liked products; we could wipe our own face. The business was making money, which was good. But we also did some – we bootstrapped the business, so we did consulting work… We actually had Microsoft New Zealand come to the party on day one of founding our company and give us a quarter of a million dollar contract. And that was really our seed capital.

We built some software for Microsoft that demonstrated how teams should build modern web apps, and they used that as demoware around the world. Now, that actually came out of the relationship that Jeremy had built with Microsoft New Zealand. And this goes back to your point of who you associate with, where are you investing your time. He was putting a lot of his time into supporting the local software development community, running user groups and all of that… So he was well-known and well-liked, and generally appreciated in the local software community. Surprise… It worked out for us.

And to use that famous Steve Jobs quote, “You can’t connect the dots looking forward. You can only connect them looking back.” Jeremy wasn’t supporting the community because he wanted a deal out of it; he was doing it because that’s what he loved to do, and then this positive came. So he had actually put in not expecting anything out, and then got something out of it. And that was hugely valuable to us.

[27:49] Over those few years we also partnered up and built other companies. So we were fairly well-known as being a pretty good software development team, high-performance software, good outcomes… But we took equity in those companies. So we built New Zealand’s largest philanthropic website as the technical partner, where we owned a part of that; we built a business valuation SaaS company, we build an email mining software company… We did a few things like that on the side. That was all good.

In 2012 we decided we were gonna build the error tracking software, and that we were gonna call it Raygun. Now, our company at this point had actually been called Minescape… And I’ll tell you want - nerds naming companies, man… How did we come up with the name? I mentioned we had three founders at the beginning… And we basically all just came up with these names, put them in an Excel file, ordered them, and them we all scored them one to ten, and then we added those three rows together and sorted the sum, and then we just went with the top one, which was Minescape.

And I remember there was this one right down the list, Raygun… And I just remember thinking – well, it was actually Railgun, but anyway… It was from JB, and I just remember thinking “That’s the stupidest name for a company I’ve ever heard of.” [laughter]

Max Howell: Well, you’re totally right that a lot of developers are very anti-crypto, and so it’s been a battle from the start. Hacker News hate me even more than usual. But inside the cryptosphere, it’s very popular. Like, 1.7 million signups is pretty unheard of. And what it turns out to be the case, to my surprise… I’ve spoken to over 300 open source devs over the last three years, just for market research reasons. A lot of them don’t care if it’s crypto or not. They like crypto in the respect that they like technology. Open source devs aren’t as anti-crypto as the others, the rest of the devs. And yeah, I think we have a reasonable chance of showing that crypto is just a technology. We’re not a scam. There’s nothing scammy about what’s going on with us at all. They’ll see that once we’ve gone live and no one’s rugging the token, or anything like that. And it’s all open. That’s one of the beautiful things about Web3. All those smart contracts are transparent, readable. You can see what’s going on.

Luis Villa: There’s no test frameworks, there’s no linters, there’s no – by the way, when you compile it, there will be somebody else trying to persuade the compiler to do things totally different from what you intended… It’s a very adversarial system that is not set up for robustness. Don’t get me wrong, it works reasonably well in a lot of cases, but it’s mostly because of humans – this was one thing that drove me nuts about all the smart contract stuff… Like, contracts only work because humans are around to smooth off the rough edges, bridge the gaps… Like, as soon as you start making contracts into code, you’re just doomed to failure, because of all the failure modes of code that we as programmers know, and that don’t go away when you have contracts; you just have a more forgiving, at the end of the day, execution environment… Because at the end of the day, humans are in the loop, in a way that they aren’t with smart contracts, and in the way they are with code.

Ken Kantzer: Oh, yeah. Auditing, and also just bug bounties. I think I read somewhere that someone found a bug in Ethereum, and the bounty was like $10 million, which is – that’s a lot of money. It definitely dwarfs – the interesting thing is it dwarfs a lot of the bug bounty money in like traditional software. I think Apple three or four years ago made huge news in that they upped their like top bounty, if you find a vulnerability in iOS or something to like a million, and that was huge news, because it was like 10 times more than anyone else was giving out. And now you have the smart contracts, where there’s tons of money flowing through, and the bounties are even higher.

Jeronimo Irazabal: [24:01] Yes. Before giving the explanation how I ended up here… But actually using immudb for an application developer is exactly the same as using a traditional database. You can download the immudb binary or local container, and you will use any other key-value store, or SQL database as well.

So before I joined Codenotary, I was working as a software engineer for IBM, and the last projects were related to digital rights management, and that was related to applied cryptography there for generating the crypto materials… And also, I was a contributor for Hyperledger Fabric. By then also I worked also in an experimental project where we added SQL to Hyperledger Fabric. We added SQL support into the changelogs, actually in the smart contracts. But by then, I was convinced that the complexity of the project was quite big. There were many companies or organizations willing to use blockchain just to be sure or to prove themselves or to their clients that the data was not changed. But then they had to run a very complex system, so I always thought about the possibility to have just a traditional database with the verification possibilities. So to have the same verification capabilities like a blockchain provided, but thinking of singular organizations being the owners of the data, but yet to fulfill with auditory requirements or to prove to their clients that the data has not been changed.

So by then I just started to think about this type of systems, and I got to know about the company and the initial release of immudb. By then, immudb was implemented relying on another key-value store that was written in Go. So that’s where I just started to work,

And related to immutability, I think tampering detection is one of the types of verifications we can do, but there are many other things that are to be included, like what is the latest record that was modified, being able to verify when you are dealing with higher-level data models, like SQL… If you have a database and you have a document like a data model, and you have queries, and you have to verify them. So there is a lot of things to investigate, to explore, and of course, to implement.

Nader Dabit: Yeah, no magic feeling… And again, how fortunate was I to even be able to say something like that, right? So I recognized all that stuff. But I was looking for something, the next thing to excite me. I’d been speculating as an investor in crypto since like 2015, so buying and selling Bitcoin back in the day, and then other things more recently… And one of the ones that I looked into was something called the Graph Protocol. And it was the first time I kind of looked past the token as an investment and tried to understand why it was there and why people were buying it and using it… And when I started kind of looking into that project, the first thing I realized was that they were using GraphQL, and that was kind of like a big part of it… And that was really the first big eye-opening moment for me as a developer, when I started realizing that these tokens and these crypto-economic protocols were actually there for something other than speculation… Because honestly, at that point, even 6-7 months ago, I didn’t understand how smart contracts work. I didn’t understand anything past just buying and selling.

[08:12] So I started diving into why they use GraphQL, and that kind of took me down what a lot of people call this rabbit hole of the entire ecosystem, and Web3, and all of the people, all of the communities, all of the projects and all of the stuff happening. Conferences, all types of stuff. And I became really, really fascinated, and I became really excited, and I wanted to continue doing research. I started reading and I started buying books, and really getting – I couldn’t stop thinking about it. It was one of those things, like “Oh, this is freaking cool.” The reason that these things have become popular isn’t just because it’s like another currency, but there’s actually other stuff… And when you start peeling back the layers and seeing all this stuff, it really spoke to me as an engineer, because the problems that they were solving and the stuff that they were doing seemed so much more fun and interesting than anything I’d worked on really in my career, ever.

So yeah, at that point I did start considering “What if I wanted to make a move in this industry?”, but it wasn’t really anything I took too seriously, because of a bunch of different reasons… One of those being it just seemed so risky to me to turn down a half a million dollar type of opportunity that I had on the table to maybe join an industry that I knew nothing about; and I had zero experience… Like, why would anyone hire me anyway? And if I did, would I be successful? That’s kind of where I was maybe a couple of weeks before I ultimately tried to get into the role I’m in now. So that’s hopefully an overview of how I decided to even consider this area.

Preston Van Loon: I think the amazing part about this technology is that while there are no rules, you really get your opportunity to define those rules. So you’re able to develop what are called smart contracts, or these decentralized applications which - everyone can read them and understand what’s going on, what are the constraints of this protocol that you’ve launched on Ethereum. And if you’re talking about true decentralization, there are these concepts where you can have what’s called a DAO (decentralized autonomous organization), which is a group of anonymous people that have voting rights in the protocol and are able to make the upgrades and change the rules.

With Ethereum, or really any blockchain, it’s like a write-once-and-read-many-time kind of style database, and you’re not able to remove or delete or change anything unless that’s part of the code that you’ve written. And as a participant or user in your protocol, I can see that and I can base my risk assessment “Do I wanna be part of this application?” I can read the code and find out for myself.

Tim Coulter: [35:49] Yeah, so right now the library that I’ve described so far is what we call Truffle Contract, and this is actually a library that’s existed for quite a while. This one turns everything into a promise, so you can use async/await with Promises, but it doesn’t use callbacks because what generally happens when writing a front-end for a smart contract application, or what you generally do is you say “Make this transaction, then do this transaction” after a specific request happened. There’s also other pieces of that, so not all interactions with the blockchain are right, not all of them are transactions. Sometimes you can call a function that will get executed, but it’s actually just there for reading data, so it doesn’t get recorded on the blockchain, it doesn’t cost you any Ether to do that… So a lot of the times it becomes like “Perform this transaction, read some data, perform another transaction…” and so this library provides it to you as sort of a promise chain - “I’m doing this, then I’m doing this, then I’m doing this…”, so you can have easy control flow.

Something that we released today actually is a library called Drizzle, which is the next evolution of this idea. The JavaScript world and the front-end world has been moving away from effectively home-grown transactions, direct interaction with the server, and instead you’re moving to a React and a Redux architecture. Well, wouldn’t it be great if you could take those Truffle artifacts that Truffle creates for you, throw them into a library and you’ve all of a sudden got a Redux store that is tailored to your smart contracts? That’s what Drizzle provides - if you like the React and Redux world, there’s almost no work involved to fit your contracts into your front-end application.

Kevin Owocki: Oh yeah, I’m [01:05:59.20] in this that I forget that I’m using buzz words, even when… But anyway, so Truffle is - think of it sort of like Ruby on Rails, but for the Web 3.0 ecosystem. It allows you to manage your smart contracts and your tests for them, your migrations, your deployments, stuff like that. So it just makes Solidity and smart contract programming way easier.

Brian Behlendorf: You know, some of these idealists have certainly seen our technologies used to create some really horrible things, so you have to be very cognizant of that. I think for businesses it means that a lot of the processes and rules about how markets work that are enforced by lawyers and enforced by people having to watch what goes on and can’t automate what they do – there’ll be a lot more automation, and because of more automation, there’ll be more audit ability, and it’ll just be either impossible or much harder to commit certain kinds of fraud.

Hopefully, it means more stability in the financial markets, because I think the mortgage crisis had that industry used, distributed ledgers and smart contracts to define these structures, we would have had much less of a run-up and also a bit more of an orderly landing…

Anna Derbakova: So if you just say “What is blockchain?”, blockchain is an architectural concept almost; it’s something that you build on top of, but there’s different types of blockchains. The blockchain that supports Bitcoin is the first one that was out there, but then we got other types of blockchains, like the Ethereum blockchain, and that’s huge now. A lot of people are very excited about it, it provides some awesome capabilities to write smart contracts, and really complex ones - so that’s a blockchain too, but they are different blockchains. A lot of the underlying ideas are the same, like how you store your blocks, and things like that, but…

Jerod Santo: Computer programs that govern the dynamics of value, and as we’ll find as we get deeper in this conversation, there seems to be a divide in the Ethereum community - and in the cryptocurrency community - over the idealist side of that, which is that the program has the final say, because that’s what the agreement is, versus what you might call the pragmatic look at it, which is that the program is supposed to represent the agreement, but programs have defects and bugs as we’ve found out, so it’s less black and white.

Before that, let’s tee this up and we’ll take our first break. So we have smart contracts, we can program them with this Solidity programming language (or set of languages) and run these things on the Ethereum Virtual Machine… But the question that comes to my mind, and I think the one that’s interesting to our audience so much as developers is like, that sounds great and all, but what can I build? What does this open up? What are some applications that now I can create using this that I couldn’t create previously? Don’t answer that, Gavin, we’ll take a break and you can answer it on the other side. We’ll be right back.

Carmen Huidobro: What a plug. I love it. And I don’t know, there’s just something so joyful about like playing around with technologies in a way that’s, for example perhaps annoying… Or in the case of the Bad Website Club, what I love about that initiative more than anything is that it’s sort of like a leveling ground for emerging developers. And especially in this day and age, I find myself – I will admit, I have found myself contemplating about how much of an urgency there is to get into tech given that there’s a market issue right now, and how do I do this in a way that is ethical and more lucid, right? Because of course you’ll have folks come in, they want to learn to code, and they’ll be like “Okay–” Like, I remember when we started doing this a couple of years ago, they were like “Cool. How do I go from here to being a smart contract developer as soon as possible?” And I’m like “Gosh, okay… So first of all, I’m not a smart contract developer, and I don’t know anything about it. And the only thing I recommend with Web 3 is don’t pay for resources when there’s really good ones available for free.” Cool. But yeah, so I think doing so in a way that is empowering for developers is what I find really important.

One final story before I let – because I don’t want to keep y’all any further… One of my favorite experiences when I was teaching at the Bad Website Club is I was doing one of the freeCodeCamp exercises, and if you’re familiar with them, there’s one that is notoriously difficult, where you have to manipulate a particularly complex JavaScript object. And I was streaming, doing these exercises and explaining them as I go, and what I found was that I just – my brain shut down, and I couldn’t process the exercise anymore… And I got stuck. I mean, super-stuck.

[01:09:18.23] And I was there, I was struggling, and folks in the chat were like “Oh, just do this. Just do that. Oh, I’m so lost. I’m so confused.” And I felt rotten. I did figure it out in the end, and I wrapped up, and then I went down and I had a very stoic little cry… And I did get a DM later from one of the participants of our bootcamp, and they said “Hey, Carmen, thanks for the lesson. That looked really hard doing that. I just really wanted to say thank you so much for showing me that it doesn’t all have to be perfect. Even a senior software engineer like yourself, who’s been around doing this for 15 years, gets stuck, struggles. And what you showed me more than anything was your problem solving methodologies, which goes way beyond how to understand complex object manipulation.” And I find that empowerment the most valuable thing to do for us, especially as we think about how these tools are going to empower the next generation of software developers. And I’m done rambling.

Jerod Santo: This audience is software developer; we are pedantic, so we wanna have the specifics laid out… In code, if possible. Can you put it in a smart contract, Adam? That would be appreciated.

Kevin Ball: If there’s no mechanism for changing, for example, a smart contract, because conditions have changed - like, what use is this situation, right? And in the non-software-as-law world - recourse, you end up going to court, you end up doing whatever, and interacting with humans who can look at a situation and say, “Oh yes, here are these things that we didn’t account for, and yes, we need to deal with that, and here’s the mechanisms we have for doing that.” And that whole ecosystem of discretion, of “We’re actually trying to do something real here, and we acknowledge that the world is variable and unpredictable, and we have frameworks for dealing with that” doesn’t exist in this world.

Jerod Santo: This audience is software developer; we are pedantic, so we wanna have the specifics laid out… In code, if possible. Can you put it in a smart contract, Adam? That would be appreciated.

Peter Bourgon: This is so good. This is so good, because it gets to the core point of everything here, I think, which is package management. Do you know any blockchain people? Like, Bitcoin maximalists, or something? Some of them are like “We can express government as a smart contract. We can define an algorithm which will solve all human problems.” And it’s like, that’s broken into the same way. Package management is not a technical problem. It is a social problem. It is a human problem. At the core of it is a human social thing, and the technical stuff is just the ancillary stuff you have to do to solve the human problem. But it’s my belief at this point that the people building and maintaining modules don’t understand that. They think it’s a technical problem, and the human factor can just be coerced into the right space, and that’s absolutely backwards. That’s backwards. And I don’t know how to convince them… I’ve been yelling at Bryan Mills in the Modules channel for like a year now, and this is not effective. This is probably my mistake, I’m not very good at human stuff, so I’m probably not the guy to solve this… But I don’t know how to do this. I don’t know how to do this.

Nathan Sobo: So my understanding of this situation - and it’s basically like we created Electron and came up with this innovative model of doing an editor in web technology… And there was this team in Switzerland at Microsoft that was doing Monaco, and doing like Visual Studio Online; like a browser-based version of the editor. That team included like Erich Gamma, who worked on Eclipse etc. But when they saw us doing Electron and we open sourced Electron, that was this opportunity to like “Oh, we see Atom taking off… Let’s sort of take this tech that we’re building on the web and fuse it with” what was at the time called Atom Shell. I don’t think we’d renamed it to Electron yet. And yeah, that kind of gave them this head-start.

So I think there was a big advantage of coming with some existing tech, some existing experience, and then having us sort of taken all the arrows figuring out how to get Electron done… It was like a pretty good flanking maneuver. But then I think a lot of Atom’s wounds, honestly, were like self-inflicted on a number of levels. I made some mistakes… And I think one part of it was just never having clear leadership.

[18:05] I had been there from the beginning to almost the end. I was the longest-serving member of the team, but I never had any official authority on the team… Yeah, so that was part of it. But there was also just having started the Dark Ages - like, we started in CoffeeScript. It took us too long to switch to JavaScript; we should have gone straight to TypeScript. So just using inferior tools… And it’s never the right time to like take a step back and be like “We should level up our tools. We should pause, do what we need to do to switch, we’re going to take a short-term hit, and move forward.” We never really pulled the trigger on that, so that was a mistake.

And then we just focused so much on this very broad definition of extensibility, of like “You could come in, run your JavaScript on the main thread, and you could do anything.” And I think Microsoft’s approach was a lot more conservative, and a lot more focused… But of course, they were coming from the position of working on IDEs, working on the TypeScript’s tooling… And so I just think, from their perspective, the language server protocol, and focusing in a more pinpoint way, precision way on the specific types of extensibility that would matter, gave them a big advantage… Because it’s hard to maintain all that surface area; it takes a lot of time and energy and resources. And then early decisions we had made that weren’t the most performant were sort of sealed in concrete by these API contracts we had made…

So a combination of just like – yeah, I don’t know… I mean, a lot of the things that made Atom exciting were also the things that made it tough to compete, which was just the freedom that people had to do anything. I remember supporting a team at Facebook working on Nuclide; Atom was the basis of all the tooling at Facebook. And that was a much bigger team than our team. It was just like a weird situation, right?

So with Zed, we were determined to do things differently… One thing is have a clear business model, and a clear objective from the beginning, so that we could really start a company that we would aim to be self-sufficient around this tool, that had a clear leadership structure, that sort of me, Max and Antonio, my co-founders and I were clearly in charge of… So that if a decision was made or not made, that was like our responsibility, and there was no doubt about that.

So from a structure perspective, that was one thing. But then I just felt like we had really reached the end of the line on Electron and on JavaScript. Like, if you think about a tool as sophisticated as a code editor, you don’t have time to like pause and clean up garbage, or use a language that’s just like kicking out garbage constantly, and confined to a single thread, unless you’re going through some pretty complex gymnastics to spin up worker threads, that don’t have shared memory etc. It’s just like – I don’t know, I thought it’d be a push to JavaScript to the absolute limit, and then beyond, doing things like V8 snapshots to start up faster… And just – crazy. We just were like “We can’t do this anymore.” So switching to a better language. Rust had hit 1.0. in the meantime, and I’d been following Rust for years, being like “This seems like it could be the right tool to solve the kind of problems we have.”

And then there was just like making sure that we had a solid core product, that met a lot of people’s needs right out of the box in terms of the languages they wanted to work with, interacting with the language server protocol… Just having all the pieces in place, and fast and polished and dialed in before we even worried about extensibility at all.

[22:08] When the iPhone came out, it did a couple of things. It made calls, and you could browse the web. And it wasn’t until a little bit later that they launched extensibility. And so that’s the strategy we’re taking as well. I mean, I am one of the creators of Atom; obviously, I love extensibility. That’s a huge value of life. But Zed, as it comes out in this initial beta launch, really, it’s not extensible. That’s something we’ll add. But we want to make sure that we get that core experience right, so that when we start opening up any of those contracts, which we’re going to be much more judicious about, we know that the core is solid, and we don’t need to go back and change anything and violate those APIs.

Adam Stacoviak: Aligned. Yeah, that kind of investor would have to be not only money, but smart money. Like, connected, networked. It would have to add value to the giveaways. You know, “Can you help me connect with better publishers, have deeper relationships with publishers? Have longer contracts, more reliable revenue, etc?” That’s how I would view that kind of investment.

Sean Griffin: Basically, I wrote a 3D rendering engine; it turns out when you do one of those, there’s a lot of 3D rendering engine contracts that come out of the weeds. And why I ended up having to write one is a story that’ll take me ten minutes to tell, but basically out of necessity I ended up having to write one for a different project, at sort of the beginnings of WebGL being supported by browsers, and then the projects kept coming.

[40:10] Anyway, so we’re on this project that was C++, and it was for mobile, and because of the polycount requirements we couldn’t take the overhead of a framework like Unity. So it was a kind of raw OpenGL engine. Rust had just gone 1.0 at this point in time, so for fun, on nights and weekends, I sort of ported it to Rust, just to get a feel for the language. And got sold on it if nothing else as a replacement for C++, because I had a segfault that was having a bear of a time tracking down, and this was like me just one-to-one, as close as I could porting the C++ code over. And this was not modern C++, this was a guy with a decent understanding of C trying to write C++, so not using templates in any meaningful way, certainly not using smart pointers the way I should have been…

So it was generally pretty easy to directly port to Rust, but then I finally figured out where the segfault was coming from, because it wouldn’t compile. So I never shipped the Rust port, but just the act of porting it fixed a bug in my code. So I still [unintelligible 00:41:23.13] a replacement for C++ immediately there, but I don’t generally write C++, because I’m very bad at it, so that wasn’t terribly interesting… But I did realize how amazing its type system was as a part of that project.

The Rust type system, to a certain extent it is what if you took Haskell’s type system, removed higher kind of types, but in exchange gave certain other types of genericism in type class instances, and you end up with the Rust type system.

That was really cool, and so then I started wondering, “Hm, I wonder if Rust could work as a high-level language…” So Diesel was me trying to answer that question originally.

Ashley Jeffs: But yeah, it got to the point where it was like I’m really enjoying the lifestyle of just spending as much free time as I want, not really having like serious obligations… Because I had like contracts and stuff, but that was easy to keep going. It definitely wasn’t like a full-time job. But I was definitely working a full-time job in terms of Benthos stuff, and support on the channels, and things.

It got to the point where it was like – this project is going to reach a point where it’s going to be too popular amongst enterprises for other companies to ignore that. They’re going to start building up a Benthos shop. And then I’m at the risk of “Well, if their team is bigger than my team, and their team is more productive than my team, we’re just going to lose all of our momentum. Like, they’re just going to basically maintain a fork, and I won’t be able to control the direction of the project anymore, or any of those things. I won’t get my something out of it.” So I figured “Well, I’ll call it now.”

And it just so happened in 2024 that it really got to the point where in the data engineering industry people were taking it seriously, o there was like lots of companies suddenly discovering it, and realizing “Oh, hang on a minute… We can get rid of Kafka Connect.” And desperately then clinging onto it. It got to the point where it was like “Okay, well, I’m freaking out now, because if I’m still just me, then somebody else is going to fill the void. And also, I still have to support the community on the various community spaces… So I’m just going to ignore everyone. I kind of have to support everyone, to an extent… And it’s just going to take up too much of my time. I’m going to be immobile and I won’t be able to do anything. So I was kind of getting a little bit – freaking out.

And then it just so happened that it got to the point where it was like “Okay, there’s enough industry interest in this that we can arrange something.” And then Redpanda reached out, and they had a great plan…

Avindra Fernando: [01:00:25.09] Getting started with it. Learned a ton there. Constant innovation, constantly grinding… Great, great, great time there. What I was thinking to myself at that time was “Okay, now I’ve got the product startup perspective. How does services or consulting work? Let me go experiment that.” So I joined a services startup, which their motto was consulting. A couple of guys, amazing, amazing dudes. Got to work with them, see how they negotiate contracts, bring in different contracts…

One of the contracts was so interesting to me. I was working on an app for someone; that was his hobby. He wanted this idea of a virtual bar. So he was mapping out all the bars in the cities that they go to, and would give the ability for someone to purchase a seat in the virtual world, which was a fascinating idea. I was like “People pay for this stuff?” It’s like, yeah, this is cool stuff. So I got really motivated by that.

And then I eventually decided, “Okay, I’m just going to start this journey on my own and see how things go.” So that’s fast forward to 2021.

Adam Azzam: If I return back to – I gave like five or six theses of things where I was encountering failure a lot… And I think that some of those things are sources of failure that many folks are familiar with. I’m calling out to an external service, and the service is flaky, and it’s bad… That’s existed forever. As long as people are building data pipelines, upstream sources being flaky - that makes sense. Hitting deterministic errors of like - I’m ingesting data, but somebody added a new field, or they removed a field, and I’m dependent on that, and now my pipelines broken. Or I’m scraping something, and target.com, instead of labeling the name of their product with a div whose name or ID is this, they’ve changed it to that, and now all of my data is corrupted, because I couldn’t detect that in real time.

[00:16:12.02] And then the last piece is the loading part, where you’ve gotten, you’ve cleaned all your data, and now you want to go put it in a persistent place where you can go query, or do analytics on. So that’s like classical stuff. The extraction, calling out to an external service, the transformation that you’re doing deterministically in the loading. So the sort of ETL business of all of that. That’s like a persistent problem that exists far before Gen AI, or ML workflows… And that’s sort of what’s been category-defining for workflow orchestration. That’s like the single case that people usually break in an orchestrator for is when they’re doing ETL type jobs.

I would say that what’s unique these days is that since workflows in LLM land are now more dynamic, so you really can’t plot out every single thing that’s going to happen from the start, they are now basically – we’re dealing in English now, or you may not know the full space of their responses at the beginning of a workflow. So I think that LLMs introduce a dynamism component that’s hard to reason about, and has kind of escaped classical workflow orchestration.

I think the second piece is that the nature of errors here just feel totally new. So the fact that you can ask an LLM for a particular shape of a response, and then you can get parsing errors out on the end of it - that’s a new source of failure that’s now buttoned up with, say, some commercial LLM providers that give you very structured, guaranteed output, so you don’t see as much parsing errors… But now – I like to joke that you can lead an LLM to JSON, but you cannot make it think, where - like, you can say “Look, I’ve got this job description, and the title is… I’m going to give you a schema that says the job title, the location”, or whatever. And sometimes you’ll say the title, it’s required, it has to be a string, and the response you get out is “I’m sorry, but I could not find a title.” And when you’re doing this at tens of thousands of jobs, now you also have to reason about “Okay, I got now the parsing error, but the error was pushed down deeper in the stack.” Now there’s data quality errors that I have to reason about, that I didn’t really have to account for in last generation’s ETL. Things were much more deterministic, we had stronger contracts about what you’re going to get.

And then I would say the last piece is – what makes this harder is that so far… And I hate to keep throwing out random definitions. I hate being a merchant of complexity, and talking about why things are super-hard… But trying to at least motivate why this is a new source of difficulty, right? Like, we’ve got tools to handle this, but why do we have these tools in the first place? And the last piece is around like agentic workflows. Now, this is a buzz term… So what do I mean when I say “agentic workflows”? Everything that I’ve talked about so far, of like I get a document, and I want to extract stuff from it. Or maybe I want to classify it, or I want to summarize it. These are all sort of modern takes on classical ML problems, right? You don’t have to bring as much training data, you don’t have to train a model first… You’re basically throwing the weight of the compressed internet at every problem that you come across.

But with agentic workflows, what I mean by that - those are things that operate in a loop, are able to call out to external tools should it choose to, and can create, refine and reflect on its own plan, which means when you’re orchestrating agentic workflows, you have to do this interplay between who’s doing the orchestration… There are some times where I’m coming correct with a plan, and I’m saying “First extract this topic, then classify it, then write an email and send it off”, but now I have to be able to add resiliency to a workflow that I’m unaware of at the beginning of it. So if it decides “Call out to this tool, call out to this API”, I now need to be able to reason about resiliency for a workflow that I don’t have any visibility to at the beginning.

And so I would say that those last three pieces around parsing, around not knowing your your full decision space at the beginning, and then how that feeds into now having to hand off some bits of orchestration to the LLM to create its own tasks that you now have to execute, I think that’s what makes the new generation of orchestration a much harder and a much more interesting problem.

Break: [00:20:44.08]

Adam Jacob: I’m not. I was the guy who when they tried to fire him, for example – so they tried to kick him off the board, because he was a bad guy. And so the people that I had actually worked for - I wound up working there through acquisition. This is actually an interesting story. So I wound up working there through acquisition. I was working for this company called GoToNet, and they got bought by Infospace.

[00:23:59.08] Then what happened was the GoToNet guys basically staged a coup. Tried to. And I was the lead systems administrator for all the corporate stuff, so I had set up all the automation for your accounts and stuff, your email. And so they had told me in advance that they were going to have this board meeting, and that they were going to fire the CEO, and that they were going to call me. And in that moment, as soon as I got that call, I needed to turn it off stat, because I didn’t want him to leave the boardroom and write an email and be mean. So I had to turn everything off immediately.

So they called me, and I turned everything off, and then time goes by, and I get another phone call, and they’re like “You’ve got to turn everything back on.” And I’m like “We don’t really have a back on… You know?” I have the off automation, but there wasn’t really a back on, so it’s going to take a minute… So yeah, it turns out that what had happened was he had staged this coup, and he threatened basically to take all of the revenue, all the contracts, many of which were in his name, across the street to a new company that he was going to start, and bankrupt them. And so the board then decided to fire the guys that I had worked for, who had tried to kick him out, and then it took years for him to finally be replaced. His investors just relentlessly pushing him out of that company… Which eventually died the death it deserved. It was a company that had no real revenue, had no real products, but thousands of us worked there, doing stuff. It was crazy. Yeah. That was what it was like in Web 1.0 for a lot of people.