Jerod Santo: On acid, yes. Which - maybe those are the models he’s working with. But you know, a thing that you don’t ultimately know what it’s going to do. Now, you can train, and fine-tune, and guard, and watch agents watching agents all you want… However, we’ve seen at scale the internet operating, distributed systems at scale - things go wrong in ways that are sometimes very quick, very catastrophic, and compounding and cascading.

[00:52:09.02] One thing that I think about is some of these stock market trades, when you have quick corrections or crashes is because you have software making margin calls and trading with software programs, and eventually what happens is the New York Stock Exchange actually just stops everything and is like “Let’s chill out here.”

Autumn Nash: There was a press conference in Louisiana a few years back, and they were talking about the possibility of getting rid of fees for tickets and things. And they were like, “How do you feel about no longer–” They weren’t going to get rid of the fees, but they were going to get rid of the jail sentences, because they were saying so many people had these nonviolent crime, very nonviolent, like just tickets, stupid tickets, but they were compounding so much because people were so poor, they weren’t able to pay their tickets. So so many people were getting in prison because of their fees, but then they were losing their jobs because they were sent to prison. So you’ve created this cycle. And they asked, I think a warden of the prison, and he goes, “Well, who will wash our cars?” They were talking about people. Like, he was cool to basically enslave people because of a $20 parking fee, or just stupid fees, or stupid tickets, because he wanted them to wash their cars. I was like, “What?” It was just wild.

So I think that there’s a lot of cool things that we can do, but it’s very scary to see how we are going to societally impact and oppress the same people who have been going through this over and over and over again. Let’s go put it into things that are making our lives more efficient and easier, but let’s not trust it to do these things that we already as humans have a hard time doing.

What’s your next link?

John Nunemaker: And I would have to have a good team probably already, or I would have to have a friend who knows the exact technologies being used, that could just step in… Because the marketing and things like that don’t matter. You can standardize that stuff at the top of the holding company, or whatever. It’s just the tech that I would like somebody – again, I was listening to a podcast on Inquiring Minds, and it’s like a guy who buys manufacturing companies. And I was just like nodding my head profusely the whole time. The first podcast ever I’m probably going to listen to a second time, and maybe a third time… Because I was just like exactly what this person is saying. He’s like “Look, I’m not going to buy a manufacturing company that I can’t step in and like run the machine if I have to.” He’s like “All the way through, I want to know that worst case I could go in there by myself and I could keep the thing going.” He’s that conservative on stuff. But yet, they’re compounding it 20% or more; they’re doing great in growth and all that kind of stuff. And so I’m like, “That’s in my head, I guess.” In my head it’s like “I should be able to step in, if push came to shove, and like run the thing”, and I just… I don’t know, maybe I’m too old, too many gray hairs to like totally switch languages and frameworks, and really feel like I could be – it would take me some time, I think, to be successful doing that.

Emily Freeman: No. Like literally removing junk, buying perfect little containers, labeling the containers like a psychopath, putting things in… I mean, it’s beautiful. But yeah, it’s a whole process. And I’m just now to a place where it’s like “Oh, I have energy. I actually like the industry again.” Because I was real bearish on tech. The companies, the people, honestly… It’s just like, “What are we doing here?” And that’s not a good attitude. And certainly not a place for creativity and hopeful inspiration.

And so yeah, I needed a break. But I think when you’re in burnout, you have this thought of “Oh, I’ll just take a vacation and it’ll be fine.” It’s like, no. This is years of compounding stress on your body, on your mind, on your soul… And it has to heal. That takes time.

Jerod Santo: [01:00:13.03] So this wasn’t a moment, but I think that – it’s taken time, but it has been profound to me, learning the power of just consistently showing up; like, just the consistency, and how compounding consistency is when you’re building something. I think that we’ve had times where we’ve been less consistent, even with just production. So I don’t know if you listen to podcasts like I do, but I’m also a podcast listener to this day…

Kris Brandow: Yeah. It’s a compounding knowledge. Yeah.

Techno Tim: It is. So yeah, it had this compounding effect of less power, which - you know, those servers then generated less heat… And so I have an enclosed server, that enclosed server rack, and it has a temperature control unit… So as the heat would get past, I think, 87 degrees, it would kick on those fans… And so now that whole thing is no longer happening; it’s all being passively cooled. Well, passively from the rest of my house, but…

Daniel Whitenack: Yeah, yeah, that makes sense. Actually, early on when we were building the Prediction Guard backend, this was actually front of my mind, and has since kind of evolved a little bit. The fact that there’s all of these models and I want to choose the right one for my use case - you can very much automate that process, and it’s actually still implemented in the Prediction Guard backend, where you can give some examples, and evaluate a whole bunch of models on the backend.

I think where this is headed though, and where the Prediction Guard system is headed is making it easier for people to get output from multiple models in a typed way, because they know how to do the evaluation. They’re familiar with this sort of thing, whether you’re a developer doing sort of integration tests or unit tests, and you’re checking and you’re asserting certain values, or you’re a data scientist that’s running a larger-scale test against the test set, people kind of know what they want to do with that sort of thing. What they need is an easy way to get that typed output from multiple models. So like if I have a test set and I’m comparing two scores on the output, like float numbers, I need to get float numbers out of a whole bunch of different large language model was to compare them to my baseline or to my test set. Right now that’s very difficult, because all of these different kind of structuring/guidance/control systems work not for all models, and they don’t work in the same way for all models, and you have to implement it for all of the models. And so it becomes this compounding problem to figure out how to do that.

And so how we’re approaching that with the Prediction Guard system is there’s a standardized API to all of these different models, along with the typed and structure control on the output. So I can do a query that says “Give me the float output for these 100 prompts, using these five models, and then I’ll just compare all the float outputs and figure out which is the best.” That’s not the hard problem. It’s the getting that structured output from the variety of models in a robust and consistent way; that’s actually a more difficult problem.

Mihail Eric: No, no, no… This will be soon. But it’s a really interesting thought experiment, and again, one that is closer than I think we believe. This idea of what happens when the bulk of these datasets are actually generated by systems. And I can see some positive effects and some that you would be like “Okay, this might be a questionable downside.”

On the positive side, these models, like the generative ones, especially of the language variety, are like very good. I mean, grammatically correct. They’re like very semantically good in terms of what they output. Does this mean that now the bulk of the data that it’s being trained on is just going to be like way higher quality writing than you would typically find in a typical Reddit post, or something just in the deepest corners of the web? And then after training on data that’s significantly better, you have this compounding effect of like “Well, now the data I’m training on is better”, so the model is only getting even better at writing of some kind, and then we mix it in with some other more diverse writing… And then it’s going to continue to compound on itself in terms of quality?

The downside - I mean, one you could hypothesize is that these models are only able to generate certain kinds of distributions of data, right? Only certain kinds of things they can write about or talk about. So now, when you’re injecting all these training sets with like a very skewed distribution of topics, of ideas on these topics etc. How do you ensure that actually you’re still giving your model enough of a versatility and what it sees is next generations that have been trained on these datasets, to ensure that it’s still a general-purpose model? What if 90% of the content that’s put out there is just really bad marketing copy? Facebook ads, or something.

Mat Ryer: [09:58] Nice. Okay, well, I’m gonna tell you about a horror story in tech of mine, that happened quite recently… I have this project which interacts with Twitter, and it interacts with the Twitter API… And so it polls results and then compares them, and stuff. And that’s just one of the things it does at a regular interval. And then what happened recently was something happened where the API key changed, and that request failed. And because of the way I was doing it in GCP, it meant essentially that it would retry. And because it was scheduled, it kept compounding. And this ran up a $1,000 bill for me, for yours truly… $1,000 given, paid, gone… So that’s a bit of a tech horror story. And the advice for me –

James Long: Yeah, this was one of the things that was like a rare combination of events; I implemented it to do to get to a certain need, and then there were these other compounding factors that sort of played out, which was surprising in a very, very good way. The reason is - because IndexDB is just very, very slow. That’s one of the critical problems with it. When you’re doing more than a couple of reads and writes, it just gets very, very slow. Especially in certain browsers. I think Chrome - I can say this with confidence, actually, Chrome was one of the slowest ones. It was pretty bad.

You can use transactions, and you can sort of manually optimize it, but it gets very difficult to optimize, especially when you’re doing things at the application scale. And so with SQLite, what it does is because it’s writing down these 4k blocks, when you make a bunch of writes, 4k is actually a lot of data, and you can do a lot of updates. You can update thousands of rows in SQLite, and it will end up only having to write two of those 4k blocks. And so what you end up with at the file system - or not at the file system layer, but at the browser layer, at the IndexDB layer, is two writes. You write these two 4k blobs. And actually writing those 4k blobs, or writing one 4k blob is fast; it’s like something about the actual overhead of a write in IndexDB that is really, really slow. Writing 4k, that amount of data is totally fine. It’s the same as writing like two bytes. It’s just something about the actual write call, and the transactional semantics and all of that that is just really, really slow.

So if you update a thousand rows in Absurd SQL and just use SQLite, you automatically get that batching down to just two IndexDB writes, instead of doing a thousand IndexDB rights, which is just incredibly slow… Unless you do it in a super-specific way; but that means that your entire app, if you’re writing it against IndexDB, has to be aware of these very specific pitfalls of performance. Whereas with Absurd SQL, you get all of these improvements for free. So you can just do as many writes and reads as you want to, and it’s just super, super-fast. For reads it’s the same thing. You can read 5,000 rows and it ends up only doing three reads.

If you look at the graphs on my announcement post, it was insane when I actually benchmarked it out. The performance of SQLite just grew as a huge curve, and then if the y axis is the milliseconds, you’ll see IndexDB is just this small blue line as the rows are getting larger at the very bottom, but SQLite is still going all the way up.

Connor Sears: That’s really interesting, I didn’t know he said that. I don’t know if I would use those words, but making sure you’re focused on the right thing is what keeps me up at night. Or when I feel like we’re not focused on the right thing, and I’m at fault for that, because I sent us all down this path, and maybe we’ve learned a little bit more. Oftentimes – again, this goes back to the… At the end of the day, it’s pride and an inflated sense of self when you feel, especially a CEO, whatever that title is, is inflated enough itself. But you’re taking all the responsibility of a decision, and if at any point you feel like, “Oh, I can’t change my mind,” or “With new information, we can’t shift”, that’s when you’re just in your own head. And honestly, it is pride that like, “Oh well, everybody’s expecting me to have known it before.” No, they don’t. They don’t expect that of you. I’ve always respected leaders, and what I hope the people that I work with see in me from time to time is, “Hey, there’s new information. We’ve learned a little bit more, and we need to change things.”

[01:27:43.23] So yeah, the things that keep me up at night is usually the things that feel out of your control, but actually are, and so you just sort of have to walk yourself back from that. But I do think there’s some merit into the sequencing thing. I don’t know what his answer exactly was, but when I think about it, you’re lining up dominos; you’re hopeful that this is compounding value and impact on the business as you go. So you’re afraid you’ve made the wrong bet. And there’s ways of de-risking that. I think the way you do it is talking to customers, talking to your team, walking through, and playing out the expected story in the future.

It’s one of the things - I know that’s played out now, but Amazon always writes the press release, for instance, for a feature. Well, I think you can do that and project it out into the future and say, “What are all the things that this is expected to do?” but also, do like a premortem. I know people do post mortems all the time, but like “How’s this going to fail? What are we going to run into? What are all the different ways that you think we could fail going forward and how can we mitigate that?”

I think honestly, planning… And when I say planning, I don’t mean like a six-month process to plan, or a business plan necessarily, but just talking through that… It helps. Again, we have open, transparent conversation, and I try to record my raw thoughts sometimes, and make it clear when they’re wrong. “I don’t have a high level of conviction around this idea yet, but here it is. What if we did X, Y, and Z?” I think those are the sorts of things that help with that quite a bit.

Peer Richelsen: Yeah, absolutely right. It’s a mix of compounding effects, like every day trying to do your best, and really small iterations, and growth and revenues growing exponentially, usage is growing exponentially… So eventually, that by design ends up being a big company.

And the second thing is there’s some people who overly obsess about making a million dollars or making 10 million or having this fixed mindset of goals. And for me, revenue is just the outcome of a very good product. It’s an outcome of solving a massive pain, in a massive market. And if you check both of the boxes and you do a good product in a massive market, it’s really hard to not make revenue. If you really take a look at the companies who solve a massive problem, they may struggle to turn a profit because they have huge costs, but it’s really hard to solve a bananasly big problem and not make any revenue. So for me, it’s more like the outcome, not the goal.

Mat Ryer: Yeah. I see. By the way, just mentioning another taboo subject doesn’t cancel out the other one. You’re just compounding your crimes. I asked Daniel and Michael to find a list of the things that they’re sort of excited about, or interested in, that we can go through and talk about… And obviously, Michael worked on module workspaces as well, so we’ll carve some time out at the end to talk about that in particular. But Daniel, maybe you could kick us off… There’s a really interesting one that to me seemed like a silly, unnecessary helped, but it turns out to be actually quite worthy. That was strings.cut. Could you tell us about that?

Joe Percoco: One word, which is the word “sequencing”. There’s no question that the next Fidelity will be built. The question is, what is the execution path to get to the other side of the river, that is a compounding, wildly profitable enterprise that will outlast me personally? There’s a path and configuration where you go Stone A, Stone G, Stone B, then stone P, and you make it to the other side of the river, or there’s your A, then B, then G… You sort of see what I’m saying?

Paul Copplestone: Yeah, I mean, usually what happens is that – you know, it’s got this compounding effect… So you might get on, say, GitHub Trending… I don’t even know how that algorithm works, but somehow we get onto GitHub Trending usually on these weeks; or on Hacker News, we try to make sure that we’re always publicizing the Postgres stuff on Hacker News, so people will see Supabase multiple times that week.

Twitter actually is the main channel for us… So people get this sort of flooded Twitter feed of Supabase mentions or launches, so that’s always good… Just to make sure that we’re top of mind during that week.

Sam Lambert: [01:04:15.09] Yes. That astounds me, honestly… And again, it’s like that compounding improvement daily, and just the pace. We talk about pace internally… Pace is an incredible competitive advantage. I think any company can be quick and develop something quickly for three months, or whatever, and then it becomes reckless, or whatever. We obsess over having a pace that is aggressive, but short-footed.

Asim Aslam: Yeah. Do it cross-team, cross-org… And then think about the velocity of development and compounding value for the world, where rather than everything being siloed in a Google, in an Amazon, in a Facebook, in an Uber, actually the next biggest things are built by a small team. There are dozens and dozens if not hundreds of these companies all running on a single platform where they all share that, and they can all go compete against a Google. They can all compete against Facebook, whoever else.

Ramiro Berrelleza: Yeah, I would like to extend on what Simon said, because we kind of went through the same in Okteto. I think there’s a misleading boost you get at first with things like Node, where you find a library that just does exactly what you need… I like the verbosity of Go, because (touching on what you were saying before) it kind of forces the developer to be more intentional on what they’re doing. For instance, we wrote an SSH server for some of our internal components, and if someone had to express very carefully what the server is doing, what it takes as an input, serialization, all those things - it makes it a lot easier when anybody else is looking at that codebase to understand “Okay, this is what’s going on. This is what we’re trying to do.” And that as a product and as a company, as you have more people, has this compounding effect, where - sure, the first developer maybe it took her an hour instead of ten minutes, but then every other developer is not spending hours trying to figure out what is [unintelligible 00:15:40.10] supposed to do, what is this call that’s hidden here, and all that at the end pays off tremendously just in having everybody onboard quicker, understand the codebase better, and be able to react then and implement on top of it as we build new stuff. That for us has been one of the secret advantages of building Okteto in Go, compared to what I’ve done before on Python, PHP, Java and other platforms.

David Sweet: We can talk about them… Maybe I’ll try to do them from most interesting to least, but… One thing - there’s this interesting cultural dynamic in finance – in trading specifically. I’ll even narrow it down more - quantitative trading, where people, especially when they’re new to the field, they wanna come in and they wanna try the latest and greatest algorithms and ideas, and everything they’ve learned recently in school, from papers and whatnot, and make some money. Build the magic machine that makes a ton of money.

On the other side, you’ve got people who’ve been doing it for a while, usually [unintelligible 00:14:02.12] who roll their eyes at every new thing, like “Ahh… That’s not gonna work. Neural networks don’t work. SVMs don’t work.” And sometimes they’re right, sometimes they’re wrong… I think if you say something’s not gonna work, you’ll usually be right, but you just won’t be productive… So it’s one of the unfortunate aspects of the distribution of quality of new ideas in engineering.

So what I find is - I’ve seen people try, or I’ve been one of those who have tried all kinds of things. Basically, if you wanted to just randomly throw out ideas [unintelligible 00:14:34.14] And some of the things stick. Some people figure out how to get things to work.

The big problems with financial data are the signal-to-noise ratio is very low. The signals aren’t just small, but they’re competed away. The act of going and trading on signals which your competitors are seeing as well, is squashing the signals. So it creates this non-stationarity where over time your strategies become less and less tradable, sometimes very quickly. So you constantly have to adapt and look for new ways to predict or to trade.

One thing that – you mentioned reinforcement learning, and that brought to mind… I don’t think reinforcement learning is ready to just turn it on and get a usable answer out of, in finance. I haven’t seen that. And I say that only – I say it because it’s hard. I feel like it’s still cutting-edge for solving this kind of problem. I see a lot of promise in offline reinforcement learning, what’s been going on the past year or so… It’s just amazing, and it’s very much in line with… It’s like a machine learning replacement for the old - or an AI replacement, I’ll say - for the old-school simulation optimization; like, how do you make that more automated, or more autonomous, or hyper-automated, or get that next level of automation. So yeah, I see a lot of promise, but I haven’t seen people just kind of taking that out of the box and making it work.

[15:58] A contextual bandit on the other hand, which is a limited subset of reinforcement learning - not only do I think that that’s directly useful, but I think people in finance have been doing it ad-hoc for a long time anyway… You know, if not the most super-efficient way it can be done, like people understand it these days, I think, since the beginning of my [unintelligible 00:16:17.12] doing things that kind of look to me like a contextual bandit.

What makes that easier than a full reinforcement learning problem is that you’re only predicting the immediate reward, so you don’t have to worry about your decision now affecting the state of the world for your decision later, and then have this compounding of state changes based on previous decisions. That’s a more IID sample, so to speak, to build your model with.

Adam Stacoviak: Yeah. Not to mention, though, that whenever something does happen, it could be small or large, in terms of, say, you can stub your toe. It’s like, “Oh, I’m gonna go to a hospital.” So now things are super– people tend to avoid or want to avoid hospitals because hey, you’re exposed to more people. So even little events get blown out of proportion because everything– because the stress levels… It’s kind of compounding problems. Small problems become bigger problems because it’s compound fear, so to speak.

Adam Stacoviak: But also what you described was just simply compounding conflict. In a relationship standpoint, if you keep smacking your head against the wall, over time you’re gonna be like “Man, that really sucks. I’m never doing it again.” And that’s essentially what seven years builds up to - smacking your head against the wall. Trying to do something in a relationship, constantly hitting some sort of hurdle, some sort of wall, some sort of stop point, and there’s pain associated with that… And given enough time and repetition, you will wanna stop it forever.

Jon Thornton: I think it's like buying something with your credit card and saying, "Oh yeah, yeah. I'm gonna pay that bill at the end of the month." Then the end of the month comes around and it's like, "Oh, but actually, I want to buy this other thing, too." Then you carry a balance, and then the interest starts compounding.

Katie Sylor-Miller: That’s always the hardest problem, I think, in infrastructure, is figuring out how do you do roll-outs effectively, how do you stay on top of adoption… We have a lot of things that are still kind of hanging around seven, eight, nine years later, because we didn’t focus as much as we probably should have on getting full adoption. And then it becomes this compounding problem where when we wanna try to build new things, we have to accommodate the four different historical architectures of Etsy.com, that are still lingering in various corners of the codebase. All of our new infrastructure projects take twice as long as they probably should, because we have to –

Nick Nisi: And I think that that’s something that’s going to get better with time, because we all took the survey, and presumably we’ll be on a mailing list about it next year, and… I know I tweeted about it and I posted it in some Slack rooms, and tried to get others to participate as well, and I think that that will just have a compounding effect as the survey goes on.

Jerod Santo: Well, this is at a Pacers versus Lakers game. And whoever’s around the Jumbotron, they put this crying filter, like just standard Snapchat filters, over the Lakers’ fans faces. And so they’ll zoom in on some Lakers fans, but they replace their face with a crying filter… And it just looks like they’re crying. And it’s absolutely hilarious. Everyone’s laughing, even the person who’s getting their face covered up with a crying face; and the fact that they start to laugh, it actually looks like they’re like sobbing and convulsing, and so it kind of compounds the effect…