Ashley Jeffs: Yeah, exactly. You can’t just throw things over the wire without a response, some sort of acknowledgement, without having a compromise in delivery guarantees. But the idea of that actually being a formal understood thing… I still think people get confused about it all the time, in all sorts of places. There’s ways that you can have strong delivery guarantees, and then ruin that by adding just some extra piece to your software, or some extra thing in the pipeline, that again, kind of goes under the radar. And there’s all these other stream processing concepts… But also just the idea of like microservices back then were quite a new thing. People were working out “What does a good microservice look like?” So things like metrics logging, and the way that the config spec works, all that kind of stuff was – there’s a lot of things being discovered and learned at that point…

And then about 2015, I had enough traumas from deploying all those services that I kind of figured it would be an interesting exercise to try and solve those things in a way where you’ve got a piece of software that will do all the – I would have just called it plumbing tasks in stream processing land, because a stream processor tends to mean aggregates of data in Windows being continuously calculated. And that’s the processing part. You’re processing a stream, and the comparison is always to databases. So you want to process a real time stream as if it was just a database with static data.

But what we were doing was just plumbing. You’re just reading data from somewhere, changing it a bit, and then moving it along, passing it somewhere else… And trying to do that in an efficient way. But the main difficulty, in my opinion, wasn’t getting good throughput and the sort of quality of the data coming through. It was making those services nice to actually work with. So part of that is adding in a config language that allows you to make arbitrary changes without having to recompile, and stuff. So essentially, the things that Logstash and similar services were already doing for the logging side of things… But in stream processing world you need better delivery guarantees, better quality of service, and metrics, and stuff.

So I was kind of experimenting with that stuff as an open source project. I was also learning Go at the time, because I was using C++ up until then… And I was just messing around. I didn’t have like a goal. I was just seeing if I could do it.

Marcos Nils: So we made a lot of mistakes by building the platform. I guess a lot of the mistakes that we made were also contextual to the infrastructure that we had, the decisions that we made… But one of the things that I really recall - I don’t know if regretting, or like learning a lot of things the hard way, is that because we had an on-premise deployment or an on-premise thing, we wanted to build a lot of managed services for teams. For instance, if you had to deploy a memcache cluster, if you wanted a MySQL cluster, or those basic services, Elasticsearch as well - so there were two ways that you could do it. And remember, I’m talking about sub-1000 engineer organizations here. So there’s a bunch of people that usually don’t communicate with each other, and they reinvent the wheel multiple times; that usually happens.

[30:20] So what we did is that we tried to – from the architecture team, we tried to come up with services similar to whatever you can find in any cloud today RDS, Elasticache, whatever… But we tried to come up with those services ourselves. And we tried to do it in a fast-paced kind of way, where we could show developers that we were shipping fast.

So I recall that the first thing that we did was called BQ, which was an analogous thing to SQS, I would say; it was like a queue service where you could push a message and then consume it from somewhere else, and it was our initial approach to deliver an eventful system, event-driven storage system between applications, so you could get a notification when an entity changed, and then that got replicated all over the place, and you don’t lose a message, whatever…

And the thing that I remember the most was, first of all, it is very difficult to build a high scalable, high throughput distributed systems yourself, right? There’s PhDs out there that are actually doing this S3, SQS, whatever, and we were just a bunch of senior software engineers, but trying to tackle very challenging problems. So we found a lot of issues across that way. And even though we solved the problem for some time, eventually you had to basically re-adopt or basically redo all that work, because it wasn’t scaling. So we were basically trying to chew a little bit of a bigger bite than we can actually [unintelligible 00:32:11.16]

So yeah, we made some of those mistakes multiple times. So we tried to come up with this service, with [unintelligible 00:32:19.04], which - it worked pretty well for some time, but then it wasn’t scaling anymore. It was built also on top of Node.js, which was around that time 0.4, very early days, we were adopting very edgy technologies as well… So I guess we fell into the trap of, “Hey, we are at the very top of the of the wave, we’re riding the wave on the very, very edge, so let’s try to do crazy things. Let’s try to copy Amazon in what we’re trying to do”, and then those weren’t really good decisions.

Jerod Santo: No, no, no. I mean, I was half in, half out. That’s my excuse for not following. This is super-cool… One thing I did notice is when you get to a package, it’s socket.dev/npm/package. So as an information architecture nerd, I’m noticing there’s like this npm subdirectory field going on, and I’m hoping that means you have future plans to expand beyond the JavaScript and npm world, and provide similar services for other ecosystems - Go, Rust, Ruby… Is that on the roadmap, too?

Matt Heath: Yeah, I think you need those consistent approaches. If you’re going to have a lot of microservices, I think you need a lot of consistency. We use protocol buffers, and that allows us to define an API in very concrete terms for each one of the services… Similar to gRPC allows you to define a service that has many endpoints within that definition, and each one takes a very concrete request and response type.

So you get some type safety… Unfortunately, without extensions to the protocol buffers; it doesn’t support the breadth of types that you would get if you had one application… But in many cases we can support that, because we control the whole codebase.

I think one example of that would be a money type. If you’re ever using money in an application, please don’t use 64-bit integers to store pennies, or cents. [laughter] It turns out some numbers can get really big, and some numbers - you need a lot more granularity than two decimal places. So we have an internal money type that’s very well tested, and we can pass that across the boundaries by unmarshaling and marshaling.

Jerod Santo: Many SaaS companies who are providing similar services that you are with regard to sensitive data have tried the on-premise model - GitHub Enterprise, npm famously (or maybe infamously now) have had their two separate products, the public and private package repositories… Is on-premise something that you’ve thought about, or do you offer it, or have you tried that route?

Andrew Nesbitt: [15:47] Yeah, I mean, I've kind of put up with it for 2-3 years, and then as Libraries started to take off, Libraries spread across (I wanna say) something like 20 different repositories as well… So with a certain amount of automated actions to tell me that there were updated versions of dependencies as well. For lots of people who have Dependabot or similar services plugged in to their depositories, they're gonna get regular amounts of updates - either pull requests or issues - telling them that there's something to update…

So the number of projects you have kind of multiplies the amount of notifications you get, and it can quickly kind of – the thing I didn't want to do was to stop working on those projects. That would be the other way to solve my notification problems - it'd be like "I can't do this much work." But if felt like a solvable problem to actually enable me to effectively handle more things, rather than it being like "I'm overwhelmed by the amount of humans incoming." Actually, the tooling was failing me, so I kind of just tackled it in similar way that I tackled 24 Pull Requests or Libraries, which is "Let me see if I can spin up a basic Rails app that does just enough to get by, open source it", and encourage other people, if they have the same problem, to dive in and to add their take to it, or to drive it in a way that they feel would improve it.

We've had something like 80 different people contribute some very significant features and design work to Octobox over the past couple of years, that have made it into this kind of really nice, well-rounded, solid tool, that people actually really depend on to get work done now.

Perry Mitchell: Yeah, it’s really cool. It’s a PHP-based system. It has a couple derivatives also - NextCloud, which I wanna give a shout-out to… Those guys have been amazing, helping us with integrations. All of it is open source, it’s as good, if not better, in my personal opinion, than some services similar to Dropbox, obviously because it’s free, but it’s also very fast. They have all the clients, mobile applications, and you pay nothing for it.

Of course, you need to be relatively tech-savvy to set it up yourself, but other than that, these services are a lot of fun to work with, and can teach you a lot about cloud storage as a whole. And it just seemed like a natural start for us.

We also added WebDev support, which many online web hosts support – I’m not sure whether Amazon Drive does anymore, but many of these cloud services support WebDev, which is a great way to integrate with cloud storage, which we also support.

Dan Kohn: Really one of the biggest changes in the roles of open source software foundations is the emergence and ubiquity of GitHub… Because it used to be the case 20 years ago that for a foundation to provide source code hosting and website and a mailing list was just immensely valuable. Any project needed that and wanted that in order to be real and just be able to function. But today, everybody gets those things for free from GitHub and other similar services - GitLab etc. So now an open source foundation actually has to provide some additional value on top of that in order for it to make sense.

Going back to that history, the Cloud Native Computing Foundation really began with Google having created this internal software, Kubernetes, which they developed based on the 15 years of experience they had running containers on their system called Borg. So Kubernetes was built on that, factored in a lot of that learning, and then they were successful in getting companies like IBM and Red Hat and Huawei and others to engage with them and start contributing to it and becoming maintainers on it… And they said “Okay, for this software to really reach its potential, we think contributing it to an open source software foundation would help make it more successful. It would be a way of ensuring neutrality.”

Jerod Santo: So this is one of the voids where Graphcool (or similar services) is offering some solution to these things… There are six dozen ways to skin the cat, because like you said, it’s not in the specification, so it’s up to the individual API or the individual implementation to figure out how to do it, so you guys are providing some of that stuff out of the box for Graphcool users… Is that right?

Dan Kohn: Oh, definitely. We’ve needed to hire more people, and then we’ve just needed to standardize a lot of processes. CNCF started with just Kubernetes, and then it took six months or something for us to get Prometheus. Then I think Fluentd was number three after like eight months. So during those phases, if you needed something, you would just send an email to me or our Chief Operating Officer, Chris Aniszczyk, and say “Hey, can you help us with this event?” “Oh, I need to set up an account for so and so.” “Oh, I’m having this issue.”

That worked fine at first, but I think almost like any organization or software company, you just have to put processes in place going forward. So we have something we call the Service Desk, and it’s just a ticket tracking system… But any of the maintainers of any of those 26 projects can in principle ask for anything – I mean, conveniently, we’ve had the budget so far that we haven’t needed to turn them down for a lot or for much at all.

[39:58] But the specific requests tend to be – I mean, a lot of them are just like super-minor things, like “Oh, can we have an official Kubernetes slide deck that people can do community presentations without having to use their company’s slide decks?” Or for Prometheus, it was “Hey, we wanna run a community event for about 250 people. Can you help us organize that, handle all the money for us for the sponsors that wanna come in, help us sell it, and such?”, and so we’re very happy to do it.

And then, you know, presenting to our end-user community, engagement there… All of our projects are very eager to be involved in KubeCon + CloudNativeCon, so although we do have this competitive track system, we also have slots, internal deep-dive slots for each of the projects… So they definitely appreciate that opportunity to get in front of the audiences.

We do a lot of work with them on social media, on press relations and analyst relations, on giving them some kind of tracking on how things are going…

I mean, one way of thinking about it is that if you’re a big company, like a Red Hat or a Google and you have an open source project that you’re trying to promote, for commercial reasons, your company can provide you with a set of services. But what we’re suggesting is that for a lot of core cloud native projects, it’s much better for it to be hosted by a neutral foundation… But you’re still gonna want those services, so we try and provide a lot of those same things.

I mentioned the certification process, and we also offer training courses with it. And then thankfully, it hasn’t been that involved yet, but also offer a variety of legal services around trademarks, contributor license agreement if they want that… We generally recommend that projects instead go with a DCO (developer certificate of origin), which as you may know, originated with Linux. So we just work with the projects here…

But the sort of bigger picture is that what a foundation needs to provide has certainly changed significantly in the last 20 years. When Apache started up, it was a huge deal to have a source code repository, to have a web page, to have some basic kind of continuous integration infrastructure… And today, basically any open source project can get that for free from GitHub, from Travis, CircleCI and similar kinds of services… And we actually encourage them to do that. We don’t try and move them over to our infrastructure, but we do try and provide a set of services that remain useful.

Evan Sparks: Yeah, it definitely was a very common career path, and it’s funny - probably a few years later everybody then went into ad tech, or something like that, and now it’s probably autonomous vehicles, or something. There’s always an interesting corner and a hot area to be doing this stuff, which is one of the things that I find super-fascinating.

So after a few years of that quant finance thing, I found that other people really liked looking at P/E ratios all day, and that wasn’t for me. I was much more interested in the technology problems we were solving. I ended up going to work for a startup in the NLP space called Recorded Future. We were taking the web and throwing it through this massive NLP engine, and building structured data products based on it… And trying to figure out how we sell that structured data to places like trading firms, but also the federal government, and so on. Ultimately, that company found a good niche in threat intelligence, basically trying to build predictive indicators of where cyber attacks are gonna happen, and so on, but again, with this same data-driven machine learning technology.

In many ways, the roles were pretty similar… One being in financial services, but the other being in this totally different startup environment. But always building models and driving forward data products. In both cases though I found I was spending much more time building and maintaining my own infrastructure than I was worrying about the modeling problems. And it was really the case in those days - this is around 2010-2012 - as Hadoop was becoming popular and so on, where as soon as I was tasked with analyzing a dataset that didn’t fit in the memory on my laptop, my world just collapsed. You were forced to figure out how to write MapReduce jobs and so on. I took that as kind of a good signal to go back and invent the world that I wanted to live in in grad school. So I had the good fortune to join the AMPLab at UC Berkeley right around the time that Apache Spark was born, and my co-founder at Determined AI,

Ameet Talwalkar and I got to work on it right away, building out the machine learning ecosystem around Spark. So we were among the designers and initial contributors to MLlib, which is the standard library for machine learning in that ecosystem.

The rest of my Ph.D. was really focused on “How do we give people tools to build, and machine learning applications and optimize them in a large scale, distributed fashion?”

Scott Ford: Yes, it has been challenging. So we’ve lost a significant amount of our revenue. Our team is probably about a quarter of the size as it was a year and a half ago… And I’ve talked with other business owners that have companies of a similar business model to ours, software services, and there are a lot that have been hit really hard. A lot have gone out of business… Andrea said she had read an article with a - I forget who it was… I could probably find it if you wanted it for show notes… But it had a quote in there that there’s like an extinction-level event for small software companies going on right now. And there’s a lot more talent on the market, so from a services perspective it’s a lot easier for companies to hire full-time than it used to be… So I think there’s less motivation to work with contractors, or stretch your team out that way.

I also think it’s just a way that organizations have been trying to cut expenses and cut costs. And when you look at a balance sheet, when you look at a profit and loss statement, contractors come out of a different part of that than full-time employees do. So for your investors, it can look like the organization is doing better if you cut those expenses kind of further down on the profit statement.

Adam Stacoviak: Yeah. You said that sometimes you swap out 8-terabyte drives, so I’ve gotta imagine the largest SSDs out there tend to be 4 to 8 terabytes. But if you compare the costs to an 8-terabyte hard drive, it’s probably double; especially the e8-terabyte SSD is probably at least maybe four times the cost of an 8-terabyte hard drive, so… I mean, yeah, I’m not going to – when I buy Backblaze for backup services, or even B2 services, for example, which is like a similar S3 replacement, and you even support the API, as you’ve mentioned before… You know, I’m not looking for the caching and the speed necessarily. I mean, I want the speed to be there, but it’s not like “Well, I will pay double for Backblaze, because you give me SSD backups.” It’s just not something I’m trying to buy as a consumer, from my perspective. And that totally makes sense for your business model. That makes a lot of sense.

That’s why I wanted to talk to you about all these details, because the way you buy hard drives, and the way you manage these systems is so unique in comparison. And I mean, we’ve just never examined the behind the scenes of a data center, or a storage data center, like you all operate, like what does it take to build that… I know we’ve barely scratched the surface here. I’ve got probably 30,000 other questions that might go deeper and technical on different softwares, and stuff like that… So we’ll leave it there for the most part, but it has been fun talking to you, Andy. Is there anything else that I haven’t asked you, that you wanted to cover, in your 10-year stat history, anything whatsoever that’s just on your mind that we can leave with before we tail off the show?

Ali Spittel: I think this also comes really in handy when you’re working with heavy data. I started my career as a data engineer, which somehow ended up as a frontend engineer a few years later… But back in my day and when I was starting out, a lot of the time I was using these task services like Cucumber, RabbitMQ, which sound very similar to what you’ve been talking about… So you do all these tasks in sequence, so you do one, and then it passes off the next piece of functionality, and you can really break your app into these different tasks that are queued up.

Francesc Campoy: Services, yeah… All the different services, right? So it’s kind of a similar idea for GraphQL. So I like that a lot. That said, the problem is that it’s a beautiful abstraction that is super-hard not to make it leak. If you have a very beautiful abstraction of “Oh, it’s all one graph. Everything is a graph, you’re just gonna use everything as graphs” - okay, first problem is authentication. Huh. Fun. [laughter] If you start considering these things, there’s actually many issues of implementing that in a very natural way.

[39:51] There’s a lot of adaptors, like Prisma, Apollo and Hasura are examples of different companies that provide adaptors on top of existing databases. That is great, because it means that you can create a GraphQL layer on top of your old system, and then eventually, little by little, start replacing things and say “Oh, these I’m gonna move into this more GraphQL-native environment”, or “I’m gonna break this monolith into microservices, and just federation on top…” All of these things work and they’re very useful.

That said, every single time I go to a GraphQL summit, people talk about the same issues of caching, the N+1 problem… Like, if you’re fetching a lot of information and you’re based on top of a relational database, you’re fetching all the data in the table basically no matter what. So you’re gonna be fetching that information, and then from there you’re gonna get a lot of data, and then before you send it, “Oh, guess what - on top of customers, now I want to get all the orders.” So you’re gonna need to get those IDs and go down and get the information for the orders, and then go down again. So it’s like that N+1 problem. And then in order to fix it, there’s things like pre-fetching some things, and basically playing with caches, and things like that, and it becomes really rough. I mean, it works, which is great, but it is hard to understand and even harder to debug.

So that’s why for us, for Dgraph, the GraphQL - it was super-easy to adapt, because our previous language that we support and we will support for a long time, it’s GraphQL+-, because it’s more or less GraphQL. It’s basically GraphQL, but adding some extra things to make it a better query language for a database, and then removing something that didn’t make much sense.

So adapting GraphQL was super-easy - “it’s basically the same language - and now what we’re doing is figuring out “How can we bring the things that we added that make the language incompatible with GraphQL, how can we add them back to GraphQL in a compatible way?” And there are ways… It’s super-interesting, actually. A lot of language design, and APIs, but inside GraphQL. Super-cool.

Calvin French-Owen: I’d say for us that it’s definitely a combination… And I should also be clear - by no means are we abandoning microservices across Segment. There are actually a lot of good reasons to use them across many pieces of our infrastructure. Within this one particular case we found we had better luck moving over to a single service. So I’d say we’re continuing to make the same tradeoff instead of balances.

When you talk about service mesh, I think that is definitely something that we are following fairly closely and are super-interested in… And actually Alex has started a project working to incorporate Envoy as part of this new future of service mesh within Segment, which we’re monitoring going forward with.

I think in our case it’s probably a bit of a combination of both… We had this team of engineers who were trying to wrangle 100+ codebases across 100+ services, and when all of them do a similar thing, that’s really just hard to manage and you have to build a lot of tooling around it… And we figured, well, we’d rather take the relatively slow rate of changes being made to a single place, versus having to manage this many codebases and this many services.

[40:43] I think the one other thing that changed here as well - originally, we had anticipated that third-parties would be adding a lot of their own code into these integrations, so you might imagine we support Amplitude, and Mixpanel as places that we send data… We were kind of expecting that we would have engineers from those teams actually making pull requests, contributing whenever they pushed updates to their APIs, and in practice that really didn’t turn out to be true. It ended up being a team here who was working on it… So we said “Well, we thought we’d get these supposed benefits. We’re not seeing those, let’s move over.”

Justin Searls: Yeah. So what I had was I’ve got a blog, justin.searls.co, and it’s just a traditional-looking blog, with a lot of like link posts, and other stuff. I’ve added a whole bunch of kinds of multimedia, especially since starting a podcast. And what I want to do is every time I post something to the blog, I would want it to go on what had been called Twitter, and then there’s Twitter and Mastodon that people are on, and then there’s Twitter and Mastodon and Threads that people are on, and now people actually use BlueSky for something other posting now, so’ve I got to get on BlueSky. And to manually copy-paste the same message or to use a tool like Buffer and like manually click the thing and try to make sure that it looks right would have been so time-consuming. Additionally, now I’ve got four timelines that can each separately derail my train of thought and get in the way of my workflow, or whatever, and I’m going to be way less productive. And there’s a reason why I always had to keep Twitter at arm’s length, because I got addicted to the timeline scroll.

So I built a kind of hodgepodge of Docker containers, this random – somebody has this thing called Feedtoot, but that’s in Python, and I don’t know that, so I’m running in a Docker container and a Synology… So then I made a thing called like Feed to Gram, and Feed to Thread, that would do similar things, and those are just Ruby gems that you can use today. And if you’ve got an Atom feed, it’ll read the feed, and then it’ll syndicate on your behalf if you give it the right API tokens and you jump through the 800 steps that Facebook demands.

[01:43:59.02] The problem with that – it scaled to one really well, but then my wife saw me and the life of luxury of a write-only existence, where I was able to publish all I wanted and be in all these places without having to get sucked into them, she wanted the same thing. And so then now I’ve got doubles of all of these Docker containers running in my Synology. But then unlike me, she can’t log into that and then understand what’s happening in the logs when she’s got like the wrong aspect ratio or whatever in Instagram… And so I built her this platform, this Rails application for her business. Betterwithbecky.com. It’s like a strength training subscription app, but it also has this whole thing called Beckygram in front of it, which is sort of like a blog for people who are Instagram-first. And it looks a lot like Instagram, and it can do video, and it can do photo carousels, and all that stuff.

Well, to get the Beckygrams into Instagram, I had to actually make it part of a real working application. So I had this Rails application now, and it’s got like all of the background jobs and all of the sort of like durability that a real user would need in terms of like error handling, and making sure that they have a way to retry and fix things and remediate if there’s a problem in publishing, and then like a link back to the post, or whatever it is.

And now I’d gone through all that work, and I’m starting to get – I’m finally done with Becky’s app, I’m starting to get people who are like right into my website, or the podcast, and be like “Man, how do you do all that stuff again? I really want to be able to do that, too. Would you release like a Hugo template?” But that would just create more problems for me, because then I would be supporting a copy-pasted project file… That’s not the solution. So what I decided to do is I’m going to build a dead simple, this does one thing, and it solves exactly my problem, and if it solves your problem too, you’re welcome to subscribe to it… A little app that I call Posse Party, so you can run with my Posse.

It will basically be a place where you sign up, you give it a feed URL, probably Atom, RSS, JSON feed when I’m all said and done… And then you add your social accounts. Now, it’s working for me in production, for me, and I’ve got the big four Twitter-like things; I probably do Instagram, LinkedIn, Facebook… And all it does is read the Atom feed and a little bit like a JSON sidecar payload that is also posted on my blog as part of the feed. And that little sidecar payload says “For each of these posts, this is how I want to present this, this is the format string I want to use”, and any customizations on a per-platform basis. Like maybe for BlueSky I want it to look this way, and for Twitter I want it to look this way.

And then it’ll of course be platform-dependent. So there’s certain things - on Instagram, it’s going to look a little bit different. And BlueSky’s got different rules about embeds, and all that. So all the app really does is read a feed, turn it into these normal database rows, and then a job runs that will syndicate those to all of the different services by reading the configuration that you supply, as well whatever defaults you configure in the app. That’s it. That’s the whole thing. And if that has value to you… Jerod and I, before the show, were talking about our information diets these days, and we’ve both reverted to RSS… But I’m not going to get my mom to use RSS, or most of my friends in my life. They’re in one or two of the social apps, and they’re casual at this point, looking at it… I just want them to be able to click my face and see my stuff. Even if the downranks me for posting via the API, even if the links get the fact that you’re linking outside to a third party, the platform doesn’t like it, as long as my stuff is literally there, I’m a lot happier, because then I don’t have to tell everyone about everything in my life.

And since getting started with the Posse Party stuff, I’ve noticed that more and more people in my life – like, I had a guy detailing my car a couple of weeks ago, and he’s like “Oh man, I really loved that clip that you did about whether or not to have children and your decision-making process.” I’m like “How’d you see that?” and he’s like “Oh, I saw it on X.” I was like “Great.” I haven’t logged into X in like three years.

[01:48:04.23] So I don’t know, that’s the theory. Now, I don’t know what the market is. I kind of don’t care, because as long as I want this, I’m happy to build it… But if it can help people, encourage people be the thing that gets people to sign up, and make their own website, and own their own content, or buy their own domain and like centralize themselves as kind of the king of their castle, and only treat these platforms at arm’s length, as those are just “newspaper stands to throw my stuff on” - there’s always the canonical URL - then that just supports the open web, and it really promotes people as thinking of themselves, less as just like me, like a take maker or just operating in the Twitter stew, and instead being somebody who’s developing a voice and curating an audience. And if I can help people along that trajectory, it seems like it’d be putting some good in the world.

Salvatore Sanfilippo: Now I see younger developers not really caring about this, but we basically started to write code, and starting to be mini lawyers, understanding all the subtle things about GPL, BSD, MIT… So yes, I care about license, because the way that you express your willing about what others can do or cannot do with your code.

Also, I think that without the copyleft idea, the computers technology could not accelerate to the point that it accelerated. Because when open source was created, basically in order to create a startup, you had to buy complicated workstations, Unix licenses, database licenses… So it was impossible to have the landscape that then it was created by the open source movement. So for me, licensing is very important.

Initially, what I did with my software was to use GPL. Then I started to realize that GPL had two problems. It created problems to myself, because sometimes I thought – and if this becomes big, I want to have a business model. And I don’t want to get some paper signaled by all the contributors that I had so far. So I started to switch to BSD, saying “This is a protection for me. Also, it’s a protection for other people based on the environment they use my software.” And sometimes it can be a problem, even if they don’t want to violate the license, and stuff like that.

[00:32:00.06] When I started Redis, I was very into this BSD stage of my life, so I released the Redis as BSD also because there is also behind that an idea about accelerating society, improving society to be more important than basically what is going to happen to me, in some way. However, then it must be said that the cloud situation changed the landscape, because even if it was very complicated to create a product business model, even before - basically, Red Hat was the only one that really succeeded in this kind of game in the open source, and a few more… But still, if you wanted to sell services, you were the to-go person as a creator of the software.

And then after AWS everything changed, because there was no longer need of somebody supporting you, because it was handled for you. And also, you couldn’t even compete with the others, because in order to compete in cloud services, you have to pay for the instances, and they have them for free. Also, even the billing is complicated. There are many companies that just because of billing will just get what AWS has, and stuff like that.

So I understood at some point that the BSD license that I picked, with the changing world of the software, created serious issues to create business. Now we can go a step back and say “But why it should create business? It’s an open source software.” And I believe that more or less every complicated open source software has in one way or the other an economic system behind it, because it’s a lot of hard work for many years. Either people are paid very well, or they will not afford to do all this kind of work. So I believe that both things are needed to redistribute to the community, and also.

And inside Redis, they didn’t want to change the license, and I didn’t want for a long time… But there was this discussion, but it was some kind of taboo. So it never happened as long as I was there that somebody asked me “But what do you think? We want to change the license.” It’s a conversation that didn’t happen. I just created the module system, and Redis the company started to have the modules that were enhancing Redis capabilities, and that was it.

Then when they changed the license, I understood that it was basically some kind of a forced move in some way, because with BSD it was too complicated to compete in this market. However, now we are realizing - me and also inside Redis - that SSPL was not accepted by the community in some part. And we care about this thing, because you know, I don’t believe that SSPL is a terrible license, because it’s very similar to other GNU licenses. It’s just a couple of sentences… But the reality is that culturally it’s not accepted. And so we are starting to discuss inside the company about this problem.

[00:35:52.25] Also, one important thing is that because of that, we are going to add in Redis a lot of the features that are now only for the paying users. For example, now I’m working a lot to vector sets, which is the first fundamental data type that Redis gets after many years… And it will be released in the community edition, like normal Redis. And like everything I did in the past, it’s no dependencies, so it compiles because the data structure – so the HNSW data structure for vector similarity I wrote from scratch, the quantization I wrote from scratch, the hybrid search… So it’s some code, it’s like 6,000 lines of code in total, the other Redis data structure. So you can open the code and understand how it works. And it was impossible before to do that, but still, maybe there are setups that make everybody happy, enough protection and the community will be more happy. So there is an ongoing discussion inside the community now. I’m not sure what will happen, but we are focused on the problem.

Nick Sweeting: Yeah, that’s sort of what I’m gravitating towards as an initial carrot to offer - if you agree to time-unlock, then I’ll host your stuff for free as a backup. It gets dicey when I have to re-host content for other people… So the way archive.org works is they basically operate as a library. They’re a nonprofit institution, they don’t earn income from their hosting… They have a separate LLC that does some paid services, but it’s a separate LLC, and they’re basically not earning revenue directly off of re-hosting often copyrighted content.

[31:49] If I ran a public hosted service where I’m mirroring people’s content, I would have to either be a library like them, in which case I can’t accept payment for hosting at all. So this is the only way that I could offer to host people’s stuff. Or I have to figure out some other new legal system that hasn’t been invented yet to do this. Basically, you’re trying to make a business out of BitTorrent, right? It’s a very similar problem. It’s very hard to charge for this and not be legally liable for re-hosting copyrighted content. So there’s probably some middle ground where it’s a – people are buying an app that they are running locally, that they are operating, that’s connecting them to other people running this app. But I am never hosting public stuff unless I have a signed release from the archiver saying that they own the content, and it’s okay to be time unlocked. And then I’m taking on the central moderation labor of, unfortunately, delisting that stuff if I get copyright complaints. If someone sends a DMCA notice and says I have to take it down, I have to comply as a central agency. But the people running those individual archiving apps can still share it if they want to. Something like that. That’s sort of a middle ground option.

Break: [33:00]

Adam Stacoviak: The question for you is this. Could you do, like, similar to a commercial open source company forms a company around open source, could you form a company around the podcast? Like a services business?

Jerod Santo: Yeah, 100% agree. I have self-hosted many things throughout my life, and I learned so much doing so. Both production, small business, and friends and family stuff, whether it’s self-hosted on machines in my house, which I have done for a long time… I had a Linux server that ran in my home, and I ran all kinds of services off of that. And then also VPS, self-hosting, shared hosting, self-hosting… You know, just managing your own things. And yeah, the amount of learning is really not to be compared with. You can’t fake it, it’s just real.

But the learning comes through trial, and it comes through things going wrong. And this post that he writes is like “Here are the –” One of the sections is like “Things that broke in the last six months”, you know? And it’s like, yeah, you’re going to have that kind of stuff. And it’s similar to any sort of maintenance. I don’t know how many large appliances and vehicles you own, Adam, but you and I are both old enough to know… A house, a car, a laundry machine, anything that’s significantly expensive and significantly complicated breaks. And when you own it, you own the break, you know? And you’re going to either pay to fix it, or learn to fix it. And for me as a guy in my 40s with a pretty large family, I don’t have the patience or the time to self-host anymore, but I think everybody else should. [laughs] Not everybody else, but given your circumstance, I think everyone should try it, and some people should do it. And there’s certainly things that I’ve thought about self-hosting, even though I don’t have the time, because of privacy and autonomy, and the independence that Christian speaks of. For me more importantly than the learning, because I’ve already learned enough, I think, in the category, is there are things where I want the independence and the privacy. And so that’s why I consider it, even though I don’t want to do it, because I know how much of a headache it can be.

Suz Hinton: [laughs] There are threat hunters at - yeah, like private institutions. So for example CrowdStrike’s threat hunters are actually threat hunters for hire. So they work with companies directly. And if you look at the product offering online, it’s called Overwatch. There are different tiers of it, where they’ll even give you briefings on the latest threats to look out for, and things to maybe specifically look at for your industry even. So if this company is a financial tech industry and they’re working with CrowdStrike, the Overwatch team, the threat hunters can give actual briefings on what they’re seeing as trends in that financial industry based on attacks on other companies that are similar to them.

The threat hunters do a lot of different services, and so it’s going to depend on whether you’re in the private or public sectors to like what tools you use as well. So I’m sure that there are teams that use JIRA to keep track of intrusions, and dump a bunch of data in there… But I think that a lot of these tools tend to be very proprietary. And so they’ve been designed and developed and incrementally improved based on the specific kind of work that these threat hunters are doing at their institution. That’s all I can say.