John Spurlock: There are certain agreed upon ways… So there’s a few things. I could talk about this for a long time, but let me try to keep it short. Basically, you need to identify the things that are common, like what the app is, right? So you have to know, is this a request coming in from an app that a user is using, versus Chrome, or a bot, or something like that? So that’s one dimension, is like what the app is; you can get that from the user agent. The harder thing is IP address, right? So there are ranges where you know to expect some interesting data coming in from them, for various reasons. Some are VPNs, some are corporate networks, where you see thousands of requests coming in from the same IP, with different user agents, so they’re legitimate… And then you see the bad actors, you see the bots… Any open system is open to all sorts of automated traffic that you have to identify. And they often use Overcast as their user agent, right? So you have to do some application of filtering at the IP level as well.

That used to be easier, because you could kind of identify users versus servers, and just kind of block all servers. But now it’s harder, because a lot of people use VPNs, and a lot of people use Amazon Web Services for listener apps. So a lot of traffic goes through Amazon that is actually representing a listener hitting Play on an episode. So it has to be sophisticated.

There are shared – obviously, Amazon publishes their ranges, there’s some public server lists, and I plan on incorporating as much public info into the calculation as I can… But as I’m sure you know, there’s like day-to-day things that you have to identify. Like, you kind of rate limit some things, and then certain ones pop up and you have to add exceptions for those… That will just be part of the operation of the site. But that’s part of why sponsorship is needed, because there will be some aspect to this maintenance-wise going forward.

There are certain – and I’ve talked to certain people that are doing something similar. So they are doing their own for-pay analytic service. They have the same problems, they need to identify the app, so there’s common ways of identifying user agents… And I’ve been having good conversations on sharing a lot of the even IP ranges as well, because that’s something that – it’s not super-high-value-add, especially for identifying VPNs, Tor, that sort of thing.

So I could see – there’s some existing server lists, but I could see kind of getting together and putting those in the public as well, specifically for podcasting user agents, because that traffic actually looks a little bit different than regular web traffic.

So Cloudflare actually has really good bot detection, but it’s really horrible for podcasts, because their bots look different. It’s something that will have to be sort of domain-specific, but I do have – it’s not impossible, right? And it’s something that you do the best you can. We haven’t even talked about – I guess we have a little bit, the downloads themselves are not listens. So it’s already sort of like – you already take a percentage and just say… You just pick a number and say – so you really do the best you can and you identify certain anomalies as they come up… But anyone in accurate digital marketing has to deal with this.

Chris Beams: [01:24:04.19] Yeah, for sure. And some percentage are actually out and out scams, right? I think a lot of people are too quick to say “Oh, it’s just all scams”, or something like that. Mostly, the things that you run into, I think it’s still safe to say, are actually well-intentioned people saying “Hey, I’ve got a bright idea. I’ve got some sort of notion of a kind of economics that I can add into this application, that I can sort of tokenize it, and that I can build a legitimate way to fund the development of this project. People will hold this token and pay for using the application with the token” or whatever different kind of model.

So there’s hours and hours that we could talk about on all of those models, right? But what Bisq is doing is something if not unique, very unusual, by comparison, to this kind of norm that we’ve been seeing in this space. Basically, one thing that makes it unusual is that Bisq is working, running software. We’ve actually built the system, it’s out there, it works, people use it, and people are happy to pay to use it.

When people initiate a trade, both on the maker side and the taker side (both counterparties), they pay a fee into Bisq, in Bitcoin; that fee goes to the arbitrators today. I’m one of those arbitrators, the founder of Bisq is one of the arbitrators. So people are willing to pay to use this application in this network, because it’s valuable to them to be able to trade. Same reason people pay a fee on a centralized exchange.

So that works to a degree, in terms of economically incentivizing the development of the application. That can help pay the bills and so on, if you will, but it doesn’t scale very well with regard to getting that JavaFX UI person to come and contribute that chart implementation, or fix that bug, or whatever it is. That’s where this idea of a DAO comes in.

What we’re up to here and what we’re rolling out is a token; we added a token into the sort of larger Bisq system, and that token is actually something based on Bitcoin. This is something that is not a new idea, but not a lot of people do - it’s called a colored coin. What that means is that each BSQ token (we actually call the token Bisq as well, but its ticker symbol would be BSQ) is actually backed by a small fraction of a Bitcoin, a thousand Satoshis, where a Satoshi is one one-hundred-millionth of a Bitcoin. So 1,000 of those, which is a very, very small amount of Bitcoin, actually backs this coin.
What this means is that – and before I get too much into the details of what BSQ is, what can you do with it? Well, we haven’t launched it yet, and I can get into the details of the roadmap later, but when it’s fully functional, what people will be able to do is buy BSQ, trade for BSQ on the Bisq exchange - so they might trade Bitcoin for it - and with that BSQ they’ll be able to pay their trading fees at a lower rate than they would pay for it in Bitcoin. So it’ll be a cost-savings to them, they’ll be able to do more trades more cheaply when they pay for them in BSQ.

[01:28:04.27] Okay, so what? What good does that do? Well, on the other side, and getting back to the original question you asked, we wanna bring people in, have more people contribute, we wanna fully decentralized the operation and maintenance and development of the whole network and the application and so on. So this is the second role of the BSQ token.

The first role is that you can pay to use the application with it, so it’s a kind of access token in that way. The second utility that it has is that you can be paid in it for the work that you do for Bisq or on Bisq. Somebody comes along and implements the chart that we were just talking about - they would at the end of that month period issue what we call a compensation request, and fundamentally that’s gonna be a document saying “Hey, I did this work.” It might be a link to a pull request, or one or more pull requests or commits, something like that that says “Hey, I contributed a chart and it’s merged. That took me this much time, and this is my market rates, what I usually get paid.” Again, just sort of thinking in the units of currency that we denominate our lives in, “Hey, this was $1,000 worth of work for me”, and then they would look at what the market price of BSQ is, and they would issue a compensation request for that amount of BSQ.

What happens is then DAO (Decentralized Autonomous Organization) votes on his compensation request and anybody else’s requests that were issued during that period, and they’re voting in the affirmative or negative - “Yes, we’re going to pay for that work”, “No, we’re not.” And people don’t have any guarantee.

Daniel Whitenack: [33:32] It’s even in your domain of expertise, but let’s say that we have a large, expensive and dangerous piece of equipment like an airplane or a helicopter. And there is obviously a vast amount of manuals and documentation about the maintenance of that, and the operation of it, and the safety around it etc, etc. So there could be a case, and this would not even involve a bad actor, but let’s say that we put a “Chat with your docs” interface on top of all of these manuals, and the operation information, and the maintenance information, and all that. You could imagine if, again, these models are essentially generating output that’s probable; they don’t know anything about, like you’re saying, reality, or intent, or anything like that. There is no knowledge there, it’s just completion. So they could complete someone’s request saying, “Well, how should I fix this issue with my airplane or helicopter?” and the model could say, “Well just take that part off. It’s a throwaway part. It doesn’t matter”, based on the text that it’s seeing. And that could be a significantly life-endangering decision if the maintenance technician or whoever it is actually trusts that as fact. Now, you could also imagine bad actors getting into that scenario, and modifying information such that it would generate out dangerous information, or something like that. So I think that’s a concrete example that would endanger lives, but does not involve AI becoming sentient, or something.

Ahmad Nassri: [15:56] I think this is a nuanced approach, but there’s a number of different ways people interpret modularity and modular software in general… Especially in the JavaScript world, when you use the word “module” or “modular”, people will either think of a package, or a package resolution methodology, as in with ESM, or otherwise. What I’m talking about when I talk about modularity - I’m talking to the age-old philosophy that started with the Unix philosophy all the way back in 1978, or something like that, where it talks about how you write code and how you write software, and some principles around that.

If I recall correctly, the 4-5 principles there was that in order to make modular software, number one is you make each program do one thing really well; everything has one job, and one job only. I think number two was there was like an output/input exchange, so every output of every program should become the input to another. If you’re ever used Unix or Linux and you pipe operations between command lines, you’re very familiar with those kind of approaches. Again, this is from 1978, so the very early days of computing.

But the things that I find most valuable, especially in the context of software as a social practice, that we all do - the tools and the way you build your tools and products and all these principles should be tailored to make sure that you [unintelligible 00:17:17.25] the programming task to other maintainers… And the idea that everything should be easily maintained and repurposed by developers other than the people who created it. So we would not be successful in the software industry if the person who wrote the code the first time is the only person who’s gonna be able to maintain it forever.

That’s why we have documentations, and we have practices, we have guidelines of how to actually make software repurposable and shareable by others, and that’s why we have patterns like forking, and cloning, and sharing code… Because the whole point of all of this is that at the end of the day software is about people, and you wanna make it so that some of these practices around modularity - you wanna make it so that it’s easy for others to come and repurpose or refactor or use your software without having to go through tomes of manuals and understanding all your individual author’s purpose and knowledge.

I think one other one that we all suffer from every day is – you know, one of the principles of the Unix philosophy was everything should be designed in a way that you can just throw it away and rebuild it… And as you know, in a monolithic world view, that’s not such an easy thing to do. But as you focus on building smaller and smaller units of code, and build them in a modular fashion, that is everything does one thing very well, every part of the program becomes an input to another, everything can be rebuilt and thrown away, and most importantly, it’s built in a way that others can just come in, understand it, do any changes or fixes and move away without having to spend years and sync up with [unintelligible 00:18:48.13] and everything.

To me, those are the key philosophy areas where the – again, the Unix philosophy did this very well back in 1978. But in today’s world we haven’t really matured that enough. We talk a lot about - especially in the JavaScript world - packages and sharing, and libraries and code, but we still have these big monolithic libraries, we still have these big, complex frameworks. And although we’ve done very well on things like sharing and making code repurposable by others other than the original maintainers, I think we’re still lacking some maturity around “How does our software become portable? How do our libraries become interchangeable?” So for me, when I talk about modularity, I talk about these kinds of topics in the general sense. And then I start talking a little bit about how to become more specific in nature about solving these problems. And just for context, people seem to like modular code; there’s no debate about that. I don’t think anybody goes into their day-to-day job and talk about building the next monolith.

I think there’s a valid debate in terms of monolithic approach to deployment and infrastructure and maintenance, but that’s separate than writing code and that’s separate than how you design your systems.

[20:09] From a numbers perspective - and this is something everybody sees every day when they go to npmjs.com, we are now at 1.159.000 packages, and these are just the open source ones. And I’m always curious by that number. I’ve always been curious about it from before I even joined npm, why are there so many packages; why does the JavaScript community create such a prolific amount of code and software to share? The answers that I came up with just based on my own personal observations is that we in the JavaScript community have had a good run of satisfying some of those human requirements, making things so easy to throw away and repurpose, making things so easy for a newcomer to jump in and get on board, making things simple and clean, and building one thing that does one thing very well, and not be concerned with big, complex challenges across different domains. That’s why we have so many packages, that’s why we have such a big JavaScript community. That’s what made my career, and that’s what made a lot of other people’s career, and its’ wonderful.

I think the challenge though - and coming back to our enterprise examples over here… The challenge is we’ve solved that in the open source world, but we haven’t solved it in a way that informs a method of building software. All of this so far has been about libraries and code packages, and patterns around that. But I haven’t seen it being adopted very widely in the way we build software at companies or at work.

So the approaches of modularization, and whether you wanna go down the path of packaging, or microservices, or any of those topics, or even the serverless world today, there’s a real pattern here to adopt, and I think - again, taking the JavaScript example, we’re in a world where JavaScript runs everywhere. Myles Borins from Google has a talk yesterday at Node+JS Interactive where he was talking about universal JavaScript. Universal JavaScript is just a new term that we’re talking about where the whole premise is you write once and run anywhere.

We’re in a world now where JavaScript is running in the browser, in your server in Node.js, you can write JavaScript on edge workers, on companies like Cloudflare, you can put them in your databases, even in productivity software like Google Spreadsheets - you can run some app scripts in there, you can do that I think in Excel nowadays… If you’ve gone to NodeConf EU this year, they gave out smart watches that were just running JavaScript… So that’s great, JavaScript is successful. But what about the portability of those software code and libraries that are being created? What about the developer experience associated with them? Wouldn’t it be great – and I think that’s the promise of JavaScript, that you can write the same software that can run in your browser, and on your smartwatch, and in your Excel spreadsheet. But the reality is there’s a lot of work involved in getting that to happen, and we’re kind of offloading a lot of that work to the developer who’s responsible for doing this. But we haven’t come up with the patterns yet of how to approach those things. I think this is where to me the Unix philosophy from so many years ago kind of touches on the key ingredients required to get there. I don’t necessarily have answers in this space, but I love asking the questions, so we can have a dialog and a debate in these conversations.

The one pattern I have noticed in terms of modernizing the way we adopt these Unix philosophies just so happens to be around package management. It’s not because I work at npm and that’s my day-to-day responsibility, but it’s true. You’ve seen the success of things like React, where people are now building design systems and iterating on them at such a large scale, and involving not just developers, but now designers, and UX designers in this kind of workflow. That’s becoming more and more attainable, and nowadays you have tools that are meant for designers that are generating the code, and generating it in a way that’s a package that is shared and distributed in a community within your company or your clients’ environments right off the bat.

[23:57] You don’t even need to write the code anymore, you can just have a designer drag and drop some things. I think the company is called Framer. I know other folks in the industry are looking at this as well; I think InVision and others are playing an interesting part in all of this… But this idea of modularization is beyond just the software and the code. It extends to UX designers, it extends to product design, it extends to every aspects of technology. I think, again, we in the JavaScript community have kind of solved or addressed that problem in a very efficient way with package management and packages in general. It would be great to start seeing that pattern being adopted more widely and more – I don’t wanna call it standard, but perhaps best practices around these ideas and patterns in the day-to-day work of people.

I know I’ve done this before, again, when I mentioned the enterprise space - when we have teams as large as 450 people, it’s not gonna be about just publishing a new version and expecting it to work. There’s a lot of workflow involved, there’s a lot of operations involved, there’s a lot of maintenance and upkeep and analytics involved. A design system with one component that has a button in it might have 15 different versions, but the adoption of it is all over the map, and we end up spending a lot of our time as community moderators and architects of the design systems, and the companies just aren’t chasing that down and trying to get the adoption going.

The way the software is built is really relying on those patterns, or at least it should become more and more embedded in the way that software is being designed, whether it’s a monolithic design or a microservices design.

The other interesting area of this - I’m using design patterns as an example because it’s an easy one to point at - but now we’re in the world of serverless; now we’re in the world of literally function-as-a-service. While you can deploy a big monolithic application as a serverless application and do that, you probably shouldn’t… But now, more and more, we as software developers, especially in a server-side context, are thinking of smaller units of code that have to be built and orchestrated and talk to each other through an events system to create the result and the output of our product. So again, those modular best practices keep coming back time and time again in all the areas of the software industry and all the different things that we’re doing.

I have a lot of questions that I get through the npm community, oddly enough, from people who are using npm in embedded systems, and they’re asking about best practices of “Well, how do we do package management and download big React libraries, or Lodash libraries and run them on these systems? …because there’s not enough memory, there’s not enough processing power.” And the answer is perhaps those libraries and those tools were not built to support those embedded system challenges, but the modularization approach allows you to have a more nuanced approach of like “I want this part of this library, I want this part of this framework, and I can just then put them together, create a modular pattern where every piece is responsible for its own logic, and the output of one can help the input of the other, and create kind of a workflow chain of how my system is gonna be designed and work. And hey, if something doesn’t work, maybe I can just throw it away, bring another library in or another part of that module in, and it will still function the same way. I don’t have to refactor my entire codebase. That’s the future I wanna see.

Marcin Kurc: So first of all, interesting. I’ve gotta go do the YouTube search, because that’s definitely not something that we see in real life. I think the situation is that Google definitely has been pushing the concepts for a long time, and they have teams that just focus on that 100%. But within engagements that we have, I would say there are a couple other companies that focus on SLOs, but every single monitoring company or observability company out there has got some kind of solution or something to say about SLOs. And that’s really like the real life situation for us. Dynatrace, Datadog, New Relic… I mean, everybody else, right? Just name them. So the real life, I guess, it’s a little different than YouTube.

And then maturity, where it is - our point of view… Like, we haven’t really done a huge market research. We’ve had conversations with a number of analysts, and they of course agree that that the market is maturing, and people understand how SLOs help them run their business.

On the base of our customers - you know, you mentioned one or two… Their SLOs are becoming the core of their operation, I would say it that way. One of our customers called it tier zero of observability, that helps them really bring it all together; it allows them to see different teams and different operations at the same level, right? It’s the same reference point, I would say. So you don’t have this issue where you have four nines that are completely differently defined versus three nines, and so forth. They really get a good idea of how things are performing, where you take dependencies, what they can offer…

And then finally, a lot of customers, I would say probably every single one of our customers is using SLOs for planning. And sometimes that’s as simple as if somebody shows up and says “I need another $5 million to spend on AWS”, and the question is “Why?” “Well, we’re running out of capacity.” And that’s usually where the conversation ends, right? Now, SLOs really enable you to provide that better insight into what needs to happen. Do we have an issue with capacity on the cloud provider? Do we have an issue with our application hitting limits? Do we have an issue of this monolith that cannot scale anymore and we have to figure out how we really transition to something different?

[00:34:01.18] It really helps people to understand how the teams are performing, too. You’re sometimes pushing out features - because everybody gets promoted on features, right? Not on maintenance. And you start seeing degradation of your service, degradation of your customers’ experience… So you just start thinking about how we pull back, when do we pull back, how much do we pull back, right? We want to stay competitive, but we don’t want to get our system to break every hour, right? So a lot of those concepts - the more people are using SLOs, the more mature they get with it very, very quickly.

Gary Bernhardt: Well, I will give another caveat here, or really repeat the same caveat that I said at the top, which is I run very small companies, I’ve never worked for Google, Facebook, Amazon, any of that; I’ve never worked on a very large system, I’ve never worked in a very large organization… So you know, anything I say, you have to read it through that lens. But my goal is to understand every aspect of the business, horizontally. So by vertically, I would mean like your specialization; probably everyone listening to this is a software developer of one sort or another. We are all specialists in that vertical section of the work, software development. Horizontally I would mean like can you think about the marketing, or at least an approximation of how a marketing expert would think about it? Can you think about design? Can you think about finance? Can you think about the business operations? I’m not good at most of those things, but I want to be able to think about them enough that I can make at least reasonable trade-offs across the whole business.

And a lot of times what I hear from friends who do work in larger organizations is that other parts of the business are basically setting constraints that the software teams then have to implement, and that can lead to tremendous inefficiencies, when you end up building things that don’t really matter. So you have the vertical component of a specialty, you have the horizontal component of all the different specialties that you want to be able to work across, and then you have time, which I guess in this analogy is depth, although it starts to get weird to think about… But all of these things evolve over time. and the thing that really matters is you want like the integral of complexity over time; you want to minimize the total complexity over a period, a long period, like ten years, and most of that complexity is not going to be in the first six months, it’s going to be in the maintenance. And many times, most of that complexity is not going to be in the code.

[52:03] One example I really like to trot out here, because it’s so horizontal, is suppose you’re building a billing system, and you’re using third-party to process credit cards. It has two modes. One mode is every single time a charge is run, we make a deposit to your bank account. We deposit the money immediately. The other mode is every day we do a deposit of all of the day’s transactions. You might not have any preference, or there might be some reason that the per transaction thing is preferable to you first from a software perspective. But what does the bookkeeper think about this? The bookkeeper wants one transaction; they do not want 5,000 transactions a day. And you know, it’s an easy to understand example, because even if you’re not a bookkeeper, it’s clear what it means. Obviously, it’s more difficult to do that in more subtle cases, but that kind of thinking, being able to think across domains other than your own area of expertise is so, so valuable. And it’s very valuable to an employer, if they’re smart, if they see what they’re getting.

Audun Fauchald Strand: We had more problems – or maybe not problems… It was more work when it was on-prem. But this, I think, is one of the good things of the managed service. Google does everything for us. So either we decide when to do it manually, which is probably for major upgrades; and for minor upgrades, it’s just a maintenance window, and it kind of happens.

One of the reasons it’s important for us to modernize the applications before we migrate to Kubernetes is these kinds of operations become easier as well… Because if the application is robust enough to be able to handle that, and the node dies, because it’s moved to another node, and then upgrading the cluster is also much easier.

Ivan Porto Carrero: [28:08] Yes, but it does more than that. It is a joint effort between Pivotal and VMware, and to some extent also Google, for the Cloud Foundry stuff on Google. Now, this doesn’t mean that PKS requires Cloud Foundry. It can be used next to it, and it can be used standalone. What I work on - at the VMware side at least - is the integrations with the VMware stack, then the optimizations we can do at a Hypervisor level to work with something like Kubernetes, so that you get these benefits of maintenance mode and the separation between your hardware and your actual workloads.

It uses [BOSH, so that’s the one component that’s required, the BOSH piece. If you’re not familiar with it, BOSH is a lifecycle manager for applications. It’s something that will monitor your infrastructure. If something goes wrong, it’s going to take immediate action. If one of the nodes becomes unresponsive, for example, or some failure condition happens, it’s gonna try to restart the processes; if the processes aren’t to blame, it’s gonna recreate the VM. That’s, in a nutshell, what it does. It’s a managed Unattended version of Kubernetes.

Earlier, somebody mentioned that operating these things is annoying; this is exactly the type of intelligence we’re trying to encapsulate in that project. It takes away the operations headaches of it. It will also do zero downtime upgrades, and so on, over time. I think that’s in a nutshell what PKS is.

Other than that, we make sure that it can leverage just all of this stuff that VMware already has, like a login site for log aggregation, and wavefront for metric segregation, and so forth. I’m employed by VMware after all…

Seth Vargo: That’s a great question. It’s actually horizontal across developers’ operations, security and machines. Developers need access to things like AWS, Google Cloud credentials; you need your IAM keys to be able to do stuff, and you might need access to be able to read some data from the database, so that you can see the columns and the query and do some analysis of the production database. With Vault, you configure that. The Vault administrator would configure the developer’s login to Vault - and I’ll talk about what that process of logging in means in a minute… They log into Vault, and then they can request those things that they’ve been given permission to do.

[24:16] The security team is obviously a lot focused on a higher level; they might be managing things like the TLS communication between services. If you’re in a microservices-oriented architecture and you have two internal apps that talk to each other, you want them to talk over an SSL connection, an HTTPS or a TLS connection.

Managing those certificates and making sure those certificates have really short [unintelligible 00:24:39.08] and really short lifetimes, but are also always valid, so that you can communicate between services, allows the security team to sleep at night, because it means even if an attacker is able to penetrate your outermost firewall, they can’t sniff traffic between the services, because all of the traffic between services is encrypted.

We have the operations team. They might be responsible for doing things like building images or doing maintenance, so they also need access to API keys and databases and the whole nine yards. This process of authentication or “logging into Vault” is really important, because Vault is a dynamic secret acquisition engine, but the way you dynamically acquire a secret is based off of your permissions in Vault. When you log into Vault, there are a number of ways to authenticate who you are.

Let’s just stop talking about Vault for one second. If I were to log into a website, I’d type in my username and password and I click Sign In. From there - I don’t know if most of the viewers are familiar with it, but you’re assigned a token or a session ID from that application, that website. Your browser stores that in a cookie - it might be encrypted, it might not be - and that’s how you’re identified moving forward in the system.

It would be pretty annoying if every time you went to Facebook or GitHub you had to type your username and password in, just to like a post or comment on an issue. So that’s how you get identified as you move through the system, and your browser handles that carrying of that session ID along with you, and it has an expiration. Also, you may be familiar with things like OAuth, where you might be able to sign in with Facebook, sign in with GitHub, sign in with Twitter, in which case you supply your Twitter login or your Facebook login to authenticate to a third-party application. That’s something called OmniAuth or OAuth.

Vault has a very similar process. It’s not OAuth, it’s not a traditional website login, but you can think of them metaphorically as the same. As a human, you can authenticate to Vault with GitHub; you supply a GitHub API token, an administrator at Vault configures Vault and says, “Hey, anyone who is a member of this GitHub team has these permissions in Vault.” You can say, “Anyone who’s a member of the engineers team on the GitHub organization HashiCorp has read-only access to all of the production data.” We don’t actually have that, I’m just giving you a hypothetical…

You can also say, “Anyone in the accounting team has the ability to SSH - because Vault can actually manage SSH for you - into some service that runs some accounting procedure”, or something like that. There’s also the ability to log in with a generic username and password; you can log in via Okta - I don’t know if you’re familiar with Okta… It’s a very popular enterprise Single Sign-On. RADIUS, another very popular Single Sign-On. LDAP, including Active Directory. So Vault doesn’t actually manage the authentication for you, it generally delegates the authentication to a third-party service that the Vault administrators have configured. That’s the authentication, typically called AuthN. Vault does manage the AuthZ, which I’ll talk about in a second.

[27:49] On the machine side of things, machines also need to authenticate. If any machine in the Vault cluster could just request credentials, that wouldn’t be very fruitful, so there’s a number of ways for machines to authenticate. They can authenticate via a token - maybe they’re supplied that token at boot time; there are ways for machines to authenticate on the cloud providers, so we have authentication for EC2 instances, and there are ways for applications to authenticate as well - things like app-id and app-role allow applications like your frontend web app to be able to supply information it’s given at boot time, and Vault will in turn validate that and give it back a token. You can think of it as username and password, but instead of human-friendly things it’s all UUIDs and special characters all over the place.

Jack Shanahan: Yeah, Daniel, I would say this is something we talked about pretty much every day. Now, Maven was a little different, because we knew the problem from the beginning. That problem could not be solved in any other way, so we knew we were going. But when I got into the JAIC, the conversations we had were generally along this line: we’re going to start with lower risk, lower consequence use cases. Solve those, learn from them, and then slowly move up the ladder. Maybe even fast up the ladder, depending on what we were talking about, to get into those higher-risk use cases.

There is a lot of talk, of course, about the dangers or the risks of so called killer drones. I’ll tell you the thing I wasn’t working on in the last five years of my career was killer drones. I had nothing to do with those, because they were too high risk, too high consequences, and we had to understand how to actually do AI before it started getting to that.

So we started with things like predictive maintenance, or there were some medical initiatives that we were taking on. There were some things that we could put on individual intelligence platforms or sensors. And so we learned from that, and then based on those lessons learned, began to sort of edge our way into more consequential use cases. And to me, there would be a clear analogy with any business that’s out there. Because again, as CEO, the risk will be different. It might not be life or death, but it could be tens of millions of dollars, and that’s very risky for a CEO.

So the idea of proving success at something… And the reason we stood up the JAIC is because we showed enough success in Maven. As difficult as it was, we did show real success, and put AI models out in combat operations within a year of standing up the organization. Okay, let’s go try some other things. And so what you’ll see now is, okay, now that we’ve learned all these lessons, they’re going to a little bit more consequential use cases. Maybe AI-enabled autonomous drones. Maybe not lethal drones, but autonomous drones. And then you’ll start seeing this. In fact, you can kind of see some of this playing out in Ukraine, as you look at the headlines what they’re doing with drones on both sides right now.

So to me, it’s like starting with something that’s manageable. Get your arms around it; learn what it means to build a data management pipeline. Because what I’ve found in the DOD, hands down what stopped people cold when trying to start AI projects was data. Much better now than it was seven years ago. Much, much better. But it’s still hard. So if you can’t get over that data hurdle, then people get despondent, they can’t reconcile what they’re hearing about the hype, with the challenges of doing it for real. And I’ll tell you, I’ve been around long enough now to know when somebody has never done a real AI project and just talks about AI… Because there’s a big difference in those two things.

Stephanie Morillo: I advise that they start off with figuring out their audience, their mission statement, and with actual drafting of their content before they get into figuring out what the right solution is. There is a section in the book that does touch upon different blogging solutions. Just to give people an idea, a sense of what’s out there, my general advice is to choose the platform that makes the most sense for you in that it’s not gonna be too time-intensive. For me, I started out with my blog – my blog was on Jekyll, and it was hosted on DO. In the beginning it was great, but it actually grew to be extraordinarily cumbersome, and I hated having to use Git hooks and SSH into servers in order to get a typo fixed. I was like “This is not gonna work for me. I’m never gonna use this site.”

So I actually moved over to a hosted solution. I moved over to Wix. For me it’s great. It’s all drag-and-drop, and as soon as I press Publish, guess what - it’s live immediately. I don’t have to wait for things to build etc. But I think we get too excited with tinkering and with the tech before recognizing what the actual point of the blog is. You basically just wanna choose a blog solution that’s gonna be relatively easy for you to maintain, meaning it’s not gonna be too cumbersome… If you don’t wanna find yourself having to update a lot of stuff, then you might wanna consider something else. Don’t go for the shiny technology first; figure out what your content is, and then have a for-real conversation with yourself to decide how much actual time you wanna spend on blog maintenance mode… Because it can actually take a lot of time.

Once you have this shiny, new thing set up, “Oh wait, now I actually have to maintain it” and “Now I actually have to work on blog operations in addition to being a content creator.”

So I recommend actually sitting down and thinking about your content first, before you start thinking about the solutions.

Justin Garrison: Even just the general – like, words matter. We still call a lot of people that manage databases administrators, as like a “You do the maintenance tasks. Like, you’re not doing engineering work. You’re an admin.” And SRE became this “Oh, you can be cool in ops again.” You can do cool stuff and be an operator. And database engineering feels different than database administration.

Sam Boyer: I’m not sure I can – well, the problem is I think I hung on to something you said earlier there, Johnny, about the difference between operating and maintaining. Certainly, there are differences, but I’ve had a few thoughts swirling around in my head about what it means to do maintenance. Because I do agree, we need a definition for what that actually is.

My sense is that there are two fundamental ways in which we can think about maintenance. One is fixing bugs, and the other is adding features. And those two things have tensions involved, but what I was thinking about when you were talking was how – I think there actually is a really important commonality between operational characteristics and development time characteristics. And to me, that’s failure locality. It’s the idea that I want the computers to tell me as closely as possible how the thing is failing, so that I can fix it. And that is true whether I am writing tests, because to me a good test is something where when it fails, I know right where to look, I know right where to fix, and that’s not something that I need to rebuild a ton of context for or somebody else can come in and do.

Similarly, when something is failing in one of its operational characteristics, I wanna know as closely as possible where to look, where to go. So I think there’s a common principle there in terms of the way that we should be approaching making – and that’s mostly on the bug fixing side. Mostly, I think, at the bug fixing side.

But maybe where the paths meet in the woods of the two approaches - the bug fixing and the feature adding - is the extent to which your tests and your telemetry and whatever systems you have set up for consuming your operational information are able to tell you when the thing that you were adding over here broke some stuff over there, and guides you to the same thing, because ultimately, what we’re talking about here is like “Does the software continue to be correct or not, and how can you tell?”

But I apologize though, because like I said, it kind of took off like a thing you said at the beginning, and I feel like I missed the tail end of it, which is why I was thinking about biding my time… But then Kris called on me, so… I’m sorry. [laughter]

Sam Boyer: I’m not sure I can – well, the problem is I think I hung on to something you said earlier there, Johnny, about the difference between operating and maintaining. Certainly, there are differences, but I’ve had a few thoughts swirling around in my head about what it means to do maintenance. Because I do agree, we need a definition for what that actually is.

My sense is that there are two fundamental ways in which we can think about maintenance. One is fixing bugs, and the other is adding features. And those two things have tensions involved, but what I was thinking about when you were talking was how – I think there actually is a really important commonality between operational characteristics and development time characteristics. And to me, that’s failure locality. It’s the idea that I want the computers to tell me as closely as possible how the thing is failing, so that I can fix it. And that is true whether I am writing tests, because to me a good test is something where when it fails, I know right where to look, I know right where to fix, and that’s not something that I need to rebuild a ton of context for or somebody else can come in and do.

Similarly when something is failing in one of its operational characteristics, I wanna know as closely as possible where to look, where to go. So I think there’s a common principle there in terms of the way that we should be approaching making – and that’s mostly on the bug fixing side. Mostly, I think, at the bug fixing side.

But maybe where the paths meet in the woods of the two approaches - the bug fixing and the feature adding - is the extent to which your tests and your telemetry and whatever systems you have set up for consuming your operational information are able to tell you when the thing that you were adding over here broke some stuff over there, and [unintelligible 00:14:05.07] Because ultimately, what we’re talking about here is like “Does the software continue to be correct or not, and how can you tell?”

But I apologize though, because like I said, it kind of took off like a thing you said at the beginning, and I feel like I missed the tail end of it, which is why I was thinking about biding my time… But then Kris called on me, so… I’m sorry. [laughter]

Ahmad Nassri: We should link it, yeah… Talking about the tyranny of complexity, and how we fundamentally, again, as engineers and as developers, and some of us more than others, have this innate enthrallment and thrill and enchantment by complexity, and we wanna just dive deeper into it. And some of us enjoy it so much that we just wanna create more of it. So that’s the risk and danger, especially when you’re working with a team. You’ve gotta be considerate and empathetic to the other team members that are currently next to you, but also to the future team members that are gonna come after you. Are we actually making the right choices here? Are we investing in the right thing?

And sure, you can create the greatest deployment system, the best library tooling, the greatest internal frameworks that you wanna create, but if you’re not investing, again, in that total cost of ownership, specifically knowledge and maintenance, you’re basically putting the middle finger to the people who are gonna come after you… Because you’re probably not gonna be in the same job forever, you’re not gonna be maintaining the same thing forever… You’re gonna either switch teams, switch jobs, switch companies, and who is gonna take care of this when you’re not around? Even if you’re passionate about it, even if you’re really driven about it, chances are you’re not gonna be there forever, maintaining this thing which you’ve created, or this investment that you’ve made… So how do you lay the foundations for the people that are gonna come after you? And typically, that falls onto the management to solve that problem; it’s somebody else’s problem now… But that’s why I say it’s the “No, and…” culture that is operating, around “We’re gonna make sure we’re making the right choices and investing in the right things, and not reinventing the wheel and not solving problems that are already solved, and rebuilding things whenever we can.”

I think this is also potentially influenced by these big tech companies, or even small ones, like npm, where on the outside, looking in, people all say “Oh, company X is building this tool, and I use it all the time. They must have the best engineers in the world, they must be doing R&D all the time, they must be creating the best-in-class tooling, and things. We should totally use their tool, we should totally use their solutions in open source, the things that they’ve created.”

[40:02] But the reality is - I’ve seen first-hand in many companies, not just the one you’re thinking - that when you go on the inside and you see things, it’s usually lacking empathy to future developers, and lacking planning of how we’re gonna maintain and scale work… And it’s just shiny object syndrome, where we’re chasing things and building stuff and tweeting about it and making a blog post, and like “Look how cool this framework that I’ve created is, and I’m no longer working on it.”

So I think from the outside in a lot more people should look more critically into companies and what they create and what they do… And to use the reference example that I used earlier in a positive sense - a lot of the Netflix projects that they’ve created years ago are no longer maintained, and/or they’ve abandoned them, and/or you’re not gonna get improvements on them because they’re too busy doing other things. Well, what happened? What happened is the shiny object syndrome, and they’re probably building another project now, another open source thing now, that you’re probably not gonna see for a while, but then when you see it, you’ll be amazed by it, because they made it sound like it’s the best thing in the world, and then six months after that they’re not gonna maintain it anymore… And now you’re stuck, because you’ve implemented it thinking that this is the best-in-class technology.

Daniel Bachhuber: So it’s fun being a part of an experiment in software development and sustainability in the WordPress ecosystem. Matt himself looks upon the WP-CLI project as “This is an experiment of investing into an open source project that could be replicable or might not be, but it’s something that we’re trying out and we’re open to trying out, and we’re interested to see what the results are.”

That’s kind of the way that it’s thought about, and many people in the WordPress ecosystem are very acutely aware of the challenges of developing open source software. Because there’s kind of two aspects of it. One is “Yeah, you just write some code and throw it up on GitHub and it’s open source and that’s great”, and some people use it, and then there’s kind of an associated maintenance burden, that sort of thing. But then there’s also this challenge of within a mature open source software ecosystem - there’s companies operating in that, and often those companies are actually competitors.

I wish I had said it at the beginning of the show, because one of the reasons that I love open source so much as I do is because I think it’s this very fascinating time of companies get to collaborate without complicated partnership agreements. You know, my employee here can submit a pull request to Airbnb’s repo, and that sort of thing… And there’s so much mutual value creation. So for a mature open source software ecosystem - lots of companies operating, and really what they’re trying to do is figure out how they can all collaborate for the better of the project, because that collaboration makes the project better, and ultimately makes their businesses better.

[55:56] It’s a very challenging process, but for WP-CLI – WP-CLI exists at the intersection of all these hosting companies that are dependent on it, agencies that are dependent on it, and other WordPress professionals. And they all want to see the project succeed and be healthy and do exactly what they need to and solve more problems for them. This is an experiment in trying to deliver that value to them in a sustainable way, because it’s not a solved problem.

Nikita Shamgunov: And one can argue that “Well, they’ve got their reasons for them to be different”, but there’s a more contrarian viewpoint here that is “We live in the world of clouds, where things are abstracted away from you”, and that gives an opportunity to build ideally a serverless interface that speaks SQL, and that gives you access to all your data. It gives you access to all your data for reporting capabilities, for low-latency capabilities for operational workloads… And that would allow you to never leave your data universe as you go and move from one workload to another. And that can be huge, because whatever data you trained - for example your example; whatever data you used to trained the model lives in that ocean of data, and that data is easily accessible to you. And then you trained the model, so now you need to convert new data that is incoming, or marry new data to old data and convert it into pixels, which is your app, or your website, or whatever… And that can be done right there, off of the same dataset you’ve been operating on… Which is certainly not the case today.

Today you have a data lake, a data warehouse, and a number of operational databases that can be integrated by a third piece of software, like ETL tools or integration tools, which just generates a lot of complexity… And a lot of that can be simplified if you imagine a world of having a serverless, SQL, low-latency API to all your data. That’s the vision where we’re driving towards, and this is a multi-year, probably multi-decade – MemSQL is nine years old today, so it’s gonna be a multi-decade kind of life’s work. But the workloads that we see emerging, and the new workloads that are enabled by a system like this are real-time analytics and real-time decision support, when you need to go back and look at the history of what was happening to make a real-time decision, and do it at scale as well. So that is something that we see a lot in financial markets…

[16:20] MemSQL is a give-or-take 40-million-dollar run rate company with 70% growth. We’ve just had an article on TechCrunch that revealed our numbers… And a good amount of that revenue is coming from financial markets. And if you think about it, that’s what happens there; there’s a constant stream of information that’s coming in, modifying that data state that you have, and you need to make decisions about buy/sell, you need to make decisions in wealth management, you need to make decisions in portfolio management and trading… But also, you need to make decisions in various systems that for example monitor something that’s very large. In Morgan Stanley for example it’s a trading system, and we monitor this trading system, providing decision support to “Oh, should we provide some sort of maintenance? Should we re-route our trades?” All of those things. That’s what MemSQL is used for today, and we didn’t have a system on the market that had those capabilities before.

Jon Thornton: Yeah. I think the house analogy can be extended even further, because sometimes you can end up building things that you didn't actually need. Those things can have maintenance or operational costs that are much bigger than a different implementation might have been. You can almost compare that to buying too much house, more house than you can afford, and then you're stuck with all of this maintenance overhead.

Justin Garrison: It’s a shared operational problem, because you’re putting some of the operational burden, the maintenance side of it into the development cycle, and you’re saying “Hey, you have to be responsible upfront to tell me what access you need, and all those other things.” And the way we’ve been doing it, like you said, Taylor, with the trash bag model… I think of it as like, my older son can do a lot of things. He has a lot of responsibilities, he does chores, he does all this stuff. But when he gets in trouble, he’s grounded. And a lot of times with a Kubernetes application, I just need to remove all these things that you’re allowed to do from your capabilities. I drop all capabilities, no network access, and then we’re going to add little bits back to it and try to build backup to say “Oh, you’re allowed to do this much.” And my older son’s like “That’s terrible. I’m never gonna do that again, because I know what it’s like to have all the freedom and all the system in the world.”

But for my younger son, he doesn’t have all those same things that he can do, and he’s building up from like “Hey, if I give him a couple things that he can do, and say “Hey, will you take out the trash for me, and also, I’ll let you play this video game”, he loves it. He’s like “This is amazing. I can do all this stuff.” And he’s sharing some of the responsibility, because he doesn’t know what it’s to have access to the entire system and the entire world and every environment variable.

So we’re either dropping everything for someone that has all the access, or building up from someone that hadn’t had it before, and moving that to the developer and saying “You need to tell me upfront what you need to do, and I will give you that access in production, and then we’ll both be happy as we maintain this stuff going forward.”

Bradley Meck Farias: So it actually went through around three iterations, and three different attempts to do this. The first attempt was “Okay, we will match npm’s interface. We’ll make our own CLI; it’ll have the same commands.” That’s a lot of maintenance, especially if npm updates. So this thing needs to work with multiple versions of npm. It needs to work on old npms, it needs to work on new and PMS. So that was scrapped pretty quickly.

The next kind of attempt we did was “Okay, what if we just invoke the npm CLI? It has a dry run mode built by default. Maybe we could invoke it twice. Once in dry run mode, once without dry run mode.” So this actually doesn’t provide enough information for you to have a good user experience. It won’t tell you exactly what’s being installed; it’ll tell you the number. It actually has all the data, you just can’t get it out of the CLI, for what it’s about to do, or would do in a dry run mode.

[22:14] So after that, we reached once again a level deeper, and we actually wrote a wrapper script that will still invoke npm, but it rips out a piece of npm and replaces it with our dry run wrapper that will run in a dry run mode before it does any sort of real behavior that writes to disk. This was actually fairly pleasant to write, compared to some other ecosystems or package managers.

We looked at a few plugin systems on other package managers. npm actually was in a unique infrastructure position here, where they use a library called Arborist. And we only really needed to replace Arborist, it looked like. So we just had to swap out Arborist, which is what it does if you set your log level really high; you’ll see this little thing logged called “Build the deal tree.” That is where npm does a full resolution of the entire module graph before it does any sort of removal of packages, or installation of new packages. And that is the only thing we really needed to replace with our first iteration of this. And so Node happily uses common JS here. This would be very hard if npm was written in ES Modules.

Matt Klein: Sure. I’m very often asked – you didn’t ask this, but I’m very often asked this question about “What is Envoy’s roadmap?” Because I think people these days have an expectation, partly through vendor-driven open source, that there’s a roadmap, there’s a product manager. And as a community-driven, technology-first project, we don’t have a roadmap. I vaguely know what people are working on, but things happen incrementally, people add features as they need them in their deployments or the products…

So when it comes to EnvoyCon, I think it’s not like we’re announcing a version for this conference, or we’re announcing some big bang feature; I think we’re just seeing the iteration year-over-year of the project becoming more mature… And at this conference we have great talks, that range from large-scale control plane deployments, to large-scale API gateway edge deployments… WebAssembly I think is going to be huge; that is the biggest thing that I think is happening in Envoy right now. For those that don’t know, this is basically embedding a WebAssembly runtime within Envoy, and being able to run extensions there.

[51:50] Historically, the way that people have extended Envoy from a code perspective has been to write Lua in a very limited set of cases, and mostly C++. And apart from C++ being difficult for most developers, the main issue beyond the C++ is that just because of the way these extensions are linked into Envoy, they have to be compiled together… It’s a very painful thing. So WebAssembly is really an amazing ability for us to have a stable API, a stable ABI, allow people to write extensions in Rust, or C++, or Go, or TypeScript, or whatever they want, and have that be separate from Envoy.

So I think WebAssembly is going to be huge for Envoy, both from an edge computing standpoint, security standpoint, observability standpoint… I think it’s just – we have so many extensions today, but it’s just gonna super-charge the ability for people to write further extensions for Envoy. So we have a bunch of talks about WebAssembly.

The security investment in Envoy has been huge, mostly from Google. They’ve done a fantastic job. And that’s not just on fixing bugs, or filing CVEs. For example, there’s people at Google now that are working on software supply chains… And again, for those that don’t know what a software supply chain is, it’s this idea that we have Envoy, and Envoy is hundreds of thousands of lines of code. But Envoy depends on millions of lines of libraries. And if you don’t understand where those libraries come from and what their CVE process is, and what their maintenance process is, you can wind up in a lot of trouble. And this is not a problem just for Envoy. This is a huge problem across the industry, that people don’t have a great understanding of software provenance and what all the dependency chain is.

So we have some people that are working on some really interesting stuff around better tooling, making sure that we track our dependencies, various other security concepts, various other dev-focused talks… So I am obviously – I’m actually not a huge conference fan, but when it comes to EnvoyCon, for obvious reasons, I absolutely love it. So in this year that we’re in, I am sad that we are not gonna meet in-person. It really does make me sad, because for the last couple of years I have loved seeing people in person. So I am bummed that we are doing this virtually, but I’m still excited for the talk line-up.

Jerod Santo: That does sound like pretty green grass. It sounds pretty good. I think that this could be a huge boon for open source projects, especially when the burden of maintenance between multiple operating systems is such a problem for many developers who even would love to support Windows users; they don’t even have environments that they can test against Windows in order to provide support. So it’s often a pain point. People, when they are calling for help, it’s like, “Please if you have Windows or if you are on Windows and you use this, provide help. Give us good bug reports or even help us fix these things.” Do you think it’s feasible that once this comes out and is out there in the wild, readily available for developers, that it would be socially okay to say, “Our Windows support is via Bash on Windows or via Ubuntu and Windows, and we don’t have any first party support for the operating system”?

Adam Stacoviak: Which makes sense for linuxserver.io, because that’s a big thing that they do, really, is they provide these images, but then they also provide compose templates right alongside of it, that is pretty much copy and paste, which is great for gateway people who are like “Oh.” Like we said before, Plex is a gateway; you find it, you learn about Docker, you learn about containers, you start going further into the Linux world, and next thing you know you’re at least a Linux novice, if not a version of an expert; maybe a moldable expert, I don’t know. That’s kind of where I feel – like, I’m not an expert level by any means, but I’m certainly not novice. I’m somewhere in that middle ground. And it really is because I didn’t want to run this thing on Mac. And the primary issue I had was not with the Mac platform, it was with the inputs; the literal hardware. I would have USB-C disconnections and data corruption because Macintosh computers, these Macs are not really designed to be NAS hardware; while they can be, the Thunderbolt port is so fraught with issues. They even have adapters you can adapt to the port to make sure the cable doesn’t wiggle at all.

And I would have this thing literally racked into a rack, like a Mac Mini or something like that, and any movement whatsoever on the rack, even if you move the chassis a little bit to do maintenance on your rack - which you are going to do - I would have port disconnection issues. And I’ve lost so much data – I’m just kind of done. This is not how it’s supposed to be. I love Mac, I love the operating system, I love all the ease of it… Obviously, I use it daily as a driver, and it’s my go-to operating system… But when it comes to that kind of reliability with a connected drive, Macintosh just doesn’t compare to what you can do with a motherboard, some RAM, a CPU and Linux on top of it.

Daniel Beutel: Yeah, there is a difference. The biggest difference, I shall say, is obviously in convergence time, because you have these rounds of communication, and also the averaging process has some impact there. Often, as researchers, we make these comparisons between centralized learning and federated aspects of it. The interesting bit is that this comparison is somewhat artificial, because it’s not something that one would face in reality very often. It’s either federated learning or nothing. We’ve seen this in the past, right? If you look a little bit at a journey that machine learning and deep learning now took - the summer around 2012 we realized that by making these models bigger, we suddenly get better accuracy. So there was this ImageNet moment, and then a couple of other moments like this afterwards. And we saw that we can achieve ever greater accuracy and other performance metrics with these models.

The thing is we always – when we read a research paper, for example, and when we look at these recent advances, it’s often quite fascinating, and it’s often in the context of web-scale companies like in Google or Facebook, who have these massive amounts of data in-house. But then, often in practice, there’s this realization that, “Okay, I read about this cool technique I’m trying to apply to my problem”, and suddenly, I don’t get amazing results that I expected to have. So the question is, what happened, right? And in many cases, the answer is really that the amount of data and the diversity that you have in your local data set is just not enough. And the interesting thing, and the thing that got us very interested in federated learning, was this realization that for many of those cases, you might not have a large data set on your own, but there are a lot of others just like you, who are facing the same challenge, and who might want to train the same model, but they also have some data, but not enough data for a very good model.

I mean, we could obviously solve this if we could put all of this data just in a single destination, in a single cloud account, and then train a model on it. But that’s something that just doesn’t happen. It doesn’t happen for regulatory reasons, it doesn’t happen for confidentiality reasons… For example, corporations, they have a lot of financial data, and they might want to have models that predict certain aspects about these data, but again, it’s a thing of confidentiality. It’s something they would never share. And the types of use cases that federated learning gets used for, sometimes we are surprised ourselves from where exactly these companies are hesitant to share data.

For example, there was one case where a couple of manufacturing companies - they’re all operating the same manufacturing machine, and they want to train a model that does predictive maintenance, basically, for this machine to predict whether this machine is likely going to fail, so whether they need to do some manual maintenance or something like that. And one would think that this is the case where they could just collaborate and they could just put all of their machine sensory data in a cloud account and train a predictive maintenance model. No, they don’t. Why don’t they collaborate on this? Well, the reason is that the data that they have from running these machines could allow others to see how often they run these machines, which could allow others to draw some conclusions about how many parts they’re producing, which is highly confidential. So even in those seemingly easy cases, in reality, it’s not that easy.

Gerhard Lazu: Yeah. But I think even for that… For example if you had read replicas, you can continue operating in a degraded way. For a bank, what would that mean? Well, for a bank - you know what, no. That’s too long. I’m not going to open that can of worms. I’m not going to open it because I know there’s so many things; there are regulations as well, penalties, all sorts of things. And then people say “Okay, we have to do this, we have to do that” and people are just scared. But I see a lot of fear there, and I see a lot of like “This is too hard. Let’s just do what we’ve always done. Maintenance windows - okay, job done.” And then you know someone has to be around during those maintenance windows. Anyways… For another podcast maybe. Not another podcast; for another episode, maybe.

So what projects from the cloud-native landscape do you like most? And it should be the ones that you use day in, day out. I think we’ve already established it is Kubernetes…

Luis Villa: So I’m a former programmer… I got a CS degree in the last millennium, and worked in open source for a while. I got involved in open source while I was in college… Actually, originally, hacking on the LEGO Mindstorms, the very first generation of LEGO Mindstorms… And also just as a Linux user. This idea that people were building an entire operating system together on the internet was like… But I was also a political science major, and so I was very interested in this overlap of politics and power and computing.

It started off as just like I was interested in politics and I was interested in computers, and I really thought those two weren’t related… And then by the end of the ’90s it was like “Oh yeah, actually these are super-related.”

[04:04] So I worked at a startup called Ximian, which worked on the GNOME Linux desktop. Then, after that I was like “Actually, law school sounds like it would be fun.” Pro tip, kids - law school, not fun.

Since then I’ve worked at a series of Mozilla as an attorney, where I worked on the Mozilla public license, revision version 2.0, I worked at a big law firm for a while, working among other things on the Google Oracle lawsuit for Google, where fair use came up quite a bit, which is something we’ll talk about today. Then I worked at the Wikimedia Foundation, and now I’m the co-founder of a startup called Tidelift, where we are trying to make open source better for everyone by helping build a sort of economic and payment loop so that maintainers get paid to do all the sort of not fun parts of maintenance. They’re not fun, but they’re really important for businesses and enterprises, so we’re trying to close that loop as a business. But yeah, I’m a copyright nerd at heart, so I think that’s sort of why I’m here today.

Luis Villa: And also just as a Linux user… This idea that people were building an entire operating system together on the internet was like… But I was also a political science major, so I was very interested in this overlap of politics and power and computing. It started off as just like – I was interested in politics and I was interested in computers, and I really thought those two weren’t related, and then by the end of the ‘90s it was like “Oh yeah, actually they’re certainly super-related.”

I worked at a startup called Ximian, which worked on the Linux desktop… And basically then after that, I was like “Actually, law school sounds like it would be fun.” Pro tip, kids - law school, not fun.

[04:09] And since then I’ve worked at a series of Mozilla as an attorney, where I worked on the Mozilla public license, revisioned version 2.0. I worked at a big law firm for a while, working among other things on the Google/Oracle lawsuit for Google… Where fair use came up quite a bit, which is something we’ll talk about today.

Then I worked at the WikiMedia Foundation, and now I’m the co-founder of a startup called Tidelift, where we are trying to make open source better for everyone by helping build a sort of economic and payment loop so that maintainers get paid to do all the sort of not-fun parts of maintenance… They’re not fun, but they’re really important for businesses and enterprises, so we’re trying to close that loop for the businesses.

But yeah, I’m a copyright nerd at heart, and so I think that’s sort why I’m here today.

Andrew Guenther: Oh man, there’s so many great questions to unpack here… So I’ll try and go at it one at a time. So one of the saving graces to some degree is that as we launch more satellites, they’re all based on the same hardware designs. Very minor, minor revisions between them. Like, you have a satellite that works - don’t mess with it. Continue to launch more of the same. But then also, on the flipside, when the first one goes up, and we realize “Oh, we really should have done some things differently”, as we learn, that iteration cycle is even slower, so there’s a lot of things that we have to kind of deal with on ground, and we’re making notes for what the next gen hardware is going to look like, and additional concerns…

[00:11:42.02] And when you talk about what kind of packages are we going to use, that’s a huge concern of ours. Again, it’s running 18.04 in space, we’re trying to do machine learning and data analysis, and a lot of those libraries move very fast. They’re very quick to drop support for older operating systems… So we have to make the call as a small team, are we going to compile these ourselves? Are we going to build our own versions of these dependencies to maintain them? So we’re very cognizant, especially on the onboard data processing side, of what libraries we pull in… More so than anywhere I’ve ever been. Because not only is maintenance a concern, size is a huge concern. Pushing software updates to space is hard. It takes a while, you’re gonna test the hell out of it, and you want to make sure that it works. I always like to pick on Node.js, because you have like the Npm package system, where just everything sprawls out to infinity. You install –

Andrew Guenther: [00:23:53.00] Oh man, there’s so many great questions to unpack here… So I’ll try and go at it one at a time. So one of the saving graces to some degree is that as we launch more satellites, they’re all based on the same hardware designs. Very minor, minor revisions between them. Like, you have a satellite that works - don’t mess with it. Continue to launch more of the same. But then also, on the flipside, when the first one goes up, and we realize “Oh, we really should have done some things differently”, as we learn, that iteration cycle is even slower, so there’s a lot of things that we have to kind of deal with on ground, and we’re making notes for what the next gen hardware is going to look like, and additional concerns… And when you talk about what kind of packages are we going to use, that’s a huge concern of ours. Again, it’s running 18.04 in space, we’re trying to do machine learning and data analysis, and a lot of those libraries move very fast. They’re very quick to drop support for older operating systems… So we have to make the call as a small team, are we going to compile these ourselves? Are we going to build our own versions of these dependencies to maintain them? So we’re very cognizant, especially on the onboard data processing side, of what libraries we pull in… More so than anywhere I’ve ever been. Because not only is maintenance a concern, size is a huge concern. Pushing software updates to space is hard. It takes a while, you’re gonna test the hell out of it, and you want to make sure that it works. I always like to pick on Node.js, because you have like the Npm package system, where just everything sprawls out to infinity. You install –

Ben Cohen: Yeah, so the way I like to introduce Swift is actually - I go back to when I first saw Swift, and what really got me into Swift. So I actually wasn’t a member of the team when Swift first launched. I joined about a year or so later. And at the time I was actually working in FinTech, I was working for a bank, working on trading systems.. So we were a team, doing sort of equity and derivatives trading, we had full stack development, server-side and UIs… And we were using various different languages. So we were using, on the server side for these trading systems, that really needed high performance and low latency, we – the developers separated into two camps. You had the Java programmers, and the C++ programmers. And the Java programmers, they were really fighting the garbage collector all the time. Because every millisecond counted, they were making sure not to allocate too many objects, and using ring buffers of scalar types, and things like that… And they would look at the byte code to make sure everything was going well. And the C++ programmers found this hilarious. They were like “Why don’t you use a proper language?” But it wasn’t so funny when C++ code would segfault in the middle of the trading day, right?

So that was the server side. You had two camps of developers who had chosen a language, and that forced them into a pretty tough choice - safety versus performance. Then we had UI programmers; they targeting Windows, they were using Visual Basic, and then C#. C# - kind of a nicer language, I would say (this my personal opinion) than Java. It came out slightly after Java, and I felt like it was slightly improved, similar style. Obviously, they were enthusiastic about that, but they were, again, sitting on top of a garbage-collected language. And then we also had less latency-sensitive things; we would have Ruby programmers writing Ruby on Rails apps for things like static data maintenance.

[11:53] Ruby is such – I remember I actually listened to one of your podcasts where you had Justin Searls on, and he was talking about how Ruby, how they really focus on the joy of programming, and how it was such an enjoyable language to program in. But obviously, it doesn’t scale to the kind of latency-sensitive environments that you need for that sort of thing.

So it was really interesting that we were all using all of these different programming languages. And obviously, I think people should learn different programming languages; it sort of expands your brain, and makes you a better programmer. But it was kind of unfortunate, the choice of language really limited them in some way. And so my first impression when I saw Swift, when Chris announced it on stage, was like “Yes, this is what we need.” Because it was a non-GC language, compiles down to native, but it had that higher-level feel that actually made it really enjoyable to program in, a lot like Ruby.

So I think the sort of ambitious elevator pitch for Swift is that it’s something that you can achieve C++ level performance in. Obviously, you have to work at it; nothing comes for free when you’re actually operating that level. But the language itself feels a lot higher level, a little bit more like Ruby. Obviously, it has a static type system, but we worked hard with things like type inference, to make sure that the types aren’t in your face… We try and put an emphasis on progressive disclosure, so that – obviously, one of the key things for us is we want app development, which is the main demographic for Swift, to be easy to do. We want somebody to be able to sit down, not have the language get in their way, but also, we want the language to be powerful enough that the framework developers can create things like SwiftUI, that is like this super-high level programming language that allows people to be very productive, put together apps super-quickly. And so I guess that’s how I’d characterize it. It’s a native language, but with a high level look and feel.