Aidan Feldman: Yeah, we already have external customers, like the Environmental Protection Agency launching something that their teams are building…

Jerod Santo: Right. That reminds me, you have this great quote from the Data and Goliath book. You said “Data is the pollution problem of the information age, and protecting privacy is the environmental challenge.” I like that casting of that. I think that plays well into this whole Solid idea, isn’t it?

Salvatore Sanfilippo: Now I see younger developers not really caring about this, but we basically started to write code, and starting to be mini lawyers, understanding all the subtle things about GPL, BSD, MIT… So yes, I care about license, because the way that you express your willing about what others can do or cannot do with your code.

Also, I think that without the copyleft idea, the computers technology could not accelerate to the point that it accelerated. Because when open source was created, basically in order to create a startup, you had to buy complicated workstations, Unix licenses, database licenses… So it was impossible to have the landscape that then it was created by the open source movement. So for me, licensing is very important.

Initially, what I did with my software was to use GPL. Then I started to realize that GPL had two problems. It created problems to myself, because sometimes I thought – and if this becomes big, I want to have a business model. And I don’t want to get some paper signaled by all the contributors that I had so far. So I started to switch to BSD, saying “This is a protection for me. Also, it’s a protection for other people based on the environment they use my software.” And sometimes it can be a problem, even if they don’t want to violate the license, and stuff like that.

[00:32:00.06] When I started Redis, I was very into this BSD stage of my life, so I released the Redis as BSD also because there is also behind that an idea about accelerating society, improving society to be more important than basically what is going to happen to me, in some way. However, then it must be said that the cloud situation changed the landscape, because even if it was very complicated to create a product business model, even before - basically, Red Hat was the only one that really succeeded in this kind of game in the open source, and a few more… But still, if you wanted to sell services, you were the to-go person as a creator of the software.

And then after AWS everything changed, because there was no longer need of somebody supporting you, because it was handled for you. And also, you couldn’t even compete with the others, because in order to compete in cloud services, you have to pay for the instances, and they have them for free. Also, even the billing is complicated. There are many companies that just because of billing will just get what AWS has, and stuff like that.

So I understood at some point that the BSD license that I picked, with the changing world of the software, created serious issues to create business. Now we can go a step back and say “But why it should create business? It’s an open source software.” And I believe that more or less every complicated open source software has in one way or the other an economic system behind it, because it’s a lot of hard work for many years. Either people are paid very well, or they will not afford to do all this kind of work. So I believe that both things are needed to redistribute to the community, and also.

And inside Redis, they didn’t want to change the license, and I didn’t want for a long time… But there was this discussion, but it was some kind of taboo. So it never happened as long as I was there that somebody asked me “But what do you think? We want to change the license.” It’s a conversation that didn’t happen. I just created the module system, and Redis the company started to have the modules that were enhancing Redis capabilities, and that was it.

Then when they changed the license, I understood that it was basically some kind of a forced move in some way, because with BSD it was too complicated to compete in this market. However, now we are realizing - me and also inside Redis - that SSPL was not accepted by the community in some part. And we care about this thing, because you know, I don’t believe that SSPL is a terrible license, because it’s very similar to other GNU licenses. It’s just a couple of sentences… But the reality is that culturally it’s not accepted. And so we are starting to discuss inside the company about this problem.

[00:35:52.25] Also, one important thing is that because of that, we are going to add in Redis a lot of the features that are now only for the paying users. For example, now I’m working a lot to vector sets, which is the first fundamental data type that Redis gets after many years… And it will be released in the community edition, like normal Redis. And like everything I did in the past, it’s no dependencies, so it compiles because the data structure – so the HNSW data structure for vector similarity I wrote from scratch, the quantization I wrote from scratch, the hybrid search… So it’s some code, it’s like 6,000 lines of code in total, the other Redis data structure. So you can open the code and understand how it works. And it was impossible before to do that, but still, maybe there are setups that make everybody happy, enough protection and the community will be more happy. So there is an ongoing discussion inside the community now. I’m not sure what will happen, but we are focused on the problem.

Justin Searls: Yes, well - I mean, unfortunately, it is all too common of a situation, and I think that it is challenging to write tests that are not susceptible to several specific things that contribute to what is commonly called brittleness or flakiness, right? The most important thing to understand as we’re approaching this question from the outside in is what is our goal to have this built in the first place? And the goal is probably to have some sort of confidence that things are working. And if we get one green build out of five and no source code changes in the process – like, my confidence is not high, right? And we know this based on kind of intuitive experience that if it passes one time out of five, it means that you have proven the system can work. And you’ve probably also proven not that the system has some sort of fundamental flaw and will break in production, so much as you figured out that there are environmental timing or ordering implications that can cause one or more of your tests to not work.

And I think the first thing that a team should consider when they are running into this problem is to get back to consistent green builds as fast as possible. Because again, if you’re thinking about testing as ROI, if all nine of those people are getting an email every single time that the build breaks, and then say three of those people just independently start racing to go and screw around with timeouts and stuff - that’s a lot of dollars flying out of a window, because of one little tiny thing that might not be well understood. And then no one was no one woke up that day and was like “I’m going to really wrangle all of the flaky stuff in our test suite”, right? They’re all trying to do something else, and this is the thing blocking them from that thing. Not to say they’re half-hearted in their attempt to fix it, but what they’re really needing is a salve in that moment, as opposed to a solution.

So the first thing that I would do is I would lockdown everything. Normally, I don’t like freezing time in the system, right? But I’d probably start with the common quick fixes that I can apply, like “Hey everyone, it is now 2019, August 3rd, it’s a Tuesday, and it’s 11:33 PM, and we’re just going to lock that whole server down that way”, whether we’re using a test tool to do that or Unix. “Hey, everyone, we’re also going to no longer randomize the test order, we’re going to use this particular seed that is known and good, that we’ve seen work.” “Hey, everyone, we’re going to change all of the directory globs that are currently in an unspecified order in Linux, and that’s why Linux builds are failing, but on our Mac where it’s the alpha order, those are all passing.” Like, “Hey, we’re just going to do a sort on all of those glob requests everywhere, so that we’re just loading alphabetically, because it doesn’t really matter…” And we’re going to like go through the half dozen or so quick hit things to just try to get to consistent builds as soon as possible. And hopefully, that’s one person one day, if ever.

Jerod Santo: A spreadsheet to control your Kubernetes cluster

The pitch for xlskubectl (terrible name, obviously), is amazing: “You can finally administer your cluster from the same spreadsheet that you use to track your expenses.”

Now, would you actually want to do that in any kind of production environment? Probably not, but this is a fun project and it brought forth two thoughts I’d like to pass on:

1. Kubernetes exposes a robust API. That makes it far more useful/malleable than otherwise
2. People absolutely love spreadsheets. (I’m using the word love, here.)

Autumn Nash: I think it’s also perspective though, because think about it, when you go to school for computer science, or you first get into computers, you’re writing Python, or Java, and nobody tells you about DevOps, nobody tells you about scripting, unless you somehow just stumble on that.

So I think for people that maybe started in systems, or started with scripting, yeah. But it’s a huge barrier for people who just started writing the high-level code, and then all of a sudden they’re thrown into a production environment. Well, now they have to manage infrastructure.

I think you have more people coming from the, like, writing code… Because think about it, when you hear bootcamps, or school, or you go to these different ways that people are being educated to get into the tech industry, everybody tells you about the code and nobody tells you about the scripting, and the version control, and the DevOps, and CI/CD. So I think this is a great tool for those people, and making it more accessible to get into DevOps and CI/CD, and to be able to maintain your infrastructure… Because everybody talks about making this really cool app, and nobody talks about maintaining that really cool app, and releasing it. You know?

Matteo Collina: Otherwise your staging environment is not equal to your production environment, and then you are in trouble.

Chris Benson: Yeah. It feels like you just gave a eulogy for Jupyter Notebooks, to some degree. It’s a reminder that things are changing constantly. So you bring a great point that we’re taking some of the same challenges that we had in that environment, and we’re just recreating them in the newer tools that are out there.

There was another company I worked at - I won’t name the company - before I was at Lockheed… And at that company, I remember thinking there were people at the time that knew - they knew the AI modeling bit, and there were people that knew the software bit, but they never seemed to cross over. And when you raise the point about your kind of in-process development workflow, and then how do you actually get that to some level of production, I think there’s a lot of people out there that aren’t going to know that. And unless times have really changed in that area - and my gut says they probably haven’t; people tend to focus on the thing that they want to do. What is the right development workflow, and how do you start getting to that production environment? I know you’ve gotten tons and tons of experience at that in recent years. How do you think about it? Can you frame it a little bit before you dive into it?

David Heinemeier Hansson: …like the Rust compilations that were the thing before we had SSDs and NVMEs and so forth… It wouldn’t have worked. SQLite would not have been a great storage for caching, for example. It would just have been too slow. You have that freaking reader head on those disks that have to move. It just can’t do it at the same pace that RAM can do it.

So SQLite, I think, started out with a certain context, and that context was largely embedded situations; you can put it inside your app for a single user - great, fine. And it was held back by this constraint that the underlying storage is quite slow, as it was before SSDs, and then also that compute was quite slow. So the amount of work a single machine could do in, say, when we started… The first box we started Basecamp on was a one-core Celeron 2.8 GHz machine. That machine today is two orders of magnitude slower than what you can buy for the same amount of money.

That means what in 2004 required 100 machines, and the coordination of how they access a shared data store and whatnot, can now be done by one computer. And there is a vast, vast simplification possible when you can remove the entire network layer of distributed computing. When you can run an even large application on one box. That is really, in many ways, kind of the golden ticket for simplicity, is to run one box. And that’s what SQLite is optimized for. It is optimized to run on a single computer, on a local drive, and then it can actually be staggeringly fast. Almost unbelievably fast, expectation premise busting fast.

This was one of the things we found out when we started doing these benchmarks, is “Holy smokes, how is it that SQLite, for example for the cache use case, can be in the same realm as a Redis, that’s moving things around purely in memory?” That’s just incredible. And then it’s even more incredible if you introduce that Redis or Memcached or whatever lived on another box, because now you’re limited by the network layer… You take the network layer out and you’re writing straight to disk… This is just incredible.

So real a-ha moment for me when you started putting all these pieces together. Computers have gotten so much faster. Storage have gotten so much faster, and so much bigger. I mean, you can now – terabytes of storage is trivial in terms of cost. Again, wind the clock back 10, 15 years, it was really expensive to get terabytes of storage. And then it was also slow.

[00:40:12.21] So the fundamentals have changed, and Rails 8 is trying to reap the maximum value out of those changes, and rediscovering the best things from the past that we could only use in certain contexts, and then go “Well, things have changed. Now the context is so much wider.” SQLite is no longer just a development database. It is a completely credible production environment for a lot of applications. Not all of them… But this is classic innovator’s dilemma. So this is one of the things that happens. The innovator’s dilemma is that a new entrance comes in. It starts looking like a “toy”. “Oh, that’s just for the low end.” “Oh, that’s just for the cheap stuff.” And then step by step, it moves up the ladder, because it has a fundamentally paradigm-shifting advantage. For SQLite it is. There’s no server. It’s so much simpler to run. It is so much cheaper to run. You can run it so much wider. Unless you’re working on that file, there’s no overhead. There’s no process even. And that just – probably an order of magnitude in terms of simplification, of how you think about data, how you think about databases. And it’s then no longer held backed by those constraints and we keep moving up the ladder. And we’ve moved up so far now that Rails 8 uses SQLite for literally everything out of the box.

We use SQLite for the jobs, the queuing backend, we use SQLite for caching, we use SQLite for the web sockets coordination… And obviously we use SQLite for the main database that your domain models are being persisted into. That’s all of it, which means that the deployment story gets so much better, gets so much simpler.

You don’t have to set up a MySQL server, and nurture that server, and feed it all the things that needs to be fed. No, you just have a bunch of files, and the only consistent or persistent process that you have is your application server. That has opened up all these other doors for lowering the cost of deployment. This is one of the things that Rails 8 is really a lot about, about the fact that you don’t need platform as a service. Rails 7 - all about no build. Rails 8 - all about no PaaS. No platform as a service required. No Heroku necessary. Certainly not with all its extensions. Now you can use any $5 VPS, or your own server, or your own hardware, or whatever, to deploy. And you can move between these things.

This is the other thing… I have tremendous respect for Heroku. It’s actually mind boggling how far ahead they were of the entire rest of the industry, and for how long. You would have thought that advantage would have been erased in a couple of years, and they had it for a decade.

Daniel Whitenack: Yeah. And I would definitely recommend people check out the Prefect docs. There’s a quick start in there… Just to give people a visual of this, I see you have this kind of converting your Python script into a Prefect workflow type of thing… That’s really helpful; there’s sort of these definitions of a workflow related to getting information about a GitHub repository, like contributors, and repo information… And then those are in Python functions, like definition [unintelligible 00:29:38.14] contributors function, and then above that you kind of put these decorators task and flow decorators with options to convert this into a Prefect workflow.

So part of my follow-up on that, just to kind of really hone in practically, to give people a sense of what it’s like to do this… So let’s say that I’ve converted my Python code into one or more Prefect workflows. I can run that, in my understanding, if I’m looking at the docs right, I can run that just Python my workflow, but obviously, like you said, there’s a deployment element to this… So could you talk a little bit about – I know one of the things that’s always a struggle in my past is maybe getting something working locally, and then because I’m going to run it in a different environment, or something like that, then I deploy my workflow on top of other infrastructure. And it’s hard to just connect that local development debugging and development staging production environment… So practically, what does that look like in the Prefect world in terms of going from that Python command to run your workflow locally, to watching things flow through your workflow in a nice dashboard that you have pulled up in production?

Autumn Nash: So that way it can’t be changed… Because I think it sometimes is used in certain production environments where they’re used to audit, too.

Łukasz Langa: Yeah, production environments. However, I would be a little more optimistic about this, saying that if you are an end user, probably the amusement park is still being built for you. So this is not something that is ready for end consumption, unless you are really just trying to just see how pure Python works, without all the libraries that you depend on. However, if you are a maintainer of a library that a lot of people depend on, you might be interested in checking it right now.

So we already see plenty of interest from data science libraries, and web frameworks, and so on and so on. And that’s exactly the right moment for them to identify whether there’s any blocking issues, or whether the gains are not enough for us to justify the change… Or maybe the opposite, or maybe they’re amazed at how better things are right now.

Funnily enough, even things that were not ever designed with free-threading in mind, like Async IO - you run the test suite of Async IO and it’s just faster on free threading, because it just can magically utilize the fact that it is now truly parallel.

So there is promise there. Definitely, we expect that once the library authors actually go through some of those hoops that are required for their things to be supported by the free-threaded version, we’re going to see uptake on more libraries, and then more users, and so on. So now is the time for the library maintainers to check it out. But for end users, it is a little soon.

Ben Burkert: Yeah. Well, first off I’d say I don’t think you’re unique, or – your experiences, I would say, lots of developers out there are in the same situation. And I certainly was annoyed, and this was a problem in my places I’ve worked… And my typical solution was to just turn off TLS locally; just figure out a way to disable it, and really not get any of the dev prod parity. And I think that’s kind of typically what developers do, is they figure out a way to bypass TLS in their local development. And that’s not so great for dev prod parity. It also can be a challenge - you know, if I’m a Go developer, and I’m writing gRPC services, that can actually impact how I develop locally.

For example, gRPC has features like bidirectional streaming. The client can create a connection to a server, and then while the server is responding, the client can keep kind of pushing parts of the request over to it. So that bidirectional streaming depends on HTTP/2. And HTTP/2 doesn’t require TLS, but it kind of uses TLS by default. And so there’s ways of working around the lack – if you lack TLS, you can still do HTTP/2, but you have to kind of do this funky, complicated upgrade procedure.

And then your development, if you’re doing all these things in development, you’re further drifting from what your production environment looks like. So it’d just be better if you as a Go developer building backend services with gRPC - you want to use gRPC streaming - it would just be better if it was easy to use TLS everywhere: development, staging production. And yeah, that’s one of the – besides the gains in security, there’s actually gains as an application developer to using TLS everywhere.

Justin Garrison: Yeah, using Docker - it was like this great boon for production environments. It was like “Hey, we can ship something that’s reproducible, and we can distribute it with these registries…” But also, developers now have a much more complicated environment to spin up. And sometimes they can reproduce it easier… Right? Because I can take a Docker Compose file from an open source repo or an internal repo and say “Oh, just run all the components here.” But how I get into those containers, how I manage and mount files into them, how I do all this stuff locally becomes really difficult. And when you add a layer of Kind, or Kubernetes API on top of that, you’re just like “Oh, actually – I don’t understand how big production is, or how complex it is. I don’t want to bring all that complexity to my machine.”

Justin Garrison: It seems like the further you go and the more complex a production environment, just the more dependencies you have. And you have to at some point be able to like collect all those dependencies to actually work together at some point. There’s still glue, and there’s still these little pieces that are going to miss, of just like “Oh, I have a Hello World thing. All I need is a load balancer and some compute, and I’m fine.” And then once you’re in production, it’s like this crazy mess of “Well, I have three tiers of load balancing, with these other random databases, that are various versions, and they have cruft in them, because we’ve done some migrations, and we’ve done some things…” In the real world that always looks so much different than what that clean “I can erase and redeploy this at any time” and “I’ve lived with this system for five plus years.” Those are very different environments.

Bill Kennedy: …so the conversation on Twitter. And kind of what I teach, and how I manage developers. So for any backend development that I’m doing - no, let’s actually step back. I have a general rule, okay? If you can’t use the tool to identify a bug in production, you cannot use the tool at your desk when you’re devving. That’s the general rule. Okay? So again, layer one rule - you can’t use it for production, don’t use it at your desk, because you’re not helping yourself. So for backend devving, I’ve never been able to connect a debugger to anything that’s running on a production environment, and use it to help find a bug. So I don’t like to, nor do I want my developers on my team using a debugger at their desk when there’s a problem happening, even when they’re in development.

Now I’ve got a 20-minute rule that sits on top of that. So if you’re working on this for 20 minutes, and you’re still not able to apply enough logs, and your mental model of the codebase is still really weak, to the point where you can’t find this bug, then I will allow you to turn the debugger on for backend dev. However, once you find it, we have to sit down. Because I have to understand why the debugger was needed to do this in the first place. Is the code a mess, or is this a mess? Or you don’t know enough? What is it? I need to understand that.

Now, when it comes to frontend dev, which unfortunately I’ve been doing too much of. My hair has really turned gray over the last few weeks thanks to JavaScript… Chrome has a debugger in the browser, right? So we’ll occasionally use it instead of console logs, because it’s there, and I can use it for any site that’s in production. It’s a tool that’s available to me in production, so it’s okay for me to use it even when I’m devving.

Now, let me say one last thing before I throw this baton over to Matt. I think the debugger is an amazing tool when you’ve been given a codebase that you have zero mental models of, talking even backend, and you need to gain those mental models. Looking at logs and just code may not be enough when you first join a team, or a company, and go crazy with the debugger for your first 2, 3, 4 weeks. But I think your goal needs to be to get off of it as quickly as possible, and start using it less and less and less, to the point where you don’t need it anymore. I can’t tell you how long that should be, because I’ve seen codebases that are ridiculously complex, and it takes months. And I’ve seen codebases that aren’t, and you only need it for a week or two.

Anyway, that was what I was sharing on Twitter, and we had people going back and forth between agreeing with me, and Matt, who didn’t agree with me. So maybe now Matt can kind of give us his sort of philosophies on this.

Johnny Boursiquot: I definitely wanna touch on sort of – and I’m gonna paraphrase what you just said, Ian, in terms of sort of expectations. What is the expectation from a performance standpoint of this particular service? Because I think those expectations are sometimes and often in production environments from a resource standpoint translated into a certain CPU allocation, or a certain memory allocation. I had sort of a nasty surprise when I first started working with orchestration tooling, for the Dockers and these kinds of things, and realizing that “Oh, my program–”, the way I experience the output of my program is that it’d be working and doing things, and it would reach a certain threshold, and all of a sudden, it’s almost like it would just stop dead in the middle of trying to perform some sort of action. It would just stop. And I’d be trying to debug, why is it that this thing was running, and all of a sudden it just stopped? No stack trace, no error message… It just got killed off. And I realized that “Oh, crap, the orchestrator–”, whatever, whether you’re using ECS, or Kubernetes, or whatever it is, that environment allocated a certain CPU and memory resources for that particular service, and whenever I would go above that threshold for a certain amount of time, my process would get killed. I didn’t realize my services were just being killed off by that environment. And the moment I realized that, I’m like “Oh, snap.” To Ian’s point, you kind of have to know ahead of time, and work out with - whether it’s either working with the ops team, or maybe you are the ops team if you’re doing SRE stuff, and trying to understand “What is it that I need to accomplish ahead of time?” Which I think feeds into sort of knowing what your application or services is supposed to be doing with whatever data it’s supposed to be doing it… Which kind of leads to, if you’re going to have to deal with a certain volume of data, and you think you have a problem with how you’re handling it, what’s the next step? How do you go about trying to figure out where your problem is?

Techno Tim: Yeah, it means so many different things to so many different people… But for me, I try to bundle it all up into having an environment, whether that be virtual, physical, but just having an environment where you can tinker and play around with technology, without the fear of breaking things, or taking down a production environment. A lot of people that get into homelab are in IT, and so they inherit this environment from their enterprise, and they inherit this architecture from their enterprise… And they don’t get to tinker much outside of that. You’re playing in someone else’s playpad, so to speak.

Adam Stacoviak: What’s the percentage of production environments or deploys that would be greatly benefited by having your paid-for control plane?

Bryan Boreham: [38:08] Oh, good question. Well, so first of all, the mechanism is basically that you supply the compiler with a profile, in the pprof format; that is a record of what the program was doing when you ran it doing its normal thing. So the compiler can then look at that and say, “Well, I see that 80% of the time in this program was in this one function. So I’m going to change the rules that I apply…” So there are certain rules inside the compiler, for instance when am I going to in-line other functions; and it normally only in-lines really small functions. But if it sees that this one thing is 80% of the whole program at runtime, then it can say “I’m going to inline a bunch more things, and I’m going to really change the rules on this one. I’m going to go all out for performance just in this one place.”

So the Go team themselves said that the PGL, profile-guided optimization gave them a 6% performance improvement on the compiler, and their benchmarks on the Go compiler. So it’s obviously case by case, but it’s a little bit more work, because you have to come up with some kind of representative profile. Maybe if you only work in one environment, that’s pulled from your production environment. If you have a wide range of use cases, you maybe use benchmarks and profile that. So…

Kevin Ball: They built, yeah. And it was easier to do in that environment because they were operating in a world where even the development environment was connecting to a cloud database… So you didn’t have this local versus cloud split in the same way. So the environmental change was not that different… Though if you’re already implementing a cloud environment for your production, and you’re already thinking about how that works, adding a local prod version of that, so long as you have your tunneling things figured out, which is kind of the key question, is like permissions and tunneling, and how are you doing it… We were operating in a trustless mode for everything; we didn’t have, “Oh, this is a trusted network and this is not”, which once again, that setup makes that a lot easier to generalize… If you’re having the “Oh, we’re inside the cluster, so we’re trusted” type of setup, it’s a lot harder to implement a local production environment.

Coming back to this question of debugging - you’re totally right, Jerod, a lot of debugging is in flow with development. When I think about debugging, I don’t even think about that. That’s just like, that’s development. Okay, I’m working on that in whatever the tool is for the job. And when I think about debugging, a lot of times the things that I would get brought into, especially my last few positions, it’s like “This is something that happens somewhere in production, and we don’t know why, and we don’t know what reproduces it.” So sometimes it’s figuring out how do you get your reproducibility, and in some cases there is no way to get to reproducibility. It only happens when certain things align.

[34:15] And so then it really becomes about how do you – it is this observability question, extension of logging, of like “How do you get enough stuff in place in your application environment, so that when the problem happens, the information is captured, enough information is captured that you can reproduce what actually went on there?” And I do think in that situation – the “console.log” debugging is a closer mindset to that, but it’s really… Like, we’re starting to talk about observability, and this question of “How do you log out sufficient application state to understand what went on when you’re looking at it after the fact?”, rather than “I have an environment where I’ve reproduced an issue, and I’m doing things, and I’m able to tinker and go back and forth”, which is where both the inflow of development debugging happens, and also where a debugger or tools like that tend to be very useful.

Thomas Eckert: No, I think you’re hitting on an entire class of really difficult bugs, where the actual source of the bug is the environment in which the code is running… And so it could be mysterious. And I think we’ve gotten better in general at replicating the dev, and test, and production environments… Docker has been a huge benefit here… But it’s still not perfect, and those are really hard to debug, because there are invisible things that you might never realize that you were even setting in the first place.

Kemal Akkoyun: I think I’m going to disagree with Bartek, at least partially, about testing this stuff… So here comes my first unpopular opinion. So even though testing and doing end-to-end tests or synthetic benchmarks on your development machine or on your CI, your development environment is not the same with your production environment. I am a big fan of collecting these metrics in production, or just profiling the actual workload in the production. Because the CPU, the machines, the network stack that actually this software runs on is totally different than what you have in the CI or in your local machine. So if you can do this on a production environment, that data is also the actually holy grail, right?

On top of that, one other thing is with the prom bench, or with the Prometheus project, we actually know what is the hot path, so that’s why we have those benchmarking tools. We know what to optimize, and we know beforehand what shouldn’t go out of range, so that we have that in CI. So it’s really helpful.

But most of the other projects, you don’t have this information, and you actually don’t know what to optimize. So there’s a lot of unknown unknowns. For those types of things, actually doing this on production, with certain metrics, or like certain facilities to collect this data is more valuable to me.

Mauricio Salatino: Or on Twitter as well. But yes, there is a step-by-step tutorial where you can reproduce in your own Kubernetes clusters, even running locally [unintelligible 00:26:22.18] the entire demo experience… Which basically goes from creating a new development environment, creating, building and deploying a function to that development environment, and finally, moving that new function that we created into a production environment using Argo CD.

I think that we tried to create a demo that people can run, because we see that maybe when you look at the presentation it looks nice, but when you actually try these tools, you learn a lot, and then you map to your scenarios, to your day-to-day job, all these tools that we are showing.

Jon Calhoun: It’s kind of nice that they took their time with it, at least a little bit, because when Go was first released… You have to imagine - at least in my mind - it’s young enough that they didn’t have enough customers using Go in high enough production environments that they really had enough data to decide what needs to be done, versus what necessarily doesn’t… At least that would be my perspective as an outsider. Maybe I’m wrong.

Kenneth Auchenberg: Yeah. And I think you’re spot on in that regard; like you said, opportunity in the sense that we will kind of see new services emerge from cloud infrastructure that solves these needs. It’s kind of a moment in time where we are kind of pulling back, in a sense that we are actually reverting back to a more self-host model, or we are reverting to a model that instead of software we buys as infrastructure, what we buy is hosted by default. We have a kind of different deployment model where you bring the data plane of your offering behind your firewall, but you have a centralized control plane that that’s kind of like what you purchase.

I think there are some really interesting questions, and I think there’s this company called Replicated, that is enabling any kind of SaaS company to provide a self-hosted version by wrapping up your service in a Docker image, and then you can sell it. So you can kind of sell GitHub Enterprise, and then what a product in Enterprise Edition.

I think there’s some interesting things that we haven’t figured out… And also, you can kind of tie it back to the developer experience and internal developer productivity is that we spent the first decade moving our servers and our production environments to the cloud. What’s happening now with internal developer productivity is that we are now moving our local development boxes to a cloud-hosted development box. And now we’re kind of moving some of production services from the central cloud to the edge. So you can kind of see the dev box kind of tailing a bit behind, also because the local development environments become so complicated or complex, they have so many dependencies that it’s no longer feasible to run them locally anymore. So it’s just an interesting progression, and I’m wondering if this is a point in time where we have to stop and really rewire how we do in a lot of these things.