Johnny Boursiquot: Seriously, seriously. I think we all have a few things we could complain about. But thank you so much, Carlana, for joining us yet again for another episode of “What’s New in Go one dot whatever.” At this point, it might all be one dot something, because I don’t see any two dot zero on the horizon any anytime soon. So thank you for joining, and Ian, thank you for co-hosting, as usual… Yeah, until we bring you back for the next one.

Natalie Pistunovich: Yeah, we’ve been zooming for two years now… Once the meetup is gonna go back to in-person, it’s just gonna be weird, at some point. Just as it was weird transitioning to Zoom, it’s gonna be weird transitioning, but we’ll manage.

So Ole, you are a backend engineer since the ‘90s. That’s your fun catchphrase. And you’ve been working in different companies that are big and small, and you’ve had lots of projects that you saw fail and succeed. And you love being part of the Go community, and you’re working for Ardan Labs… And hey, you’re my co-organizer at my Go user group… Since we were just trying to remember, as we were preparing for this call - we’ve said, what was it, 2017, 2018?

Erik St. Martin: So actually now’s probably a good time for a sponsored break, and then we’ll jump into those. I’ve got a really cool one that I wanna talk about, too. Our second sponsor for today is Ardan Labs, with their Ultimate Go Series.

Matt Holt: Yeah, good question. I think I would describe v2 as predominantly unrelated to the funding/sustainability stuff. v2 was a technical goal. And when it became clear we needed to sustain Caddy with some funding, we tried a couple things, like a sponsorship package, and what we called an engineering package, where you could gain access to Caddy expert engineering for your company… And with v1 that didn’t work out too well, because people largely saw Caddy v1 as a toy, which isn’t super-wrong… I mean, you could totally use it in production. It was built for production use, but maybe not like for enterprise use, and it didn’t have a lot of advanced features, so I think we missed out on that market.

Then we experimented with shipping custom binaries from our website, so I built this whole thing that generated a custom build of Caddy for you, click-click-click, download… And those binaries that you downloaded were licensed for non-commercial use, unless you paid, basically.

[36:26] So if you’re gonna use our build server, and take advantage of using Caddy in your company, we wanted to see if we can make that sustainable, since we knew a lot of companies were automating downloads from our website… And I don’t think it was that confusing, because a lot of software packages do stuff like this, where they’ll take an open source project and then either brand or license something else on top of it. You see this all the time… Chrome, for example, is a rebranding of Chromium. And there’s a ton of examples; the Java world has something like this, I think even VS Code…

Anyways, so Caddy has always been open source. It’s always been Apache-licensed, and it has never, ever changed, and it’s not going to. We experiment with shipping binaries from our website that were licensed differently for non-commercial use. That was actually more sustainable, probably because it kind of forced a lot of companies into paying for it… And that helped quite a bit, but it was very confusing to people.

So at some point, Bill Kennedy reached out to me, from Ardan Labs, and talked about kind of a partnership, some way to make Caddy more positively sustainable, commercially-viable. So Ardan funded the development for Caddy 2, for the most part, that first ten or so months. So I was able to drop the commercial licensing binaries and just revert everything back to simple open source. So that’s kind of where it’s at now; it’s purely an open source project, with no commercial licensing whatsoever. It’s sustained purely on sponsorships.

Carlana Johnson: Yeah, thank you for having me back again. It’s always fun to come on and talk about what’s new in Go.

Erik St. Martin: So this is funny, this is kind of sidelining here… But somebody made a comment about needing a barbecue Gopher. They were like, “We should totally see whether the Ardan guys will create one for us.” And there, apparently, already exists one. There is already a gopher standing at the grill, or - I forget what it looks like now.

Adam Stacoviak: It seems about the decentralization of it versus the centralization of it, because as soon as you cross that, you’ve now broken your cardinal rule, which is the motivation for the whole thing.

Bill Kennedy: And that’s for anybody, because again, everybody’s an educator at the end of the day, whether you’re doing in front of people or you’re just doing it at work. You’re constantly educating.

[45:46] I think one of the big jobs that are missing in the industry, and we can’t have it because there’s no way to measure it, is that every company should have the tech lead who’s a floater. This is what I want; if I ever sold Ardan, I wanna work at different companies for like a month to three months. I have no responsibility for product, but I would – let’s say I went to Grafana, I sat down with you and your team, and every day I pair-programmed with somebody different, for a couple of days, to help them learn how to refactor, look at the code, think about it, and make everybody a better - at least based on my design philosophies and guidelines… But try to make everybody better on that team, and get everybody rowing the boat in the same direction. And then move on to the next team, without having any responsibility for it. But the problem with that role is after you do that for a year, you don’t have a single commit, right?

Frederic Branczyk: [43:41] But the characteristics of paying for cardinality are dramatically different. In Prometheus we want to keep series of data alive for as long as possible, because that improves compression, and that’s ultimately one of the pieces that make Prometheus as efficient as it is. Again, that’s why I keep going back to - this is a good design for Prometheus, because it allows Prometheus to exploit several pieces of that equation to be able to serve things like the super-low latency queries like Prometheus does.

In ArcticDB we’re not paying per series, we’re basically paying per row that we’re inserting. And the point is we’re kind of bringing the cost of inserting a row down so much that we don’t care anymore how many columns we have in that row. Basically, our cost is at the row level, as opposed to the cardinality level.

Constance Caramanolis: He was the presenter for this game show a while back; there’s a new presenter now, his first name is Drew Carey or Drew Carey Flint to give people more hints if they want to Google it afterwards.

Ed Welch: Actually, it’s been about three years that I’ve been working on Loki now, so I’m kind of trying to think of how I’ve evolved describing it over time… This may dovetail a little bit into what Jon was saying, but… Loki I tend to describe now as the time series database for string. So I guess before I talk about Loki, the part about the metrics versus logs that I think is interesting is the information that they can contain and store - there’s an associated cost with that.

One of the things when I talk about Loki as the time series database for strings is… When we talked about that context you can put in a log line - you can put a lot of information in the log line, right? Within Loki we log for every query, the results of that query and things like throughput and the lines process per second. And the query, and the org ID, or the tenant ID, and all this information… And it tends to be extremely high cardinality. So where metrics systems - it gets a bit more costly when you’re trying to store huge amounts of cardinality. If I had a metric for every one of those things I’ve just said that had a tenant dimension on it, it would be a lot.

The other side of that coin though is that’s a long string, and to process that in a system requires parsing it, and doing math on it that’s more expensive, it’s more data stored on a disk. So to me, the two play very nicely together in the sense of metrics make a really nice roundoup or sort of overall approximation of high volumes of that kind of data. So you can look at metrics, because it’s a float, over much longer periods of time than it’s gonna take to parse a 80-byte or 200-byte string.

Where those become a bit interesting is – so with Loki we have a thing for recording rules where I can generate a metric from those log lines as they come in. So what we’ll tend to do is generate roll-ups. So we’ll generate a 99% quintile on the query times by tenant. So now maybe I have cardinality of thousands of tenants, but the 99th percentile is taking all of those individual values and rolling them up into one metric.

So the systems (I think) complement each other really well when you set them up. You can have very high resolution, very high cardinality data, it’s just gonna be a lot more expensive to try to query it over time and put those in your logs, and then you can roll that stuff up. Or create metrics, just use the metrics you have in your system for higher volume requests, being able to keep an eye on a system that’s doing thousands of requests per second a lot easier, showing that data over time a lot easier… So I think they play nicely with each other. I think maybe both extremes – I guess I could argue that you could monitor an app only with… Well, you can make that argument either direction.

[47:58] To be honest, once your apps start getting big enough, you probably want specialized tooling for those types of aspects, metrics, logs… And traces, for big distributed systems, being able to view how your information propagates through - a lot easier in a tracing system than it tends to be in searching for a trace ID amongst all your logs, and looking at how each of those logs were formatted differently, and the timestamps between them, and trying to kind of reconstruct that same view.

Wesley Beary: Well, I mean, first off, we do have Cobra and Bubble Tea in the app. We are still using those. It was more just there were some, I don’t know, sharp edges, rough edges, whatever, areas where what those were providing to us didn’t quite cut it for what we were trying to accomplish, or we weren’t quite satisfied with the way that they solved for it… I think we’ll get into it more deeply. One of the things we ended up doing relating to some of the testing stuff we were talking about was we now use a lot of golden file stuff. So we basically say “Here are the different things that we expect to happen in the CLI interaction… Like, wait for this prompt to appear. And once it does, then type in these characters and press Enter.” And then at the end of it, we can say “Okay, now take all of the stuff that happened in the display space and print it out to a file. And when I run this in the future, I want you to do that again, and I want you to compare against that file, and I want to know if there’s any discrepancies. Because if there are, we want to make sure those were intentional, we want to update the files if we need to”, all that kind of thing. That Bubble Tea has that is great, but at least last I saw, it was just in an experimental library that Charm ships. It’s not really part of the core Bubble Tea. We had to go hunting a little bit to even find it. And when we did find it, we found that there were some gotchas related to it that made it difficult to use. In particular, there are race conditions basically that come along with it, because of the nature of when you are doing things that cause the UX to render more things, versus when you actually do the check to see what rendered out.

[00:11:58.12] A concrete example is we have some places where in the UX it will say “I’m fetching resources from the API.” It has a little spinner; it’s a pretty common element that you’ve probably seen in a lot of CLIs. The problem was that from the way that it did its golden file tests in the library from Charm, if that spins just until it finishes and then disappears, depending on how long it takes to disappear on different test runs, it might or might not appear in the golden file output. And so now you have this issue of either we need to, I don’t know, mock it out or something, so that we can more consistently know how long it’s going to take, or… You know, it suddenly becomes much more complicated, and you need to remember that and deal with that every time you’re dealing with the test. And we were not keen on doing that. That got to be a headache pretty quickly.

So we actually like are still doing something that’s very similar. The core of it is the same thing that they were doing, but we’ve changed it so that we are making sure that whenever we’re rendering something to the screen, we’re getting some kind of output related to it into our golden files, so that again, it’s consistent from run to run. There’s not those timing inconsistencies. Even if things appear and disappear every time something has changed, it will appear on the screen.

And then because we’re already in the mix of doing all of that, it gave us more capabilities that more recently we added the ability to redact certain things from the golden files, because in some cases it’s like “Oh, well, the name that gets placed here is – we randomly generate a value there, so there’s not always the same thing.” But in terms of the golden file, we just want a name value to appear there. We don’t really care what the name value is, and we don’t want that to like write a new golden file, or whatever.

So since we were already kind of in the guts of things and we already had a pipeline that was touching the golden files, it was pretty easy to make those changes. So it wasn’t that we were like totally starting from scratch, and I don’t think, or at least I hope I didn’t commit the cardinal sin of I came to this language from a different language and I think I know better… I did try pretty hard to see what was there, and try to take from and use it… And then, like I said, it was just like, a lot of it got us pretty far along the way, but not quite all the way there. And so what we’ve done in a lot of cases is just take it and build a little bit more or a little bit different on top of it to just get us over those little bumps that we ran into along the way.

Drew DeVault: Well, Linux is frozen in a sense, which is that the cardinal rule of Linux is “We never break user space.” And so Linux is infamously unstable in terms of like if you compiled a binary five years ago, you can’t run it today. But that’s entirely because the userspace is unstable. And glibc does symbol breakage all the time.

[00:41:56.12] If you were to statically link something where it just relies on the Linux syscall ABI, or on the SYSFS or PROCFS layouts, that kind of thing; if it’s just talking to the kernel, something that you wrote 20 years ago for Linux in this manner would still run today, because Linux has made that guarantee.

David Flanagan: [58:04] Yeah. Really sound advice there, making sure that you’re taking small steps, and not doing any big migrations. Those never pan out, right? We know this now, and I think this is why we’re seeing new architectures emerge. Microservices and event-driven architectures really give you that flexibility… Or ports and adapters, whatever ecosystem you come from; they name it all a little bit different. But these allow you to be more flexible with these decisions as well. When you start having very loosely-coupled systems, or event-driven bus systems… I mean, you can even replace the bus; you stick a proxy in front and you write that to two buses. And then those buses have different subscribers, and you can start to replace those subscribers, or even duplicate them in some way… As long as they all have their own data store and you’re not making that cardinal sin of writing everything back to a single data store, you do have that flexibility to be very agile in the way that you try out these systems and go “Alright, we’re actually gonna run M3, Thanos and Cortex for a month, and we’re gonna see which one gives us the most operational pain, and then drop the other two. It’s very easy with that architecture to build, to do that.

We’re also seeing a split or a divide, I think… There’s a lot of companies that are just going all-in on a particular cloud and using every managed service in the world… And then there’s the other side, which is the Kubernetes ecosystem, and doing everything agnostic. And I’m curious to see how that pans out in a few years as well. I mean, the cost of all these managed services on cloud providers is extortion. I personally could never justify it. But maybe I’ve just not been lucky enough to work for a company that’s got the budget to do that. Maybe that would be the simpler life, I’m not sure.

Bill Kennedy: [01:14:09.06] What I wanna do, Kris, when my time with Ardan is up, I would love to be hired as a coach, and every week work with a different team at some company, to improve the design philosophies and the engineering and what they’re doing. Not sit down and write the code, but to be pairing with them and teaching them.

I think this role is incredibly critical moving forward, but there’s a problem with this role. At the end of the year, when your performance review comes up, you’ve got nothing to show for it.

Carmen Andoh: Sure. So one of the things that we have a problem with in terms of strategy is we have a finite number of engineers on the Go team… And I always am hoping to tap into them to write blog posts about modules, or update documentation… I know Effective Go needs to be rewritten with modules in mind… There’s just a lot with regards to educating.

So we launched learn.go.dev, which was kind of a beta site curation that takes in and just helps organize some of the information… But at this point it’s pretty static, and it could be akin to the Wiki on GitHub for Go. So we realized pretty soon on, either Google has to hire a ton, dozens and dozens of content creators and tech writers to scale this for the next 10 million (or however many million), or we need to think about an open source strategy for this.

The other thing - when I researched the learning landscape of Go, I recognized that the current material that is existing that is produced by the Go team is static, and it’s text-heavy. We do have the Playground, and we do have the Go Tour, but that’s the extent in terms of being able to provide modalities of material.

So we rely largely on third-party people to either do training, so Ardan Labs or Gopher Guides, or we have people who are enterprising enough to create Gophercises, such as yourself, Jon.

[19:48] So the missing link, I feel, is that as both Paul and Marcos have alluded to, there was a big gap in what I call the interactivity and the ability to make documentation not stale, and have it come alive, so that people can try out what’s being meant by it… And it also is a change in – so we have a change in people for Go coming to the language; lots and lots of people. But we also have people that are a different kind of persona, which is mostly enterprise. And when you dig deeper into the persona of an enterprise or a person who uses Go professionally, they’re not similar to the early adopter.

The early adopter is curious, and they probably want to try out all the new languages, whereas the professional programmer using it at work - they just want to get the job done. They wanna see what it looks like, and they’re very practical and pragmatic, and they don’t wanna wax eloquent about the philosophy a bit, or the context a bit… They just wanna get it done.

So we’ve heard that we can call it as meat and potatoes, or beans and rice, but really to the heart of it… And I think that play-with-go.dev is that gap that fills for everything that’s happening with Go right now.

So a long-winded way, Jon, of saying “Yes, we’re looking at ways, with limited funds, to be able to scale out content that would be useful to a lot of people, but also get the ecosystem involved, and empower them…” Because some of the most useful materials that I’ve found is when people wanna share and teach. Everyone wants to feel useful. So I think this is a convergence of both strategy and product that is trying to meet all of those things at once… And I have a lot of hope and confidence that we’ll achieve that.

Brett Cannon: [55:46] And I would totally understand two if they were not planning for a three. It seemed they were basically ending it at two. Then it would have gone like “Okay, we’ve got one. Now we’ve gotta find motivation for two, so now we’re done.” And “Oh, look, we’ve got a third. Well, let’s maybe tie it back to his wife”, versus – there was obviously gonna be a connection from two to three, so it was like… Yeah, it went from “wife, to me, to wife”, instead of some – and maybe they just couldn’t think of a better way to do the flying through… But that was my problem with two - that part of the story felt kind of out of place. It’s like “I’m now the one that’s angry and mad, and now I’m just gonna go kill everyone, because they want me to come back here and be in this world again, and I don’t want to be.”

But then if that’s the driving force, he’s way overdoing this… And then it gets to the end, where it seems like the driving force is “I don’t wanna be part of this anymore, but I’m being forced to.” And then it gets to the end, and it’s like “Oh, well now I’m just gonna make war life really hell, and just totally break one of the cardinal rules of this world, and really throw everything out the window”, versus something else that would lead to him like, I don’t know, watching a guard outside slip up, doing something that would allow him to more walk away, versus his “Lose my patience, and I’m gonna totally overblow it.” And then go into three, where it’s suddenly like “Oh, I overblew it. Now I’ve gotta try to fix the whole thing, so I can be the one who remembers my wife etc.”, which also felt a little stretched… But once again, that’s because of two, because he did this thing knowing full well that he would end up in that situation.

Bill Kennedy: Oh, rockstar – blah! Anyway, I found my way to Go back in 2013. I own a company named Ardan Labs with my best friend, brother, and business partner, Ed Gonzales down in Miami. In 2013 we had to switch off of Windows and move into a Linux environment. I was coding a lot of C# at that time, and that’s when we found Go, and I started writing some blogs, and Erik and Brian, the other hosts on the show, invited me to join them in writing the “Go in Action” book, and that led to talking at the very first GopherCon.

Then between the blog and the book, I started developing training material, and probably started training at the end of 2015, and now it’s pretty much a full-time job, day in and day out, and I still make it a point to write one blog post every month on the GoingGo.net blog, which we’ve now moved over to Ardan Labs, but everything gets redirected. So I’m really just a software developer who’s been writing software professionally since 1991, and I’ve been working in Go for almost five years.

Erik St. Martin: Well, huge thank you to everybody on the show, thanks to all the listeners listening right now. Huge shoutout to our sponsors, StackImpact and Ardan Labs. If you haven’t checked them out, please do. We will put links in the show notes. Definitely share this show with friends and colleagues. An easy way to subscribe is to go to GoTime.FM. We will have a weekly email coming out shortly, so go ahead and get signed up to that. We are @GoTimeFM on Twitter, we have a GoTimeFM channel in the Gophers Slack. If you wanna be on the show or have suggestions for topics or guests for the show, github.com/GoTimeFM/ping. With that, goodbye everybody! We’ll see you next week. Mat Ryer will be joining us for next week’s show.

Erik St. Martin: It’s hard, because there’s no cardinal rule, right? It’s just like the other topics we’ve had here today… It’s about looking at your programming, determining how important these things are to you. Are they a necessity to operate and maintain this application, or are they really just fluff that makes you feel comfortable that you could open a log and see that data if you wanted to. The number of times I’ve seen applications that are heavy on the logging side and nobody ever looks at the log - that’s probably the better majority of the applications I’ve worked on. Most of the times, you’re looking at your metrics dashboards and things like that. It’s too hard to go grepping through logs, especially when you don’t have a central logging place for these things.

At high scale it also becomes its own problem, because that’s another system that can fail, when you’re doing your distributed logging out to one place, and you have to make the decisions, how worthwhile is that additional complexity to you, and the additional storage to store all these logs, or are you really trying to get a rough state of your application.

Matthew Boyle: Yeah, so the next sort of rung on the ladder up is metrics. We’ve touched on this a little bit throughout our conversation already, but I think these for me are probably the most valuable thing you can add to your application. And these are where you can be a little bit more liberal than logs, I think. I think you probably can and should expect your metrics to be sampled, but we tend to use metrics more for looking for patterns, rather than for investigating individual use cases and issues. So I’m a big fan of Prometheus. Hand heart, being truthful, it’s probably because I haven’t explored so many other options. Prometheus has always been present most of the way through my career, so I’ve tended to use it, and it works really, really well, so I’ve never really explored too many other options. But being able to have metrics like “Oh, user signed up. Transaction failure.” And then you can add cardinality to those things. So you can say like “Transaction failure. Card type: Amex.” And then you can build beautiful dashboards using Grafana, and make alerts based on these things.

[00:41:59.16] So you can say things like “Oh, we’ve just seen a spike in transaction failures for American Express.” That’s a really, really useful signal from your system. And then you can go and jump into other systems; maybe in this instance it is your logs. Or maybe you jump onto American Express’s page, and you see that they’re having an outage too, and it all makes sense, and you can send out customer communications, or put a something in place on your website to make people aware that those issues are ongoing, and you’re working on it.

But having metrics of patterns that you can see going on in your system, success, failures, errors, with various levels of cardinality is really, really powerful. And there’s a massive asterisk here that I’ll add quickly though, which is this only kind of works if you have a reasonable amount of traffic. So you do need to have a certain amount of scale before metrics are useful. Before that, logs are probably fine. But once you start to see a few hundred, a few thousand requests a day going through your system, these things can be incredibly useful to you.

Bartlomiej Płotka: Yeah, so I think that the mistake is really around – we didn’t have time to review exactly properly everything, and that change was pretty big. And as maintainers of Client library, I think we just get that from Bjorn’s hands, so we are kind of pretty new. And essentially, usually, it’s really about memory runtime metrics that Golang allows you to achieve. So usually what you could do - you could create this runtime.memstats structure, and you can just program runtime.readmemstats and provide the structure, and that will fill you the structure with statistics. And this is how we used to kind of get those metrics, and give you kind of a very easy one-line statement that you provide in the beginning of your application, that continuously exposes information, up to date, so Prometheus can scrape it and have up-to-date heap information; how many allocations you did on the heap, what’s your GC latency, and so on.

Now, what happened in Go 1.17 (I think) was that runtime/metrics package was introduced, that kind of gave even more information about internal Go runtime, and we wanted to switch, essentially, to this new format… So totally go away from this memstats logic.

However, what’s happening with especially metrics is that we all care about cardinality. We make sure the metric is stable, because this is what allows us to aggregate over time, and also over different series, and kind of compare to each other. This is the key concept behind metrics. And also, the cost of the metrics system really depends on the uniqueness of those series. So yeah, we just introduce a new unique series, so new names… And this is kind of what we could avoid, what we are trying to fix right now, is to add essentially some optionality, some configuration pieces. Especially because this library is so used, so popular, we need to make sure it by default doesn’t increase your metric count twice.

So this is generally kind of what we try to achieve. So we have a community who wants to have this new runtime, amazing, granular metrics… So we kind of approve it and merge it, and then suddenly, the rest of the world was like, “Oh, I need to pay twice more. I don’t understand really.” So this is really the trade-offs we have to make for such a popular module. It’s much more stressful and much more work than if you could just break the API on every release.

Evan Kaplan: Yeah. So the big transition over the last few years has really been building a cloud-native platform. So we had had a managed service cloud - we still do - that’s pretty successful, and has large customers, like Salesforce, and Google, and that stuff on it. But Paul particularly advocated starting even longer of that the right model for the future is something like an Amazon native database, or an Azure native database in which people are paying for reads/writes/queries, they pay for what they use; if developers find it useful, they build interesting things on it, and they run it in the cloud… And then they only pay for what they use. And his view was that’s the best way to monetize open source. There will always be people in the open source world, of course, who won’t wanna use that, but that’s the best way to monetize if you’re a company. And I think we’re seeing that; certainly, the predecessor for us was Mongo Atlas doing that really well, which now represents I think more than 50% of the revenue, and growing faster than that. And that is a core feature of our business, and that’s going phenomenally well.

And then IOx… The work on IOx was - you know, after 7 years of working on time series storage engines, realizing that there are always gonna be issues around cardinality, and performance, and query performance, and things like that. You could do this way better. So we started the IOx project, and its relationship with Apache Arrow, and the Parquet file format… I thought it was a super-clever way to go about it. We’ve invested in that now for well over two years, and we’re super-excited. That will be an accelerant to the cloud. But we don’t get to just be cloud, because a lot of our business and a growing portion of our business is in IoT. Most IoT architectures have an edge in cloud, sort of a two-tier or even a three-tier architecture of local storage, and analytics, and so building that edge capability and the on-prem capability continues to be in the raw open source… The fact that we continue to produce new open source that people will use at the edge, or in the enterprise, and that sort of stuff - all these things come together into what we view as our competitive advantage, a comprehensive platform. Cloud-native, at the edge, in the enterprise, being able to use this all together, with a common API, and that sort of stuff.

[40:26] So that’s our view of where our tech goes. In terms of where the market goes - to me, that’s what’s super-exciting. Listen - obviously, sensors are getting cheaper and cheaper, we’re sensorifying all the physical world, in every way, shape and form, whether it’s the 12 sensors I have on my body from this Apple Watch, to the Whoop band, to the Oura Ring, to whatever. They’re not slowing down, whether it’s your car, or your home, or your new Tesla Powerwall - whatever you’re doing, we’re seeing the physical world being instrumented in really meaningful ways, and building that sort of in the broader… I think we are well-positioned for that, because the lingua franca of IoT is time series. It’s pressure, volume, temperature, humidity, light, whatever it is… What happened, what happened, what happened, what happened… And that kind of data, super-fast processing in real time… So we’re super-excited about that, and having the vehicles to serve it. So that’s what’s driving our product vision, things like that. And to continue the foundation in open source.

Gerhard Lazu: I really like the way you think about this, I have to say… There is something pragmatic about, and something like - you can have this within five minutes… But I also am very wary, because I’ve been following Charity Majors’ Honeycomb and those perspectives for many years, and my understanding is that the only thing you should care about is events. And if you have a data store that understands arbitrarily-wide events, something that can query them just in time, at scale, then you don’t have to trade off the cardinality constraints that metrics have, versus the volume of logs that is just too much, and the indexing, and how basically that happens behind the scenes. So the implementation that limits you to how you use those logs.

So I think that perspective is very interesting, and I will definitely follow up on that some more in the context of this show, of Ship It. But I’m also aware of where we are today - and when I say “we”, I mean the Changelog app - what we have already set up, and that ideal, which is that everything is an event. I think whether we want to or not, I can see how we are going on the journey, maybe some are more frustrated, others are more enlightened, but I can see how events potentially have the answer to all these things. But right now, the reality is that we still have to make this choice between metrics or logs. Traces as well. They’re like separate components. And I think that Grafana Cloud is doing a pretty good job with Cortex, which is a Prometheus that scales, basically, Loki, which is for indexing logs, and it’s great to derive insights out of that, and Tempo, which I haven’t used yet, which is for traces. But these are the three components in the Grafana Cloud that serve these three different functions.

I think it’s very interesting to get to that tool which unifies them all, and Grafana Cloud could be it, but there are others as well. Now, I’m not going to go through all the names, because that’s boring, but what is interesting is that we seem to be going in the same direction. And we may argue between ourselves whether the pillars of observability are a thing, or are just a big joke - different perspectives - but I think ultimately what really matters is being able to understand what is happening in your application, or what is happening with your website, or your service, or whatever. Unknown unknowns. I’m not going to open that can of worms… But the point being is “Do you understand what is happening?” It may be imperfect, it may be limited, but do you have at least an idea of where to look, where the problems are?

[19:59] And I do know that PromEx helped us or helped you with the N+1 queries. It was very obvious “Hey, we have a problem in Ecto, and this is what that problem looks like, and this is how we fix it. And yes, we fixed it. Does Erlang 24 improve things to Erlang 23, and in what way?” And we can answer those questions as well.

So I think that monitoring is not going anywhere, and I think everybody respects it for what it is… But we also are aware that there are better ways, and we should improve this. So with that in mind, where do you see PromEx going? What are the hopes and the goals for the project?

Bharat Sandu: [unintelligible 00:42:10.03] AI is a fascinating space, because I think it’s not just in the incubation space now. It’s getting more and more mainstream, in many other technologies… And even when you compare other technologies, it’s gaining mainstream adoption faster, and it’s really happening, I just believe, because they’re what we call digital-native companies especially, that have been building the foundation, they’ve been built in the cloud and have used AI and ML as a way to really separate themselves.

So the things that I’m excited about, to be more precise - we see companies that you would not imagine using machine learning, and doing it not just for descriptive analysis, but actually running ML pipelines, building thousands of models. [unintelligible 00:42:51.25] Australia. They make windmills, and now they’re building a machine learning model per windmill. Not just one generic machine learning model.

[unintelligible 00:43:07.10] is another commonly-known name, but it’s not a born-in-the-cloud company; it’s a very old company, that has really adopted technology to really help them out-innovate their competition… And they’re also, by the way, using Azure machine learning in this case to do the email phishing attacks, to detect them; because they have a huge employee base. And they’re using very sophisticated ML pipelines to not only train the model once, but to actually look at all the emails that come in, and to kind of then stack rank them in the risk factor, and then to run more sophisticated machine learning models to really reduce the amount of things that are happening.

[unintelligible 00:43:43.13] is another one. They use machine learning to do fraud detection activities. And when they wanna say “Hey, Chris, you’re a loyalty member, but you did something bad”, they need to be very sure before they ask you that question. And they’re using, again, machine learning, and the responsible machine learning capabilities to make sure the models they build have super-high confidence. They’re using model interpretability, and all that thing.

[44:08] So what really gets me excited is a) we have many more mainstream companies using AI not just in the lab, but literally doing thousands of models in production, and really revolutionizing the businesses now. So it’s really not in the hands of, let’s say, Google, Microsoft, Apple only. It’s really what I’ll call – I’m sure Lockheed Martin also does a bunch of machine learning, although I don’t work with them anymore… A little bit, right? But these mainstream companies are benefitting from this a lot. That gets me excited every single day.

The second thing is the space is innovating at a super-fast pace. We have GANs, we have all these new techniques coming in. And it almost seems like what was cool two years ago is quite passé now. We all use scikit-learn, but nobody really talks about it that much in this sense, right? Now we’re all about putting deep learning models in production at high scale… And the space of getting new AI research into the world is just super-fascinating.

What keeps me excited - I think everybody in this field is - AI should never be in the hands of a few, and increasing it’s not. Now, as it does get to reach more people, you have to do it in a very responsible fashion, and that word cannot be used loosely… In the sense that yeah, anybody can build a model [unintelligible 00:45:26.01] give them the ability to look under the hood, to understand does that data have any issues, were there any biases built into the data, or they have high cardinality issues…

You know, just because a model is 99% accurate, if it’s classifying the wrong thing and it’s missing 9 of the 10 cancer detection, it’s a pretty damn bad model. But being able to know – bringing the sophistication of being a data scientist to people who are not data scientists is not easy. We should never trivialize using API [unintelligible 00:45:57.15] you’re happy and you go and deploy around the world…

Now the fun part is “Okay, now that we simplified some aspects of machine learning, how do we make sure it’s applied in a way that people who are applying it fully understand the implications?” And that’s a super-exciting space for me, and what the industry is doing, what Microsoft is doing… Not just simplifying it, but making it practical at the end of the day, so more people can benefit from it.

And it’s a never-ending cycle - new research comes in, taking to market, and doing it in a way that most people can benefit from it; controlling the hype around this topic, but really driving the benefits for our customers.

So it’s a long-winded answer, but it is kind of making sure AI is applicable to large sets of customers, but in a practical fashion, not in a buzz-worthy fashion. Not just saying “One click, and boom, you have a model.”

And the second one is just the pace of innovation and new techniques coming in, and the opportunity it offers to customers, even who don’t have data, to do machine learning. Things like reinforcement learning techniques, when applied correctly, can solve some of these issues. So it’s really making this not a special topic, but a really widely-useful topic is what excites me and keeps me up.

Simon Eskildsen: Yeah. I think maybe two that come to mind through history here are - one would be probably the biggest project, or the biggest project that I did the earliest was our podding project. Essentially, we’ve done sharding, so essentially taking our database and then splitting it into multiple. That was done around the same time I joined, and not by me, but by other teams… But we wanted to extend that even further, so that we could have essentially these groups of shops that live together and are isolated together, and those shops should be able to run in multiple data centers… Because before that, we would have one data center, all of the shops would be active in that data center. If something went horribly wrong, we would fail all of them over to the other data center. But we wanted to isolate these shops in a way where we could run them out of multiple data centers at the same time. That was a lot of engineering effort to make sure that there’s nothing relying on the fact that everything is in one data center at the time. So that was one of the biggest projects that I did a few years ago, with a really good team.

This year, one of the things that I’m most excited about that we worked on was that a lot of my focus has been around capacity planning and resiliency, so essentially finding out that when a system becomes slow, how do you make sure that it doesn’t jam up the entire system? It’s a lot worse when a database is slow than if it’s down, because it can clog up the systems and cause queues in all these different places, and cause much more cascading failures.

And one of the things that we’ve had great success with here is this technique called load shedding. The idea behind load shedding is essentially that when a system is overloaded or close to be overloaded, you wanna start prioritizing what type of traffic that you okay, and that you send through to the system. So if we have a store that is having a lot of malicious traffic or some kind of sophisticated DDOS, we wanna make sure that we start to drop that traffic before we start to drop the traffic at other merchants, to protect the platform.

We’ve done a lot of work in that, and we’ve done a lot of work at that at the edge, so that the load balancer can prioritize traffic, to make sure that our merchants have as much uptime as possible… But we wanted to go even further and start providing that at the database level.

One of the things that to me is very disappointing about the database realm today is that a lot of companies are SaaS companies; they’re multi-tenant companies, and they run all these tenancies on one or a few databases. But one of these tenants might have a disproportionate impact on that database. They might have an API client that is doing a lot more requests than anyone else. Or they might have a customer that has a million orders because of some peculiarity in the way that they work. So you have all of this cardinality and all of this uniqueness to the merchants, or your customer… And that’s not just a Shopify problem, that’s a SaaS problem. Because what you get with SaaS is that you get these cost efficiencies of running on the same infrastructure, but I don’t think the infrastructure has really caught up to that.

In a database today - you don’t create a schema for every database. MySQL would scream at you if you tried to do that when you have enough tenants, because it’s just not made for that. So it doesn’t really give you any primitives to be able to do that. And by default, the way you design your database is really not set up for multi-tenancy at all. So to go back maybe to this example of a single tenant overwhelming the system, MySQL or Postgres or any – there’s no database that has a good mechanism for prioritizing traffic between these merchants.

[15:59] So what we have been looking at - we’ve found out that in the MySQL protocol you can send an arbitrary string back with a query. So we thought “What if along with the query results” - this could be a bunch of customers, a bunch of orders, whatever it might be, “what if we also sent back to the application how many resources that query took?” How many nanoseconds on the CPU? How much i/o time? How much memory was loaded into the page cache in MySQL? How expensive was this query? The kinds of things that you would see in a slow query log.

So you might think “Why is that useful that you’re gonna look at that information?” Well, imagine an API throttle that is not some arbitrary number taken out of a hat, of “You can do 100 requests per second”, but rather the API throttle was actually based on how the database is doing and how heavy the queries that your API calls are causing actually are?

Doing API throttling with something like GraphQL on an external API is incredibly difficult to do correctly, and you’re almost always gonna either underestimate the query complexity or over-estimate it. But if we build systems that have multi-tenancy and databases that have multi-tenancy built in to that caliber, where they can feed it back to API throttling, that helps a lot.

You can then also feed it to your load shedding mechanism, so you can see “Oh, this tenant is being really bad at the database, even though they’re only doing very few queries.” So I think that’s a really important thing for more databases to adopt, and we’ve been working on a patch to MySQL to expose this.

Matt Jaffee: That’s correct… Yeah, B-trees and binary trees and such are very popular structures for indexes, and if that had worked to sort of solve our original problem, we probably would have never built Pilosa… But the problems that we were having at Umbel were high cardinality segmentation of, you know, when you have millions or hundreds of millions of possible attributes across hundreds of millions of items, and you wanna drill down to very specific or very complex segments of those things.

B-tree indexing starts to fall apart, but bitmap indexing can be very good, so it’s my highly biased opinion that bitmap indexes are a highly under-used data structure.