Craig Kerstiens: Yeah, I’m sure it’s not any better, with coin mining and other things…

Kris Brandow: Yeah… I’ve been watching some Coffeezilla on YouTube recently; he’s like this investigator that goes into all of these different types of scams. And I’m just like, “It is so easy to just create a new coin, and then just extract money from people.” And I’m this is a wild, wild space. It all started because of this blockchain thing. But then again, like most of finance also - we’ve had Ponzi schemes for a very, very, very long time, so…

Adam Stacoviak: Well, I haven’t stopped thinking about what Swyx said last time we talked, which was above or below the API, which is almost the same side of the coin that we talked about last time, which is like, this the same thing.

Nishant Roy: [20:06] Completely. Like you said, if you don’t have a large enough sample size, this sort of testing is not really going to give you meaningful results; it’s just going to be a coin flip at that point whether your test passes or fails. So having those more deterministic unit tests that actually test the app behavior on one single request is a lot more important at an early stage.

Jerod Santo: [38:35] I have a couple of meta questions while we’re talking meta Muse here… Not the podcast, just about Muse and your desires there. Two phenomena that happened that I want to ask - they’re kind of on other sides of the coin. The first one is the sophomore album. So anytime you have a band - of course, pulling from the music industry - that comes out with their huge hit record, and it blows minds, and it’s amazing… And then they have this time off to regenerate their creativity, and they have money now, and they have fame… And then they have their sophomore album. And so historically, it’s been a stumbling block for a lot of bands, because what do you do? Do you give people more of what you started with? Do you redefine yourself? Did you catch a flying star, or whatever, and then you can’t do it – can you not recreate the magic, you know? And some bands fail, and other bands succeed at having that second album be great. And I’m just curious, if you have – I’m not going to ask you if you’re gonna succeed or fail, or whatever, but do you have any of that trepidation of like “Gosh, what if I can’t do it again?” Or do you even want to do it again? Or what are your thoughts on Adam Wiggins’ sophomore album?

Jerod Santo: Okay. So on the other side of the coin then, if you are an Elixirist who wants some work, maybe reach out to you as well.

Adam Stacoviak: David Heinemeier Hansson said this a while back; this is not like new news. I’m not saying that you can’t take this and coin this, by any means, but way back in 2013 there was a startup school, I believe, where DHH was talking about planning as guessing. And then they said it again, I think, in one of their books, in Rework, potentially, and then again, on their podcast in 2021, where planning is guessing. So like this is – this is kind of like, almost everybody knows, you shouldn’t plan, you should trajectory, essentially. But is this systemic? Is it like a big issue? Do you think people roadmap versus trajectory?

Adam Stacoviak: It’s not like a month ago or a year ago this thing happened, and it took a while to incubate it etc. The one thing to point out, since we’re on that note - and we talked about this very, very early - is the openness. If this was closed like OpenAI, this - Jerod, you wanted to coin this - Cambrian explosion would not have happened, right? I’m sorry I took it from you…

Angelica Hill: I mean, it’s probably two sides of the same coin, and part of me is like, I really want to continue doing it. I really enjoy doing it. But also, another part of me is like, developers already have so much they need to do in their lives. We put so much accountability on them. I feel like this is something that almost like we as product or project can, with heavy input from them, drive the boat on.

Jeff Hernandez: [31:58] It’s really important having that team understanding. At least in my previous company we had shared standards as to how things – we had pillars in everything, we had a standards committee that we were trying to… For code style, and stuff like that.

But on the other side of the coin, I’m kind of – sometimes if I see a spelling mistake, I will point that out in the PR, just because it’s… Like, if it’s already committed and I see it, I wish someone had called it out in the PR. In my PR I will fix it…

So it’s just like those little tiny things that kind of add to your personality as a reviewer. You might be known as that person that calls out your grammar mistakes, which I feel like I used to be, and I’m not as much these days… But when I see it, I will sometimes call it out.

Adam Stacoviak: This is great, because you’ve kind of got both sides of the coin, really. You’ve got feeds at large, and you can consume podcasts, and for those feeds inside of Feedbin that is a podcast feed. You could say “Hey, this is a great experience here, obviously, because I made it, but there’s also these other cool things you could check out so that you have a way to promote Airshow via Feedbin by way of saying there’s a simple or more complete and lovable experience elsewhere should you wanna take it. And there’s free sync. You’re welcome.”

Adam Stacoviak: Like, you’ve gotta have enough experience points and have enough coin to go on a certain quest. Because that would motivate me to 1) be immersed longer if I’m really enjoying myself. I’ll want to upskill to get there… And I’d want to make sure I have a bank account that can support my quest habits, so to speak. [laughter]

Jerod Santo: Didn’t you coin the term “security theater”?

Adam Stacoviak: Gotcha. But for some teams who have the fear of enterprise, from a technical perspective, not just a deal perspective, will sometimes push away, because they want to focus on product. And by no means is this an ad for WorkOS, because they are a partner of ours, and sponsor of ours, so I’m familiar with the process. That’s what gets me excited about them, and even your story, because you focused on enterprise early. I think it’s a lesson, not so much that, “Oh, hey, go buy WorkOS”, but more like, “You know what? Don’t shy away from these enterprise deals.”

[28:06] Because like you had said, you had learned so much about the process. And I bet you that the non-enterprise deals get so much easier, because you’ve gained so much confidence, and it gave you a roadmap. And I’m sure when you went and you said, “Hey, Venture Capitalist, we’re not really interested, but can you give us some feedback?” It gave you, for lack of a better term, some ammunition when talking to them, some leverage, because you already had some of the hardest parts about that enterprise chasm that Michael did coin, down. That’s a lot.

A venture capitalist will look at that and say, “Okay, you’ve already stepped to the line of the enterprise chasm, and it’s easy for you to cross.” In many ways, Scott already built the tooling, so hey, let’s just do it, kind of thing. It gave you so much confidence and awareness of the product direction and ability when it came to bringing in investors to make your company possible.

Adam Stacoviak: Yeah. A couple shows back before this show will be out - this show will be out sometime in March… So in late January, early February we released a show with Honeycomb’s CEO, Christine Yen. She’s awesome; I mean, if you haven’t listened to that show, Sid, go listen to that show. She’s super-wise, I love their journey… And they really coined the term “observability”.

[12:10] She shared a story with the conference Monitorama, when Honeycomb was kind of early days in what they would call the observability space. And the tagline for the conference was like “DevOps” or “Opsy things”. It didn’t have the term “observability” at all. And her CTO, former CEO and co-founder, Charity Majors, was giving a talk… And this is probably five or six – I forget what the timeframe is. Listen to the show; it’s in the show notes, it’s in the transcript… I’ll paraphrase most of it. But Charity was giving a talk at the conference, and it was something around observability. She started to coin the term.

So there was one talk that year at Monitorama, and no tagline change. The next year there were four or five talks with the term “observability”, and the tagline for the conference was still the same. The next year, there was like 11 talks with “observability” in it, and then the tagline for the conference changed from just “Opsy and monitoring” to also observability. So this really started to be a trend. And that’s really what Opstrace was really pioneering, was another area of observability. Did you not have observability prior to Opstrace? What was monitoring like within GitLab at that time, or before the M&A?

Björn Rabenstein: [32:48] I think there is a problem that - who knows if you use Prometheus for metrics collection? Perhaps you use something else, and then you want to instrument not just for metrics, but for all the other things. And of course, you have a lot of choices to make. And then there are a lot of efforts to unify this. OpenTelementry is definitely a huge effort of tying up all the loose ends, and it’s really hard, right?

So the first question in a practical context is you should look at what your organization you’re working in is doing. Often, they use some framework, right? They use – whatever. Let’s start with a web router. You have your favorite Go web router, or whatever. And then from there on, you can kind of inform that decision. At SoundCloud, we had – most microservices at SoundCloud were running on the JVM, and they had… This is also like somewhere in the public, people talk about it, they advocate for that, right? I think it was called JVMKit, where they had all that framework, how to write a microservice at SoundCloud. And then they put instrumentation in there.

So just merely by using this framework, which you would use anyway, you were instrumented for traces, for metrics… You’ve got some logging, some standardized logging… It’s all in there. We phrased this coin – yeah, no, we coined the phrase that at SoundCloud it was easier to get monitoring than to not get monitoring. You couldn’t avoid it, right? And that’s – for many developers, you’re already in some framework. It may be a little software framework, or the framework of your organization. And there might be something – or if there’s nothing, you might just put the pieces in there and you could look for the right thing.

I mean, rarely you have like a greenfield approach. If you do – I mean, that’s also fun, of course. You start your first Go program, and then you could just link in the Prometheus Client Golang… It’s like, they’re very minimalist programs in the documentation where you can just link it in, expose an HTTP endpoint, and do nothing else and you already get metrics for all the Go runtime stuff, for some process metrics, that’s more like an OS-level stuff… So you don’t have to do anything to already get a ton of useful metrics out of your program. And then you can go on, you can do really low-level metrics instrumentation. I also have a talk about this somewhere in my portfolio. We might link this. But there’s also a set of middlewares. I don’t know, perhaps - Bartek, you can talk about the middlewares we have, because that’s also very nice how Go does HTTP things, it plays very well with that.

Jerod Santo: Yeah, they just dropped a lot of coin on that new spaceship campus.

Jerod Santo: Early abandoner, exactly. I like that. Coin the phrase. Alright, Chris Shaver asks “Are there other languages besides C that you admire? Why?”

Chris Benson: Yeah. As you were going through that, it struck me that in addition to accounting for the unintended consequences behaviorally as you perturb data and stuff, that the flip side of that coin becomes vulnerabilities from a security standpoint, in terms of affecting the outcomes of foundation models, or any model for that matter, by doing that. And so it all just has a greater multiplier associated with the foundation, since you’re building so much on top of that.

So as we look at this in the days ahead, it will be interesting to see as well what the security implications are of protecting the end-users that are impacted by this, meaning the model will change the behavior of lots of different executions on that, where people are using it for various use cases that affect people. I would conclude by noting that in all of this that we’re talking about, it’s starting with human beings and it’s ending with human beings. And we are a little bit at the mercy of the models as we’re trying to make improvements in other ways in our lives to do things productive in the world. It’s a curious set of problems that we face going forward.

Zach Leatherman: Yeah, that’s valid criticism. I don’t really feel like I have a great story for that. I feel like the other side of the coin is that some of these projects are better about maintainers removing the funds and not really telling a good story about what the funds are being used for. There’s a lot of trust that goes into donating to an open source project, and I would say that I haven’t seen too many folks cancel their Open Collective contributions… Because I do really want folks to understand that it is an investment in the longevity of the project. So I don’t consider this Netlify arrangement to be something that will probably exist for ten years. I mean, if it did, that would be awesome, but I think if we’re all honest, I don’t think that that will be a thing that will exist in ten years.

That is kind of a fallback strategy for me at this point, and I’m open to discussing it with folks if they have concerns about that. But I do feel that my mindset is changing a little bit with this new arrangement at Netlify, and I’ll probably start to use the money in a more community-organized way moving forward, as I feel more comfortable with the arrangement moving forward. It just feels so weird to me… Like, the entire thing just feels surreal and weird. And even just like the last 3-4 weeks that I’ve been doing – because I’m on my fourth week of this open source arrangement right now. Every day still just feels strange. It was a thing that I had to sneak in after hours… Like, I went from less than ten hours a week on this project to like now I’m doing 40, and I have to physically stop myself from doing more than that, so I don’t overwork… Because it’s just like – I love it so much. [laughs] I just don’t know how to… I don’t what to say too much about it, but yeah.

Ed Welch: Actually, it’s been about three years that I’ve been working on Loki now, so I’m kind of trying to think of how I’ve evolved describing it over time… This may dovetail a little bit into what Jon was saying, but… Loki I tend to describe now as the time series database for string. So I guess before I talk about Loki, the part about the metrics versus logs that I think is interesting is the information that they can contain and store - there’s an associated cost with that.

One of the things when I talk about Loki as the time series database for strings is… When we talked about that context you can put in a log line - you can put a lot of information in the log line, right? Within Loki we log for every query, the results of that query and things like throughput and the lines process per second. And the query, and the org ID, or the tenant ID, and all this information… And it tends to be extremely high cardinality. So where metrics systems - it gets a bit more costly when you’re trying to store huge amounts of cardinality. If I had a metric for every one of those things I’ve just said that had a tenant dimension on it, it would be a lot.

The other side of that coin though is that’s a long string, and to process that in a system requires parsing it, and doing math on it that’s more expensive, it’s more data stored on a disk. So to me, the two play very nicely together in the sense of metrics make a really nice roundoup or sort of overall approximation of high volumes of that kind of data. So you can look at metrics, because it’s a float, over much longer periods of time than it’s gonna take to parse a 80-byte or 200-byte string.

Where those become a bit interesting is – so with Loki we have a thing for recording rules where I can generate a metric from those log lines as they come in. So what we’ll tend to do is generate roll-ups. So we’ll generate a 99% quintile on the query times by tenant. So now maybe I have cardinality of thousands of tenants, but the 99th percentile is taking all of those individual values and rolling them up into one metric.

So the systems (I think) complement each other really well when you set them up. You can have very high resolution, very high cardinality data, it’s just gonna be a lot more expensive to try to query it over time and put those in your logs, and then you can roll that stuff up. Or create metrics, just use the metrics you have in your system for higher volume requests, being able to keep an eye on a system that’s doing thousands of requests per second a lot easier, showing that data over time a lot easier… So I think they play nicely with each other. I think maybe both extremes – I guess I could argue that you could monitor an app only with… Well, you can make that argument either direction.

[47:58] To be honest, once your apps start getting big enough, you probably want specialized tooling for those types of aspects, metrics, logs… And traces, for big distributed systems, being able to view how your information propagates through - a lot easier in a tracing system than it tends to be in searching for a trace ID amongst all your logs, and looking at how each of those logs were formatted differently, and the timestamps between them, and trying to kind of reconstruct that same view.

Nick Nisi: I’ll just throw out – not a question really, but more of a statement, I guess… You know, there was a tweet, I think yesterday, by Ryan Florence, one of the creators of Remix, talking about how the world seems to be ready for the post-React world… And he mentions both Solid and Marko in the tweet. So - just really cool being able to talk to you about that, being able to talk to you about the potential other side of this React coin that we’ve been in for quite a while… Very exciting stuff. I’m really excited to learn more and to unlearn some of the things that I am doing in React.

Tobie Langel: And as Chris was mentioning - I mean, you’ve audited the package once. Wonderful. I mean, I don’t know how your life is around maintaining open source software, but I just get pinged all the time because there’s like a dependency that I have to update, because there’s a security vulnerability somewhere… That probably doesn’t affect my project, because I’m just using a bit of it, but I don’t know… So that’s the other side of the coin, right? We have to audit every change, and we have to audit all of these changes just in time, because we actually wanna pull those patches in, because if not, we might be risking something else.

Alexis Richardson: Ha-ha! I mean, we have things like, you know, Helm chart repositories as artifact, and companies like Amazon have marketplaces, Red Hat has Operator Hub… So in a sense, we are, little by little, getting closer to the idea that there is a systematic way to extend one piece of enterprise software with another. But really, I was talking about – you know, it’s a castle in the sky. I think the reason I discussed the enterprise app store is because in a way, it’s such a ridiculous idea.

[08:04] If you look at the presentation that I gave, which if you find GitOpsCon, the keynotes that I gave for KubeCon - it’s all there; there’s a YouTube video and the slides are online, and I’ll give you a link afterwards that you can share with your audience… But if you look at it, you’ll see that I’m talking about how in technology we have these sort of pivotal moments. And for me, a really important technology moment in the last few years - not the only one, by any means - was the iPhone. When the iPod appeared, people said “Why do we need this? We have the Walkman already.” And then they bought it and they went “Oh, that’s actually a quite cool device.” And then we saw that the Zune was a way to do it badly… You know, don’t make it brown; tip for next time. Things like this. But it wasn’t really until that form factor also became a phone and then a camera that people went “Hang on a minute, this could be a really profound change in how we interact with technology…” And you know, not necessarily even in a good way. I mean, it brings things like Facebook into your life maybe too intrusively. But it was a really important moment. And if you look at the Apple share price and at the graph of other Apple releases over time, you’ll see there’s a big push up after this thing appears, followed closely by the app store.

And what the app store did was it made it possible to put anything on your phone. And those two together created what people call “the iPhone moment” or “the App Store moment.” It’s just a sea change in convenience and experience. And if you think about the web, we need these richer experiences, because before that we had HTML, then we had Ajax… But while that was going on, we had phone apps, and I think you may recall there was this great technology called WAP at one point, that everybody was saying was gonna change the game… And I was wondering how my Nokia 9000 or whatever it was, with its screen that was about the size of a coin, and it was LCD, was going to help me to do something cool online, like I could with Amazon on Yahoo! on my phone. And obviously, the technology just wasn’t up to it. But with the iPhone, and of course, high bandwidth, you have essentially a computer in your hand, and that means that you can start to build a whole new kind of human experience around it, for better or for worse.

So I think that it is appropriate to draw an analogy between then and now, and say that we have not had yet the iPhone moment for cloud-native technology. I mean, a smaller, but equally profound shift was with the web, which then produced Amazon, eBay etc. I think with the web we didn’t have one moment, we just had lots. You choose your favorite. My favorite might be the day that Google came out, or the day that Bill Gates said “The internet is important”, or whatever it was. With iPhone, it was a clear, single point in time. And I think whether it’s a shift of several steps or one thing, cloud-native has yet to achieve the kind of obvious change that those moments have. And the characteristic is you can’t imagine what life was like before it.

Adam Stacoviak: That’s right, I just tell you. Well, there was an episode… I think it was called “ICO or death”, or something like that… I have to look it up. But they were gonna ICO, because they saw – it was a long story short; I shouldn’t even go into this, but I’m gonna have to, really quickly. I’ll keep it short. They accidentally gave away credits on their network, and outside of their control, they got into the open market… And they couldn’t get those credits back because they had given them away, and they traded hands a few different times, and when they finally caught up with it, the credits were worth millions of dollars. And so, like any good HBO show on current trends and technology, they had to go the route of essentially finding out “How could ICOs fit into this world?”

[01:39:59.16] The episode was called “Initial coin offering.” So they just dove into it, basically. They had some value out there. It’s interesting how you can be fueled by your most loyal fans, versus simply the incumbents of venture capital. Not that those are bad people by any means, or [01:40:17.17] better than the other; it’s just a new avenue. It’s a new reduction in the barrier to entry.

Amal Hussein: [01:00:01.20] I kind of wanna throw the coin over… [laughter]

Jerome Hardaway: Yeah, I’m there. Right now, it’s only up in these industries, up in these companies. So you now have to leverage Vets Who Code to do that. That’s your goal, that’s your entire mission. If you need training, talk to me. If you need me to connect you with a high-level mentor, speak to me. If you need me to connect you to – like, if you have a passion that I may not be the best person [unintelligible 00:56:50.00] I’m connected in every part of the industry. I can connect you with people in infrastructure and SRE, even though I don’t do anything with SRE. I can [unintelligible 00:57:02.18] people that are making low-level build tools… I know a guy with a Ph.D. in Egypt who focuses on garbage collection. So I can do it all. I can do what I do well, and if I don’t do it well, I know who to connect you to. So talk to me about what you wanna do, where you wanna go.

And that’s my life’s work. That’s my legacy. I wanna be the dude that – basically, I want the HashFlag coin to be worth, on average, $85,000 to a recruiter. When that HR or recruiter sees that, “Oh, that’s an $85,000 troop there, minimum. That person right there is worth $85,000, because he/she has the HashFlag coin.” That’s my life’s work.

The average salary of entry-level is anywhere – the national average is from 49k to 94k. I want that number to be 85k. So they know “I need to start at 85k, and probably go up some, because I know that this person is gonna come into my industry, come to our job, and they’re gonna bring it.”

That’s my life’s work. When I’m done with Vets Who Code, I want people to be able to say that - “Jerome - his people come in here and they crush it.”

Gergely Orosz: [20:13] And I think it’s really interesting to see his background, and that can also speak to anyone who’s thinking for starting a company. And by the way, if you’re a senior engineer right now, and you’re not thinking about starting a company one day, the next couple of years is gonna be a wonderful time. It’s relatively easy to raise capital, there’s still a lot of opportunities…

So the background of Jeff and how he did it - he was a CTO or an early employee at a startup that was a skateboard shop, and they did a website, and he kind of went through the whole small startup that didn’t go anywhere. He had that experience. For three years he worked hard, and he saw how hard startups were… Then he went to Amazon, where he just learned the ropes. They put him in charge of this really small unit, and he learned the Amazon way, pretty much. He was there when the API started, and his team started doing it. I think he was also promoted; I might be incorrect, but - he was there for a few years, and then he started Twilio.

So when he started, he had the startup experience, and he had the large company experience. And my personal take on the book - you should get Jeff on the show to talk about this, but my feeling was that he was able to fall back on tons of stuff on what he saw at Amazon, and what worked; he put very similar structures in place early on. He also had the software development know-how, so he was able to call BS on stuff. And he is one of the good examples of how a software engineer building a software company is often very successful. Twilio is massive in terms of market capitalization, developer love, all of those things. He always protected his team, in terms of he didn’t ask unrealistic things, he didn’t do the things that non-technical founders do, who I now talk to a bunch these days, and it’s a very different world… So yeah, I think it served him well.

So what my take-away is if you wanna build a startup that can be a unicorn one day, work at a large company, work at a startup, and you now have the skills. You literally have the skills. Work at a decent large company… Amazon is a good example for this. Google might not be the best one, because if you start to copy whatever Google does in your tiny startup - yeah, it’s a coin toss. I wouldn’t trust that fully… Because Amazon was still very scrappy; they built out all their things very independently. They didn’t have central things to fall back on.

Matt Rickard: Yeah, you’ve probably never heard of it, because I tried to come up with it myself… I tried to coin the term. So it’s a new thing. But it’s me trying to describe a pattern that I’ve seen in software configuration where a configuration seems to evolve through specific, increasing levels of flexibility and complexity before returning to either hardcoded values or Bash. So you go from hardcoded values, which are the easiest/simplest configuration, but provide very little flexibility.

[08:04] And as the program surface starts to increase, and with that configuration you start to incorporate environment variables, flags, and eventually you wanna start to check that into version control, so you turn it into a configuration file, maybe Yaml, JSON, something like that… And then as you kind of turn on this heptagon of configuration - and I only call that heptagon just because a lot of the ideas came from Kubernetes, and the Kubernetes logo has got the seven points, and it just kind of worked out well…

Adam Stacoviak: But the novice is going to question. That’s a good idea, to pair those two together, because you always have that both sides of the better coin.