Jerod Santo: Show us Rust. [win alert] Number three answer, with 19 responses. So you’re on the board, but Una now has the chance to match higher. If she can pick number one or number two, then the CSS Podcast plays round one. Una, what do you think?

Danielle Rath: I would love to. This is called “The five levels of fatigue”, and it’s a pyramid with level zero at the bottom of the pyramid, and level five at the tippy-tippy-top. The reason I made it a pyramid is because if you think about the X axis, that is your productivity. So the base of the pyramid is very wide. That’s how alert and engaged and excited you feel about life. And as you get up to the tippy-top, the X axis, the tippy-top of the pyramid is very small. That’s because when you get all the way to fatigue level five of five, your productivity is very low.

[56:29] So with each level of fatigue there’s different symptoms you might feel. For example, fatigue level one is commonly associated with dehydration and drowsiness. So if you’re doing something that’s boring and repetitive, or if you’re doing something that you haven’t had enough water recently, you can feel tired, but caffeine is not the solution. You might need water, you might need to get up, you might need to take a little break and change what you’re doing. Caffeine is not the solution for fatigue level one, because your fatigue is coming from boredom and dehydration.

At fatigue level two you’re feeling more tired, and you might need some caffeine, but not a lot of it. You feel a little distracted, you feel a little lethargic, so you just need a little bit of caffeine, or you need someone to make you laugh. You might need something to boost that dopamine in other ways than caffeine.

The next level up from that is when you’re really stressed out and struggling. That’s fatigue level three. So you need more caffeine at this point, and I’ve got different recommendations in the book… But at fatigue level three, because that level is also associated with stress, there’s other things you can do to reduce your stress. At fatigue level three, because that level is associated with stress, one of the things that you can do is just to get started. Sometimes we have this stress because we’ve got 20 million things on our to-do list; but if you just get started, sometimes that momentum can carry you through and you feel less stressed because you’re actually moving forward, you’re actually doing something.

Level four is when you’re exhausted. At this point, you have the strongest amount of caffeine that you can, you have 400 milligrams, which is all you’re allowed to have in a day, and you need to start easing up, you need to start delegating. You need to start admitting that you’re in an energy emergency, and you won’t be able to do everything you plan. You need to be comfortable with letting someone down, because it’s like that point in the airplane before you crash; you need to put your mask on before you help someone else, right? Fatigue level four is that point - you need to help yourself, because you can’t pour from an empty cup.

And fatigue level five of five is zombieville. There is no amount of caffeine that will help you at fatigue level five, so you just need z’s, you just need sleep.

Roland Shoemaker: Yeah, it’s really interesting, I think… TLS is a really complicated specification, and it’s kind of additionally complicated by the fact that while it over-specifies a lot of things, it also underspecifies some things… So there’s a lot of corner cases where you kind of get weird interoperability bugs. And BoringSSL is kind of the de facto TLS implementation now, basically because it is what Chrome runs. So it has probably handled more TLS handshakes than arguably any other software; and someone will probably argue with me about that, but… And so it’s kind of the golden implementation for TLS. They have this giant test suite that was originally just for BoringSSL, and it would test every single protocol behavior that they could think of, and had kind of regression cases for tons of bugs that they had seen… And at some point, David Benjamin, one of the engineers, worked on it to make the test suite applicable to other TLS stacks. So you can wire up this complicated system to run a Go TLS client or a Go TLS server against a kind of modified version of it what it turns out is the Go TLS stack… Which is written in a way to kind of introduce bugs and cause strange behaviors, so that you can kind of see “Is your TLS stack actually doing the right thing?”

[00:40:25.09] We’ve had this long-running plan to kind of integrate this to test the real Go TLS stack… Because it’s really hard to test. It turns out that there are so many different things in TLS that you could do either correctly or incorrectly… And trying to manually write unit tests for every single possible behavior across three or four different versions of TLS, and with three or four different types of keys, or certificates… All of this is impossible for a single person to do. So BoGo is hopefully going to eventually replace a lot of our TLS tests, and instead of kind of writing individual unit tests ourselves, we can then go and contribute tests to the BoGo Test Suite… Which tests BoringSSL, but it is also used for Rust TLS, which is one of the largest Rust TLS stacks… And I think one of the Python ones also uses it. There’s a handful of people who are using it… So it lets us kind of not only test ourselves better, but it also kind of allows for the TLS ecosystem to slowly converge on what is the right answer for some of these things.

When I first ran the BoGo test suite there were a number of real bugs, which we fixed… But the vast majority where - you know, when you encounter an error in TLS, you send what’s called an alert, which is like a single [unintelligible 00:41:56.24] that indicates what it was that went wrong during your TLS handshake. And in the TLS specifications a lot of these are under-specified. It’s like “Oh, you abort the connection, but no one ever says what alert do you send.” So when we ran these tests, we got like 100 or 200 errors that say “You’re sending the wrong alert when this thing happens.” And I went and I talked to David Benjamin, who wrote a lot of these tests, and I said “Well, why did you pick this alert?” And his response was “Ehm… It’s just what made sense…” [laughs] And I could probably sit down and argue for a good hour or so about why some of the alerts we picked were better than the alerts that he picked… But in reality, this is not important to anyone; it’s probably more important that we do the same thing that BoringSSL does, than being semantically correct about which alert you’re sending…

Kelsey Hightower: I think it also helps to understand the lifecycle of data. When you collect raw bytes, that’s data; so the tire pressure, is this package on time or not - that’s just raw data, and it’s kind of not useful by itself. And then once you start to put some context around it, then you have the chance for that data to become knowledge. Now the data has knowledge. “These things are late. We’re gonna have to issue money back.”

So once the data becomes knowledge, then the people who operate - so if you’re a mechanic, having the data light up a tire sensor or telling you that you have a failing alternator… That knowledge will allow you to repair the car, so that’s who needs to know about that, because you’re the mechanic.

Now, what you report up to the higher things is that this truck is not going to make it because it’s a repair, and we need to send another truck to do something with it. So all of this, like - how can we go from data becoming just a raw collection of bytes, into knowledge.

Sometimes, when you can’t figure out what to do automatically, that’s when we have to start sending alerts. We’re at least going to advise someone, “There’s something wrong. I’m gonna try to give you as much context as possible”, and then someone who is more knowledgeable can come along and interpret that data. So the analytic parts of this – because monitoring is not just a real-time thing where you just look at the screen and react in real-time. A lot of people would say “You should also monitor the things that didn’t result in any perceived failures.”

That airplane with multiple redundant systems - maybe something did go wrong, and it didn’t result in a plane crash. But you still should go back and analyze that data, to say “Wow, why does this component always fail, even though it wasn’t catastrophic?” So I think monitoring then, at that point, becomes this analytical journey of just having as much data as you can, so people can turn that into some form of knowledge about “Maybe we can predict when we need to repair.” So I just think all of those layers – it’s just all about translating data into knowledge as fast as you can.

Dax Raad: Yeah, so - yes, we are a business, and we’re doing this not for any altruistic reasons. We’re here to try to make a lot of money. That’s straight up why we’re here. So the reason we are open source is because we actually believe in the model, like strategically. The thing I said earlier is we’re not trying to create some idealized platform for doing development, because we fundamentally don’t believe that those types of platforms work at scale. Everyone just needs to do things a little bit different.

[14:06] And yes, maybe your idealized version is like a much better experience than SST, but the reality is that technology is always shifting, there’s new frameworks coming out… You just can’t predict this stuff. So I fundamentally believe something like this needs to be open source, because we need help integrating with all kinds of things. So we’ve integrated with every single meta framework; not every single one, we’re not done with this. We’re continuing to do more and more of this, but integrating with things like Astro, with Remix, with Next.js… I didn’t know any of this was going to – or I didn’t know that a lot of this was going to exist years ago, right?

So we believe in the open source model; whenever there’s a feature that we want to build, if it can be built in an open source way, we’re going to put it in a framework. Our business model for making money is there’s certain things you need once you go to prod that doesn’t really make sense to be part of the framework. Our strategy is pretty simple - when our users go to prod, there’s additional services they’re using to round out the product experience. It might be anything like error reporting, maybe they’re using Sentry to catch exceptions. It might be something they’re using for logging. And we’re starting the process of this now, we’re building complementary, completely optional services that integrate very nicely with SST. So because we deploy all your infrastructure, we know exactly what to monitor for exceptions; no configuration will catch all your exceptions, will alert you to do all that, kind of Sentry-like experience for that.

So we can do these complimentary things. Even those things, we’re actually building in an open source way. So if you want, you could go self-host that whole control plane and do everything. We don’t think anyone is actually going to do it; we’re open sourcing it more so people can learn from how we build things. But that’s open source… But obviously, we’re gonna have our own hosted version of that, that people sign up and pay for. And this model has been like – it exists in a bunch of other places, and I think it’s well proven. The thing we look at is in the Laravel ecosystem - Laravel is the framework, kind of similar to Next.js, or something like that, just a lot more like complete… And then Vapor is a deployment platform for Laravel projects, which you pay for, and it contains a bunch of production-level features you need. So we have no shortage of ideas for stuff there, so we’re not super-concerned about the monetization part. Just given where our framework sits, there’s just like infinite opportunities.

Daniel Whitenack: This is a totally off-the-wall idea and train of thought, that has nothing to do with anything, but Chris, I know all that you do with animals… My wife is giving me a bit of a hard time, because I’m kind of geeking out over this… In our patio, she moved some house plants out there, and we have a bird that established a home in one of the pots out there, and has laid a couple of eggs so far, and so I’m trying to determine what is the appropriate live stream camera that I can set up to observe this, and then what – there has to be some type of like alert that I can do that’s AI-driven, that tells me when an egg has hatched, or when there’s feeding going on, or something. Yeah, this has consumed my thoughts for the past few days.

Dan Mangum: I definitely can’t take credit for the term, but the term is ClickOps, where you go in and provision your infrastructure by clicking around in the console. I don’t know who to attribute for that, but it’s certainly not myself. But hopefully, that’s not what most organizations are doing… But kind of the next evolution of that - with things like Terraform or Pulumi or infrastructure-as-code tools. And those are really great, because you can version that config that you run to go ahead and provision your infrastructure. That’s an awesome model.

One of the things that could be nice about that is that you don’t have a service that you have to worry about to provision that infrastructure. You run it from your local machine. The drawback of that is that if you’re not actively running something, then that infrastructure is free to change or be modified, and that’s especially a big deal in an organization where you have lots of people provisioning and modifying infrastructure, and things like that. So having that hosted control plane, as you’re saying, you can allow someone like Upbound to host that for you. And then you also don’t have to worry about your infrastructure getting out of sync, because as soon as it is, you can get an alert for it. As soon as something goes down, we can bring it back up for you, according to how you would like that behavior to be reflected. So I think you’re spot on with both the fact that having that hosted kind of central point of provisioning infrastructure is really important, but also just having something that’s constantly evaluating your infrastructure is a big gain over what most organizations are doing.

Maggie Johnson-Pint: [56:27] That gap is real. So one thing that I’m focused on - actually, I’m writing out the feature set this week, and the engineering team is starting on it, is helping people see things that are relevant to them. Like, the problem with a lot of mapping tools and a lot of tooling out there is it just gives you a puke of everything, from everybody. And a lot of the work that we’re putting in - like I said, part of the reason we started with alerts is you can associate them with a team - is to help people see sub-graphs of their production system that relates to them, and then on the edges of the graph, instead of telling you about the whole world, tell you “Hey, just so you know, the next cluster graph over is this one. The next cluster graph over this way is this one”, to help people build out a model of where they sit technically, and where the teams around them sit technically.

And right now, a feature I’m working on building out - I call it “Spotify style onboarding.” Like, what GitHub repos do you care about? What servers do you care about? And I think that starts to help us at least get in a place where we can reason about what belongs to us.

The nice part about graphs, when you assemble a graph in graph theory is you can actually use graph traversals to say, “Well, what’s within three traversals of me? What’s near me?” And I think this will help organizations start to reason a little bit better, and start to understand how data is flowing through their system, and understand those ownership models, and hopefully have a better shot at collaboration, instead of just “I don’t know. I don’t know whose that is, I don’t know where it belongs.” And hopefully that starts to set some new cultural paradigms we can work under.

Definitely, when you talk to people who came out of that Google SRE, they said the best thing they had going for them was this ability both to cross-observe and to cross-automate. It wasn’t like you had to go beg another team to do something for you. And so whatever we can do to enable people to work through the full scope of the system without begging and yelling…

Tom Wilkie: Thank you very much, yeah. You know, I miss so much out of what’s happened because unified alerting is a huge step in the Grafana story. I’m really pleased as the way the company came together. We used to have two alerting systems - we had the Grafana alerting system and the Prometheus alerting system. And they were worlds apart. On one hand, the Grafana alerting system is probably the easiest one that exists out there; it’s very accessible, very easy to get started with… And on the other hand, the Prometheus system is probably one of the most sophisticated and powerful ones.

So I think it was really exciting how the team could combine the power of the Prometheus system, with multi-dimensional alerts, with alert managers routing, grouping and deduping and silencing… And bundle all these features into Grafana in a way that makes them easy to use and gives you that level of user experience that people have come to expect. And best of all, we haven’t duplicated any features. We’re just using alert manager under the hood. We’re using the same API as Prometheus under the hood. So it’s true to our open source roots as well, and that’s – the team did a fantastic job with unified alerting.

I think the thing you said about cloud, the generous free tier, for instance - we launched that in January, I think…

Dave Blakey: [51:56] Yeah, exactly. But a lot of the time an anomaly can be informative. I think that’s also up to the team that gets them, to make sure that they do the right thing. If our website gets ten times the views after this podcast, I’m happy to be told. It’s not gonna go offline… But you know, sometimes informative telemetry is not necessarily a problem, but yeah, it can reach the point of spam, which then people start to ignore, which is a big problem… But you know, it’s a balancing act. With alerts and with anomaly detection it’s all about balancing it; you wanna make sure you pick stuff up.

The problem at our scale becomes so vast, because – let’s use a use case. Let’s say a banking client of ours - they might have systems in 20 different countries. Now, how many failed logins per second do you think they get? It could be 500, it could be 1,000 that they get per second… So if they’ve got 10 more, or 20 more, or 50 more, it might not detect an anomaly. But what if they get 50 more in all of their locations around the world, all from the same country? Is that a problem? Probably it is. So sometimes – that’s the funny thing about telemetry, people tend to zoom all the way in.

We were talking about trace IDEs, and “What is the individual request?” A lot of the time that’s very important, but sometimes it’s actually really important to have that 10,000-foot view, where you’re just like “What is the lay of the land? What does it all look like as a picture?” And that’s also something that’s not that easy to do now. There’s not a lot of standard stuff for that, or just best practices, like “How do you set up your dashboarding, if you’re using Prometheus, or if you’re using Datadog, or whatever it is…?” Using that is like a big failing, I think, in DevOps teams and traditional teams today. It’s making sure that you always go back to your telemetry and say “Why didn’t this tell us about this problem before it happened?” There should be almost like a root cause analysis of the issues… And it doesn’t have to be this fancy process, but just going “Why were we not aware of this, now that we understand what it was?”

Jason Warner: We started on something right after I joined GitHub called Security Alerts. It was the idea that we’re gonna need to bring security to GitHub, and security manifestations, and just a stronger idea of security around the code… And the first foray into that with security alerts was met with mixed success. But as we pushed through it and got further down that path, and then eventually post-acquisition with Microsoft, we ended up buying a company called Semmle out of the U.K, and combining it with that we’ve had a lot more success… Though that was definitely more Horizon 2. It was just pretty well understood that we needed to do something; we weren’t quite sure exactly what. The first entry into it was met with a little bit of lukewarmness, but once we iterated at it and moved it further down the path, we had a lot more luck with it.

Nayana Shetty: I’m going to give an example so people can relate to it. I was in one of the teams where we used to get close to 1,500 alerts on a weekly basis… We had around 80-odd microservices. So it wasn’t like just one microservice, or anything. But then my team was three people looking at this, and it’s at that point which you realize “Are they actually looking at this thing, or is it all just being ignored as just noise? Let’s just ignore it.” And I think it’s that point where you start ignoring your alerts, is where you’ve gone to that stage where you can’t take anymore alerts. You’re fatigued with the whole alerting itself. And I think it’s better to have less alerts, for the most important things, rather than have too many and try to filter it out.

One of the exercises we did when we have these alert fatigues and like a thousand-odd alerts is we consciously stopped some of the alerts to see who will start shouting. And it happened that more than 50% of these alerts when we turned off, no one actually shouted at us. Like, “Was that even important?” Going through that exercise on a regular basis, where you see if you’re ignoring more than at least 10% of your alerts, then go and do something about it. Maybe turn them off, and no one will care.

[43:16] I think teams need to be conscious that it’s okay to miss a faulty alert, compared to missing out on a real alert which would have cost us millions of pounds, or whatever it is. So I think it’s being careful to put the right alerts in, and stopping at that, and not just going overboard with it. Let’s take an example. We have Grafana in our systems, and we have alerting with Grafana. I have this tool, so I can put as many alerts as possible. Not going wild with it, but actually knowing where to stop. That’s how I would describe this whole alert fatigue. And with time, it does happen with teams. So it’s worth going back and auditing them and making sure you keep them clean as much as possible.

Angelica Hill: Yes, product manager at the New York Times. I manage emails, newsletters, and breaking news alerts. I work with wonderful developers!

Simon Wijckmans: Yeah. So I tried this in my career, and it was incredibly noisy, and it made for alert fatigue, and it wasn’t helpful. So the angle we’ve taken is quite different, where we basically try to understand what types of data are being accessed, or what types of actions are taking place… And when there’s a substantial change in that, then we will start alerting.

If we must alert every time a script slightly changes - well, that is incredibly noisy, because a lot of these scripts are obfuscated. And if they make a small change to a timestamp, or to anything in there - it could be as simple as a version number. Because of the obfuscation working a bit like a hash, it becomes completely different. So using abstract syntax trees and all types of other tricks we pull on our site to actually understand on a higher level how the script performs, what it does, what it accesses, and then alerting when that substantially changes - that is the better approach.

The way we also work is – of course, we are always working to improve this. We actually want to make this a product that blocks on your behalf if it must be blocked. And we do not rely on threat feed intel. Of course, we use it, so that if there’s a low-hanging fruit, we’ve got it. But that’s not our core thing. We built our own detection engines, that look at changes over time. And because we’ve also got that free tier, and we’ll have a bunch of people’s blogs where so-called there were no scripts, but there were then three scripts, we can of course see these scripts on many different websites. So before it becomes a problem for one of our big, high-security customers, we probably would have already seen it on some other websites as well.

So that free tier is not just an amazing “Use our thing and hopefully someday it’ll pay for us.” No, it is also actively making the product better for everybody, safer for everybody. And so by using our thing, you’re basically making your own life on the web a little safer, if of course people use our product. And that’s a pretty powerful position to be in. So we don’t really want to go down the heavy alerting route, because it just doesn’t really help. It goes in a massive queue of alerts people receive every day, and then it really only causes trouble afterwards where it’s “Hey, why didn’t you respond to this alert?” Well, because I get 500 of them a week. I cannot deal with all of these alerts. It just creates more pressure on the engineer that’s ultimately going to be held responsible for when it goes wrong. It doesn’t help. We have to be very diligent about the alerts that we send, and when we send them.

Matvey Kukuy: So Grafana OnCall - it’s very important to understand what is this product about. It’s not about insane colors. Grafana itself is for, j-rating colors. It’s not for managing large incidents, when you have like a 17-hour long incident and you need help and communication… It’s not about that. It’s focused on notification for right people, right time, according to on-call schedules and different channels. So if you need to go to Grafana and set up your alerts, feel free to play with unified alerting, which was revealed recently also… And after you configure your alerts, rules and thresholds, you could open the Grafana OnCall, you’ll see the small onboarding instructions there, like “Do a, b, c, d.” So you click at integration there, and it’s a quick integration with current Grafana.

Crystal Williams-Brown: Well, I am working on an app called Upstate, that tracks bills in North Carolina and alerts people about anything that’s happening with them…

Haroon Meer: Okay. So effectively, what we do is we build products to let people know when they’ve been compromised. The opening logic is – so in our previous lives, most of our early team worked as pentesters, so breaking into networks. And one of the terrible secrets is that for years and years we’d break into networks all around the world, and nobody knows. Not until you hand in the report. And this hasn’t changed much. So it happens when you’re doing pentesting, it happens when real attacks happen. And so companies just find out they’ve been compromised months after. And so our whole pitch is to try to fix that.

So canaries are an old concept, which are honey pots. And honey pots have gotten a bad name historically, because mostly, they were used by the research community. So people would put up honey pots and say “Look at how many attacks were from Russia. Look at how many attacks were from China”, which is pretty useless for real world activities. And what we do is we say “If you had honey pots on your network, and they were really low effort to deploy, they end up giving you a really high signal.” So you find out at two in the morning that Bob from accounting just tried to log into this machine that really shouldn’t be there. And what it works on is just the logic that the people on your network or in your infrastructure know your infrastructure, but attackers who land there need to situate themselves. And so typically, those attackers run around like bulls in China shops, and they get there. And so they explore things and they touch stuff. And inevitably, they touch these canaries, giving you a really high signal that badness is happening. And that’s all we do with Canary. We make it super-easy to deploy them, so that people actually do it, and then we focus really hard on not generating extra noise. So customers with like hundreds of canaries will get four alerts a year, so that when they get an alert, they know they need to react to it.

[07:50] And so Canary was the first product we built, and then we built Canary tokens, which are the same concept, but much smaller trip wires. And for the broad applicability, for your audience, we’ll give you for example an AWS credential file. You put it on your CFO’s laptop, and when anyone uses that credential file, that API token, you will get a message saying “Listen, the AWS creds that were only on your CFO’s machine - somebody just used it to log it.” So again, you get a really high-quality signal that tells you someone was on your CFO’s machine.

And so Canary tokens are bunches of little tricks like that, that are really hard for attackers to resist, but gives you a really high-quality signal that something is going wrong. And for Canary tokens, we give them away free. And so literally, millions of people have used them, or use them to figure out when they’ve been breached.

Tom Wilkie: Yeah. I mean, I would take a country view there… Like, to what extent is monitoring just extended integration testing? If you know – and you should be running any sufficiently complicated system, you should be injecting kind of artificial load in your system and measure its behavior, right? So why not just call your integration tests “Actually, it’s just a staging environment, and we’re just gonna pummel it with artificial load, and use the same production alerts, the same monitoring and playbook and everything we use on prod, we’re just gonna use it on integration.”

FireHydrant Clip: PagerDuty, I don’t want to say they’re evil, but they’re an evil that we’ve had to maintain. I know all of our engineering teams as well as myself are interested in getting this moving the correct direction as right now just managing and maintaining our user seats has become problematic.

That’s, that’s, that’s really good actually. This is, this is a consistent problem for us and teams is that covering these sorts of ad hoc. time frames is very difficult. Um, you know, putting in like overrides and specific days and different new ships is, is quite onerous. Oh, and you did the most important piece, which is, didn’t tie them together.

Cause that’s half the problem with Pager duty, right? Is I get all these alerts and then I get an incident per alert. And generally speaking, when you go sideways, you get lots of alerts because lots of things are broken, but you only have one incident. Yeah. I’m super impressed with that because. Being able to assign to different teams is an issue for us because, um, like the one, the one alert fires for one team, and then it seems like to have to bounce around and it never does.

Uh, which then means that we have tons of communication issues because like people aren’t updated. No, I mean, to, to be open and honest, uh, when can we switch?

Jerod Santo: Okay, there’s our number one, HTTP. [win alert] With an overwhelming 90 people saying HTTP, only worth five points. But hey, it’s still five points. We go back to team zero and Amy.

Gregory Richardson: Ismael touched a little bit, he kind of grazed over LLMs, and I’m glad you only grazed over it because of what I’m about to say. We think LLMs, as good as they are, and they have some excellent use cases and value, I think they contribute to a lot of the noise and the hype machines that we hear in the industry right now. I’ll speak specifically for cybersecurity. I am not yet convinced of the utility, the usefulness of an LLM, particularly for its natural language ability, ability to process things via natural language… I’m not sure that that was the problem we had. I speak to psych analysts, and chief information security officers literally on a daily basis. That’s my job. I can’t remember in the last 30 years doing this, that a group of operators, SOC analysts etc. have told me, “You know what would be great, Greg? We don’t know how to extract the data from our tools. If we could only say that in natural language, that would really help.”

That’s not the problem. The people that are doing these jobs in the SOCs etc. are very adept at their tools. They don’t have the problem communicating with the tools, and writing a parsing command, or a query or whatever, to extract the data. That’s not the issue. There’s other things that AI and machine learning can help with. Classification is a big one. Ismael has already referred to prediction. I think that’s a very, very big one, that is underutilized today.

But classification - how do we classify not only files and hashes, but behaviors, indicators of attack, indicators of compromise? How are we able to classify these three things that are connected together, or in the case of a cyber attack these 50 things, these 50 behaviors or indicators we find - how can we pull them all together and say “Listen, these all belong together. These 10 things that we’ve found on your network and these 15 things that we’ve found on your endpoint, and these 12 other things that we’ve found simultaneously, in the same temporal window in your cloud environment, they all belong together and they’re all part of one attack.” That classification process - I think that’s somewhere where AI can help, because that’s where the gap is. Taking the ton of data that comes in, that swamps our security operation centers with alert fatigue, parsing through that to “make sense of it” and kind of narrow it down to a few cases. And when I say few, that few may be thousands still, but it’s an order of magnitude or more drop from the tens or hundreds of thousands of events that you get. If you can drop that down to a significantly smaller amount of cases and then tackle those cases, that’s one of the problems that I see AI solving in cybersecurity extremely well.

Mike Riley: Yeah. Yeah, that’s the other thing you could do - you could use another project in the book that utilizes the Philips Hue lighting system, so that if that threshold is exceeded, then you can have a light that turns on… And if you’ve got the colored lighting, you could make it a red light. You could even make it pulsating, so that when you come into the physical area, you’ll see that red pulsating light saying “Oh, I’ve got a problem.”

I actually wrote an article about this similar type of scenario… It was a Medium article. And it was basically using Grafana. Because I was making sure that I’ve got a constant internet connection. And if the internet connection went down - well, rather than hearing from my family, “Hey, there’s no internet”, I automatically get a red alert with the lighting that I have, strip lighting I have across my room here, which will pulsate red when Grafana says “Your internet has been down for more than a minute. You may wanna check on things.” So I know about it visually before anybody else does.

Chris Benson: I would argue you’re getting there, but I want to throw in a couple of things that would raise the value proposition of applying a data fabric. You mentioned the fact that all these producers, the actual machines or whatever, are going to S3, as an example… But what I would argue is the use cases that you described in the example - none of those are real-time. None of those imply that there might be a life or death concern there. It might be that as we have more and more machines that are constantly producing data, and sensors constantly producing data, that you could have critical care patients in that hospital, that it’s not just an analytical thing, but there’s also – but even the analytics themselves, there are analytics that are needed in near real-time to save a life, and there are analytics that might be able to happen kind of whenever, because you’re doing a study maybe or something, and they’re not urgent. And there might also be sensor data on that patient that needs to alert a condition that the sensor is picking up, and you need a doctor in that room working on that patient immediately. So it’s not going to go to an S3 bucket, or at least not just to an S3 bucket; it also needs to alert with specific data that goes right to whatever kind of unit that doctor is carrying around, who’s in a completely different part of the hospital, and that doctor understands, “I have an urgent situation and it’s measured in seconds, and I need to do something.” And things like that happen in real life. It happens to date in real life, but we don’t have very good mechanisms to deal with that.

So if you can automate that urgent thing to where you’re able to prioritize the data that’s coming from a sensor that is critical immediately to a patient’s life would get priority over a data that’s just going to the S3 bucket at any point if there’s any kind of resource contention. And so you’re able to get the information to that. And it may be that not all things are going to the S3 bucket. As a data store, there might be data stores that are collecting things for historical reasons, but you also may be sharing directly between one device and another device, without putting a store between them, so that you get more efficient. Or you may be meshing multiple sensors together regarding that patient together, so that you get a complete picture of that patient’s condition at the time.

And so as you do that, the data fabric gives you that logistical routing prioritization, the metadata and the intelligent services wrapped around it, which may include AI, though you may have lots of deep learning models that are part of that mesh that are able to actually do the metadata tagging. It looks at the data, it looks at other considerations that are inputs to the model, and the model is giving you inference that say, “This is what you do with the data right now.” So the model itself can be a service and supporting the data fabric.

Björn Rabenstein: [11:44] Actually, we have a bunch of people at Grafana working on Loki. It’s like a big deal, obviously. I don’t even feel like I would do them justice if I now tell them. You should probably ask later… I mean, perhaps you should take it from the other way - people see Prometheus, they realize it’s this hot thing that they should use, they see all the success they have, and then they try to shoehorn all their observability use cases into Prometheus, and then they start to use Prometheus for event logging… And Prometheus is a really bad event logging system. That’s a lot where we have to fight – not fight; where we have to convince people that they shouldn’t do this, even if they’re angry at us.

But then there’s also the other – whatever the backlash where the logs processing people try to solve everything… Yeah, we kind of have more this inclusive picture; you need all those tools, you need to combine them nicely, and Loki has this idea where you take some parts of Prometheus, which is like service discovery and labeling, and use the exact same thing for logs collection, and then it’s easy to connect the dots and jump from an alert with certain labels, into the appropriate logs that you have collected. It goes into that area, but I guess you will talk a lot about that with Tom.

Jerod Santo: Is Ruby one of the most favorite languages of our listeners? [win alert] Number three, with eight respondents naming Ruby. So we have one through four matched. That’s TypeScript, JavaScript, Ruby and Rust. The fifth one’s still on the board. You have one strike, so two more chances before the chance to steal. We go back to you, Chuck.

Jerod Santo: Going back to the bindgen conversation and what you’re providing there - it sounds like all the plumbing necessary to convert things into the right formatting, the serialization or really the marshalling into classes and stuff on the JavaScript side… Recently I saw a Hello world Rust thing in WebAssembly; by the way, there’s WebAssembly Studio, which I hadn’t heard of until maybe today even, which is pretty cool. For those out there who would like to play around with this stuff, webassembly.studio (we’ll link that up as well), and you can see examples of people doing things… And they have a very simple Hello world where it’s using wasm-bindgen, and there is a function defined in the Rust side called greet, which takes a string as an argument, and then there is a function on the JavaScript side which I think you’re probably familiar with - it’s called alert… So that one’s built right in.

They are both using the opposite functions - one of the JavaScript side, one on the Rust side. That is pretty cool. You can imagine all the places that could go, Lin. I mean, once you get this native support for all the functions over here and all the functions over there, now you have access to the best of both worlds, right?