Adam Stacoviak: What you’re saying is resonating with me, because I kind of feel like it’s a when versus who. And we began the conversation with shift left, and the idea was - Deepak, you mentioned it’s not after deployment, it’s predeployment. And so we began with a conversation assuming “Oh, we’re putting more burden and pressure on the developer”, which - yes, kind of… Unless you do what you just said, which is it’s more of when, versus who. It’s at development, not so much by the developer. It’s meant to happen prior to going into production, so that you’re not shipping the defaults, in the old Mongo days or other scenarios, like a modem, you’ve got insecure defaults. It’s so that you can think about those things prior to shipping, and it doesn’t mean that it has to be the developer necessarily, it’s the team behind development… Which includes developers, but also may include compliance, it may include this MVSP, which is more of a kind of policy in a way of how to secure cloud applications… And as I’m reading this, there’s like security headers in there, there’s how you allow your customers to test the system, there’s all these sort of protocols that are in there, that can be tested… But this is something that the whole team says “Okay, this is what we want to do when it comes to shipping a cloud application that’s secure.” And that happens before development, it happens during development, but prior to production. So it’s more of a when versus a who; that’s kind of how I see your perspective shifting, at least the way you described it.

Daniel Whitenack: Basically, you kind of have to split things up a little bit by stage of your project, and also the use case that you’re considering… What I mean by stage of your project is I really encourage people, especially if they have a generative AI use case, the best thing you can do to get a sense of like - let’s say that I want to summarize news articles related to stocks that I want to trade, or something like that. The very best thing you can do is not jump right to “Okay, I’m gonna fine-tune a model for that, or spin up some crazy GPU infrastructure, or something like that.” The best thing you can do is just get some off the shelf models, and if you want to either run them – the easiest cloud way to run those would be to run them, if they’re small enough, in just a colab notebook or a hosted notebook environment like that. That’s more than enough to figure out if they’re going to work for your use case.

[00:33:58.06] Or if you want to go the more local deployment route, there’s things - like I already mentioned, you know, of course, if you want to run YOLO, that’s easier now than ever, and there’s quantized versions of that that you can run on a CPU even. You don’t need even a special type of hardware. But then for the generative side of things, there’s things like Ollama, and LLM Studio, and Llama.cpp, and these things that will allow you to prompt models and figure out if they’ll work for your use case locally.

So that’s kind of exploration stage. Then you have to decide, okay, well, if this project is a work project, I figured out maybe that I can prototype this and figure out it might work… Then you kind of have to play through the scenarios in your mind “Oh, if this is a mobile app, and I’m processing customers’ private data, maybe it makes sense to try to run a model at the edge, in my mobile app, on their device.” A Qualcomm AI model from their AI Hub, on their mobile device, and that would be really good. But if it’s a web app application, and there’s not as aggressive of a security posture, probably you want to figure out how you’re going to run and host that model in a way that makes sense to you even from a public endpoint, that’s just a product, like Together AI, or Mistral, or something like that… Or you’re going to figure out how to run it in a secure local environment with either a product that can host that model in a secure environment in your own cloud, or in your own network, or your own kind of self-deployment of that model, using things in your cloud infrastructure like SageMaker in AWS, or other things like that.

Ron Evans: So it’s the same interface. So you take your regular Go code that you’ve been using on your embedded Linux, and you copy and paste it into your new code that you’re writing for your microcontroller, change a few things based on your authentication, and it just works. So it’s a secure connection from your chip, from your Arduino Nano 33 IoT chip to your secure messaging broker in whatever cloud service you’re using. That is table stakes for the internet of things. If you don’t have that, you’re not ready.

So that was sort of our, “Here we are, we’re ready for our close-up, Mr. DeMille.”

Adam Stacoviak: Where we’re getting here from is Jerod asking you about open source, and your concerns for like does it make sense, because it’s a security tool… And then I think about things like pfSense, which can be deployed on AWS, you can protect, in the cloud, with a - in their words - a secure open source firewall that everybody can trust, security starts here kind of thing. And I just wonder if the model of open source might make sense, because of the distribution and the freedoms. And so if you found a way to do that – now, I’m not ideating on the business model necessarily, but one of pfSense’s ways to make money is through support and management, and things like that. I think that’s part of their business model.

[01:12:13.25] I just wonder if there’s ways that you can help families deploy this for themselves, provide support… That also means more people potentially, but I just wonder if open source, that route gives you the distribution and the freedoms within the software to reach the widest market globally, because it’s simply out there free, and you provided a hosted service. So you consume the open source, you host the open source service with things up around it, but I can easily also stand up my own version of that with Docker on my network, or whatever. Or on a cloud, on a VPS; I can one-click Deploy Digital Ocean if I don’t want to manage local infrastructure with whatever, and all I’m doing is API calls to my $10/month thing at DigitalOcean.

I’m just thinking out loud here, because – I’m not saying all those are good, or right, or bad, it’s just, are there freedoms that come with it being open source, and distributions that come from open source, that kind of are at the core of what you’re trying to do anyways, which is provide a useful tool to the world, that might also be more secure because of more people knowing about it, caring about it, pouring back into it etc.

Techno Tim: That’s right. So I just did kind of a walkthrough of my homelab services or my self-hosted services for 2024, I just released that a couple of weeks ago. It’s kind of a walkthrough of my network, how traffic comes in, what it’s running on, and I have both logical and physical diagrams, and then everything I use. So thinking through that, what I typically use – now, I know you can use tunnels, like you mentioned, Tailscale, or Cloudflare tunnels, or whatever you want, where you expose something externally by creating a secure tunnel from home to your proxy in the cloud, and having that proxy be where it’s exposed, versus exposing it at home. I don’t do that. I guess I do it the hard way; I’d been doing it before tunnels were kind of a thing, these reverse tunnels. I actually use a reverse proxy to do that in port forwarding.

So what I typically do - and I do two layers of it. Firstly, I use CloudFlare, their reverse proxy. So I point all my DNS to Cloudflare’s reverse proxy, which does a couple things. One, it gives me some anonymity. Whoa, peace – oh, that was Apple. I liked that.

Evan Sparks: We love to see people that try and plan for this sort of thing. They try and get a sense of “Okay, I know I have this data volume coming in next year. I know roughly speaking it’s gonna take me this long, on this many GPUs, to train my models. Let’s set aside budget and bring those resources on-prem, or secure them with long-term leases on one of the cloud providers, for the most part.” Now, that does a good job at kind of helping you plan for your baseload… But then, as always, there’s gonna be things that come up, like towards the end of the quarter, or a new model family comes out, or a new project takes really high priority that you’ve just gotta ship… In which case we see real benefits to bursting onto cloud resources.

Within the context of our system, that’s a core feature that we offer. We call it Elastic AI infrastructure. The basic idea is that if the system is configured, and there’s budget within the organization and so on, you can do that dynamic provisioning of those cloud resources, spilling work over onto them; we handle the data transfer and other aspects of that planning for you. And then as the workload goes down, those resources are leased and the organization can save money.

So we think it’s a combination of having good planning, but also maintaining some flexibility in your systems and in your processes that are required to really help AI scale within the enterprise.

Chris Wilcox: We started doing this before Actions was around, which is why we made the choices we did. We didn’t have a chance to evaluate Actions. I think if we started today, we would definitely consider Actions… But there are a few constraints - Actions don’t deal very well with long-running tasks, so that can be problematic. It’s also hard if we ever wanted to scale up.

[39:44] So we used a thing called Google Cloud Functions, which ultimately takes a small bit of Node.js (or a few other languages; in our case it’s Node) and executes it for us on an event hook. It starts up a service when we need it, and shuts it down, so it costs us very little money… And we could adapt that into Docker containers fairly straightforward, and then maybe eventually we need a Kubernetes cluster… Who knows.

We’ve also extended to have some security measures. We store none of the secrets in the functions themselves; they’re all stored in the key management service - also a thing that Google Cloud provides - and allows us to be a little more secure, a little more confident. It’s also a lot easier for us to rotate our secrets… So from a convenience standpoint it’s pretty good for us.

Amal Hussein: Not just AWS. AWS commercial and AWS – all the gov clouds, all the secure clouds, too… Which is its own thing. I’ve learned so much. I’ve learned – not run on your phone yet. Maybe someday.

Derek Collison: Yeah, I’m a huge fan of Tailscale, and the folks that were either at Google, or involved in the Go programming language that are over there. But it’s an overlay on top of the constructs that we still know, which is IP-based addressing, almost everything is point-to-point… I don’t know about Tailscale, but most cloud providers write trunk UDP broadcasts, or multicasts, and so it’s a point-to-point, location-dependent type of a system that now has surfaced up over a very secure overlay with WireGuard.

What NATS was trying to do was kind of change the notion of what we called intelligent connectivity. And specifically, when folks that are listening say “Well, what does that mean?” It’s really fairly simple from an abstract standpoint, which is everything is location-independent. So one might say, “Oh, well, things are kind of location-independent today”, but I would argue that there’s a lot of unnatural apps going on below the covers to get that to appear that way. Load balancers, GSLB, AnyCast [unintelligible 00:10:27.08] So the biggest first one is location independence, and the second one is instead of one-to-one request reply as the dominant pattern - HTTP, that’s all you can kind of do, sans SSE and WebSockets - we are end-to-end and both push and pull. And the push becomes very, very interesting for certain use cases, where instead of keep asking “Hey, have you updated your temperature? Have you updated your temperature?”, just tell me whenever you update your temperature. And for scalability and distributed systems, just those primitives from the connective layer become pretty powerful at scale.

Adam Stacoviak: I think it’s mainly – it’s less about the protocol and what the attempt is. It’s more the seemingly rogue implementation every single time I experience a passkey scenario. I also find that services are defaulting to passkeys, and it bothers me. When I want to be an email/password person, it constantly just slaps me in the face like “Where’s your passkey?” And I’m like “Nah, man. I’m doing email and password, okay?” And it just seems like it always wants to default to this thing. Adobe does it… I sign into the document cloud a lot for different agreements, and stuff like that, so I’m in there doing stuff frequently… And I like to log into the actual online service, and so I’m logged into Adobe’s web services frequently, and that’s their flow.

And I’m cool with passkeys. I actually like them, except for I think the flow and the way the UX is still implemented seems to be just not the same across the board, whereas email password is pretty much the same across the board. I feel like that’s the holdback for me. And whenever I don’t want to be passkey-first, that I want to do email/password, just anything else - that service is sort of like force-feeding me passkeys and I’m like “Nah, man. Email/password, okay?”

Now, I do use 1Password though, to just identify my stack. So unlike Jerod - Apple, simple, free, with-it kind of thing…

And I don’t think – that’s not his only reasoning for using it. I know Jerod well enough. He likes to keep his stack simple, and not have to have other extra services if he doesn’t want to, kind of thing. And I think that’s cool. That’s how they use it, and that’s cool.

[00:41:59.24] I use 1Password for a lot more than passwords. I’ve got secure notes in there, I’ve got – I mean, I don’t want to tell everybody what is my attack vector. It’s a lot, okay? It would be really bad. It would be really bad if 1Password was not a good long-term security solution and they were attacked on my behalf. I use it for more than passwords.

Marc Boorshtein: Not even, no. That would have been awesome. That would have been amazing. No, this is – so we first got involved with this in 2011, when we first got involved. It was actually kind of funny, because I had just left my full-time job to work on Tremolo, and I was at a pitch contest, a mass challenge. So I was up in Massachusetts with my partner, and I was working for a consulting company at the time, that dropped this thing in my lap. And I was like “You’ve given this away. There’s no way I can implement this and be profitable.” And I got back and started talking to my customer, and I was like “Tremolo could do this.” And they’re like “Okay, let’s pitch it to them.” And the sun, the moon and the stars aligned, and they’re like “Okay, let’s do it.” It worked beautifully. But it was Active Directory. And with Active Directory, local government typically is very, very Microsoft-focused. So everything from their on-prem, all the way up to the cloud is almost entirely Microsoft, at least in this region. Different regions are going to be different. But at least my experience has been most localities are very, very heavily Microsoft-focused.

And so they had Active Directory, and if you’ve ever worked with Active Directory, the only way to get two Active Directories to talk to each other is to just take all the firewalls and throw them out the window. So they didn’t want them hooked up directly into this thing, they didn’t want them talking to each other… Active Directory – I mean, that’s the keys to your entire kingdom. So you don’t want to just like “Yeah, we’re just going to merge these things and make them all work together.” That’s a little dangerous.

So we came in and said, “Alright, we can link these things together in a secure way.” At the time, we were doing everything via LDAP virtual directory. So if you’ve ever worked with an LDAP directory, you take a proxy that knows LDAP on the front… Think of like an HTTP proxy, but for LDAP. So it talks LDAP on the front, and then it talks whatever you need on the back, and creates this big, virtual tree. So we created this environment where it was this big virtual tree of all these different infrastructures… And then we were working with - at the time, I think it was SharePoint 2011… And so we had to integrate that in, which was a whole kind of fun.

[00:08:04.29] And we got it all working beautifully, and so they’re like “Yes, this is great. We want to move forward.” And then I learned about government procurement. So it took us almost two years to get the procurement group, and we finally got it up and running…

Gregory Richardson: It’s interesting that Ismael referred to updating our threat model, and he drew that analogy back to, you know, like the weather; like you look at the clouds, and based on what you see in the clouds, you react accordingly.

[00:43:56.28] You might pack an umbrella, or something along those lines. I think that’s such an apropos analogy, because interestingly, as a kid in the ‘70s, growing up in the Caribbean, in a hurricane zone, I remember sitting around the big box TV in the living room… I think it was even black and white at one point, because I’m old and curmodgeony, as I said earlier… And watching the predicted hurricane track for some storm that left the western coast of Africa that’s barreling towards the Caribbean islands. My island is a small five mile by seven mile island. We could get - and routinely got - decimated by a hurricane. So if a hurricane is coming, you need to know.

Those predictive tracks, with the little circles, and saying the storm looks like it’s going to go there - those in the ‘70s already were drawn and calculated by AI. It was one of the first very widely used use cases for predictive AI. So it’s interesting that Ismael uses that as an analogy, because that is exactly what we’re doing. We’re taking a use case that was well developed with weather prediction, and that’s what we’re applying to attacker prediction.

So you asked how we can apply this to the customer environment. One of the things that I am maniacally focused on right now is helping customers, as I said earlier, draw this all together. So I’m not going to get into product names, because this isn’t a sales pitch, but we’ve just developed something in the category called a managed extended detect and response tool set. And what’s unique about our approach to that - that approach and that space is not unique at all. It’s been existed. Just about every large cybersecurity vendor has something that plays in that space. What’s unique about our take on it - we are heavily focused on regardless to what your security stack consists of, that’s what we’re going to ingest. Most of the other vendors use an XDR type tool to say, “Listen, to get the maximum benefits out of our tool, you should really be using all of our stuff. So you should get our firewall, you should get our endpoints, you should get our cloud stuff, and then it’s going to be maximized.” Our take is different. Our take is we understand that you, the customer, probably struggle with two things: a widely diverse ecosystem of security tools, and the second thing, especially for medium to smaller companies, you’re probably struggling with finding the human resources to do these jobs. So we have a managed solution where our threat analysts, our security analysts, our well-trained human experts, combined with predictive AI, that as Ismael said, has been well trained on sensors, and sensor data, and threat data that we’ve been receiving for the last 10, 15 years… That’s how we are able to not just ingest all of the data, classify and recognize that this is an attack that we’ve seen before, even if it’s using novel and brand new, unseen before malware, and then provide you defensive strategies against it. That’s how I believe BlackBerry can help the market, the customers the most.

I’ve mentioned that I started on the attacker side. I was never an illegal attacker. I started as a pen tester, and then pivoted into reversing code, and doing some other things like that… And I went from there. But most of my career was on the customer side. I proactively switched, or maybe was convinced to switch to the vendor side, probably about 10 years ago, because I saw that gap. I saw that as a customer, I could buy all of these new widgets and toys, and it really wasn’t making me more secure. So I came to the vendor side to try to influence the vendor defensive motion and product strategy to put out more products that legitimately can help customers solve those two problems: the manpower problem, the diversity of toolset problem.

[00:48:05.14] The amount of times I am told by a customer “Greg, we’ll rip everything out and put in whatever you tell us”, that’s infinitesimal. It has happened. I have had a couple of greenfield customers that said “Listen, none of it’s working. Take it all out and help us replace it.” But that’s rare. Most of the customers either have financial constraints, time constraints, or some other constraint, so they need to make do with what they have. Let’s build a toolset that allows customers to use what they have, and maximize the value they extract out of it.

Dinis Cruz: Yeah. And just on the second one, I think if you’re not careful, most AI deployments are ridiculously insecure. In fact, we have to take into account that we still don’t have a good understanding for how the models work. So the reality is there’s nobody today that can tell us that these models don’t have ridiculous backdoors in there. Even non-intentional. Even maybe just the way it works.

When we started this, people thought that a string copy was okay. People thought that a little catch between a memory copy in the OS was okay. And then we realized that you can drive buffer overflows, ridiculous exploits through it. So I think we’re in a nation state at the moment now, in the early days of understanding everything you can do with a model.

So my kind of view in this is that models that you want to use on that “How to use models secure” should be read-only, should not learn; you don’t want them to almost bring any content. You want to give them the content, you run them in complete isolation, and you assume that whatever you put on it is already exposed, and you verify the hell out of what comes out of it. And I think there’s a lot of companies who are rushing into pushing models. The problem is that they’re not taking into account that the models themselves are ridiculously powerful. And this is where you want to – imagine that somebody can put a payload that is then executed by a model, and that model sits now in the middle of your organization, in your cloud, in your environment, who probably has access to APIs or other assets. That is ridiculously dangerous. And that’s what we’re doing.

So I think in one end, I think we need to be very careful in putting models in line, in how we actually validate the inputs and the outputs, which is kind of why I view them in multi-tier sort of flows. And on the other hand, when we use them in a safe way, they’re ridiculously powerful, because going to your first form of how to use them for cybersecurity, what I really like is that I always felt that the model for cybersecurity is a model based on the attacker making a mistake. It’s not about you protecting everything, it’s about you almost – you want the attacker to make a mistake, i.e. make a call that was not supposed to happen, make a download, make a connection, access the application in ways that no user will access it, call web services that’s completely out of sequence…

[24:18] In the past, again, it was impossible to model this. We tried squeezing technologies, even people, and we created ridiculous installations [unintelligible 00:24:24.20] They really struggled at that. But I think we now have a good chance of doing that. So that means that we can now create much more, I would say, hostile environments for attackers, because we force them to follow the paths of the users… Which, by the way, they don’t know what those paths are, unless they’re already in your system. So I think we have a chance of using that, but what we need is we need models that are really, really reliable. So OS has an amazing top 10 for applications, it has a really good top 10 for Gen AI models, LLMs. What I think about is most of that is trying to deal with the fact that the models can learn, and the models don’t have deterministic outputs.

And I like the idea of actually turning the tables around, and say “Hey, I don’t want my model to learn. I want the data that my model has access to be completely determined by the session and the state that request comes in”, which is normal app sec. And then ideally, I don’t even want the model to have knowledge. I want to give the model the knowledge that it’s going to use so I control hallucinations.

Chris, you talked about your operation. You don’t want the Gen AI doing the operation on your back to suddenly go off piste, right?

Tim Banks: …before you could get a VM approved. Before they would launch anything for you. That was the thing. There was forms for it, there was a whole way of predicting and forecasting. And when the cloud says “You don’t have to do that anymore. You can just spin up, and we’ll bill you for it later.” “Great.” Okay. So you didn’t have no dev environments that were just sitting there, hanging out, sucking up disk space. Not to the extent you do in cloud, right? You didn’t have systems that just stayed on forever. You didn’t have petabytes of data sitting out there, unsecured in a bucket because you copied it somewhere, and then now somebody got in and grabbed it, because you don’t know how to secure it. You didn’t have it like that back then. Not saying it never existed, but not to the extent it does today. Cloud has made us reckless, because it’s too easy to do things.

[00:36:21.24] There are people that have money… And then you know, when they have so much money and the money don’t really mean nothing to them, they just go and do anything with the money, and they get a little reckless with it. Cloud is kind of like that. You have, in theory, or a promise of unlimited resources… So what does it matter how much traffic something takes? How much this, how much that. And then you get the bill. And then you’re like “Wow, this bill is a lot higher than I thought it was going to be.”

Jacob DePriest: [01:11:52.19] That’s a great question. I think it goes back to what we were talking about earlier, to be honest. I think that today there is a lot of variation and freedom… Which is a good thing, to be clear. I’m not suggesting we take that away. But there’s not necessarily clear, paved paths for open source developers, hobbyists, even more corporation-backed open source efforts, to know what the best practices are for building, securing, deploying, attesting, signing… It’s complicated, right? And I’ve been in this space for a really long time, and so when I rattle some of these things off it, it may feel like “Oh, yeah, okay, cool. That’s “easy”, but it’s not. We didn’t have SLSA frameworks 10-15 years ago. The frameworks and the thinking are there, but I don’t know that as an ecosystem - and this is beyond GitHub; GitHub is part of that ecosystem - makes it really easy for people to do the right thing. So build the right way, secure it, update it, patch it, deploy it, assign it… Like, that end-to-end flow is still complicated. People use – maybe they store their source code on GitHub, but they build it somewhere else. And then when they build it somewhere else, they’re not scanning it, or that place isn’t secure. And then when they upload it, we don’t have any way to see where it came from. And then when we download it on the other side, we don’t really have a way to automatically get a sense of the risk, because it’s difficult to tie all those things together.

And so I think if I could wave a magic wand, it would be essentially to have OS partners in industry - I think GitHub’s part of this - to make those things easier for developers to just do the right thing out of the box. And then, of course, have the freedoms if they need to do something more complex or different; that’s totally fine. But I think a lot of use cases just want to know “Okay, how do I build this thing and deploy it to this cloud provider? How do I build this thing and make it show up in PyPI, and have it trusted with a little badge on it?” And I think to do that well takes a significantly higher amount of work and expertise than would be optimal if we really want to scale this.

Cameron Balahan: I know. It’s weird, right? I think you’re not the only one who wonders that. I think product management is a very role-specific and company-specific and product-specific thing generally. So maybe it’s always a bit nebulous… But the way that I try to sort of figure out what’s best for, say, the things from the prioritization roadmap, vision perspective, and also from what is the value that we’re providing to our users and to Google, is sort of thinking about that thing I mentioned at the start, which is this productive platform for production-grade software.

So we know that that sort of formula is really effective, and that both Go users and Google really get a lot of value out of that. And we look at the market generally in like a traditional product management kind of way, where we think about the product option lifecycle, we think about what’s the angle that we’re trying to solve problems here… It turns out that Go is very successful at solving cloud stuff. A lot of the cloud’s infrastructure is written in Go. The majority of the cloud’s infrastructure is written in Go, and Go has done really well there. That also means that in this new era of like a lot of different sort of supply chain attacks, that Go plays this critical role in all of this infrastructure, so security is really important. And we also know that our users are going to want that, too. So I can think about how that fits into that production-grade aspect of what I’m looking to solve.

So I think of that framework and I think “Alright, what is it that we’re considering? How does it further the goal of productivity and production-grade? How do we make sure that we create these coherent solutions out of that?” And also, does it add the value then that we want it to add, which is that our developers are able to get faster time to value, that they’re able to lower their sort of total cost of ownership over time, that their software is more reliable and secure… And these things that contribute to our users being more successful.

So knowing that sort of path between the mission, the work and the success of the customers is sort of this fundamental guideline that I can use to think about everything else. And then sort of adding to something Sameer just said, I think I’m very fortunate that the product is Go, because - well, one, there’s the users and the community who really love it. And so the thing just continues to grow on its own. The community adds to it, builds libraries around it, finds new uses for it… It just picks it up and uses it in new sort of paradigms of computing; you just see like new stuff, “Oh, it’s written in Go. That’s awesome.” So that just happens on its own.

And then internally to Google, there is the fact that Google itself really respects what Go is. I mean, Google does a lot of cool stuff, and a lot of its stuff has a lot of users, and things like that exist… But Go is really special. I think Google recognizes how special Go is, and how Go users are really happy with it. I don’t know if you’ve seen in our latest developer survey, our CSAT was like 93%, which I think is unheard of in the industry. So that’s cool, and I think everyone recognizes that everywhere. And that sort of makes things easier for me.

Autumn Nash: I have a question, though… Usually, when you want things to be more secure, there’s always that balance between security and usability. Have you ever ran into any of those type of issues? Because you’re essentially putting a cloud in the box, right? So what were your main struggles, I guess, in putting it in the box? Where did you have to make the choice of secure versus usability in that process?

Daniel Whitenack: So we started exploring LibreChat at Prediction Guard because a bunch of our customers who are using Prediction Guard wanted a private chat interface, because Prediction Guard itself is a platform that allows you to run large language models in a private secure environment, with safeguards around them for like factuality and toxicity, and prompt injections, and a bunch of other things. And so our customers are all those kind of privacy-focused, security-conscious customers who are maybe running Prediction Guard either on their own infrastructure and want a private chat interface for the models that they’re hosting with Prediction Guard, or they want an interface that’s not a closed one for usage of our models. And so here, what you can see is we’ve taken LibreChat, which again, Danny mentioned is open source, and we’ve been able to take it into our kind of branding… And we have Prediction Guard here where you can set your API key, and use Prediction Guard running on top of our platform. And because it’s open source, because it’s transparent, we are able to take this and also integrate our own sort of flair into this.

I know an engineer from our team, Ed and Danny work together, so thanks for that, where we were able to integrate some of these checks for like toxicity, and integrate our various models into the mix. So still, kind of like Danny was showing in terms of running here - I’m running with Neural-Chat 7B; this is running in a privacy-conserving setup in Intel’s AI cloud on Gaudi 2 infrastructure. So it’s a very unique setup that we’ve kind of optimized… And we’re able to connect to our own model, use this really slick interface, which is LibreChat, it’s just sort of branded a bit with our colors and logos and that sort of thing… But also, we can integrate the unique features of our take on an AI system, right? So let’s say I’m really concerned – because I’m using an open model that doesn’t have some of the guardrails around it like closed source models, I can go into the config here and turn on a toxicity filter to make sure that the model isn’t cursing me out, or giving me any sort of like stuff that I don’t want to see. And so here you can see we have a little toxicity score… Thankfully, it wasn’t very toxic this time around. So continuing… Similar to what Danny was showing, but again, our kind of own take on that, with our models, and kind of the safeguards around that.

[00:22:13.08] One cool thing that we’ve found really useful is that a lot of our customers, they want an interface like this, but they also want it authenticated, so they have their system setup… So we’ve integrated – we’re at G Suite company, so we’ve integrated Google login here… And it’s only our org that can log in, so the Prediction Guard org, and now I’m authenticated. Here’s my chat, like Danny mentioned, that is private and searchable…

So yeah, this has been a really amazing thing for us, where we’ve been able to take and build on the great open source stuff that Danny has built at LibreChat, and create something that works really well for our customers and for our setup. So before I leave and stop screen sharing, I saw that there was a question earlier on about translation with language models. A lot of what we’ve been showing is English; some language models like Open AI, they say that they’ll do other languages, but sometimes that doesn’t always work out.

So we have a translate endpoint in our API, and so we’ve done a bit of this testing with large language models translation, and kind of standard translation systems like Google Translate, and Bing Translate, and others… Or even other models, like NLLB, No Language was Left Behind from Meta. And in our translate endpoint, you can send a translation and then actually get the results along with the score. So we’re using COMET scoring, which is a way to score translations… And I think the question was how well do large language models translate and are able to chat in different languages versus machine translating with a commercial translation system.

So what we’ve seen in scoring, both commercial translation systems and large language models, is that some large language models, depending on the language - like, if you’re going into Hindi, with Open AI, you might get a good translation, or one that is comparable to Google Translate, a small amount of the time, like 5% to 10%. But mostly, the commercial translation systems are generally better. And definitely, as you go down the longer tail of languages, it gets sort of worse and worse. Even in chat in like Mandarin a lot of models don’t do so good, even though that’s kind of the next highest represented language in datasets out there. So yeah, it’s definitely a mixed bag there. I don’t know if Danny or Chris, if you have a comment on that before we go to other questions, but…

Verónica López: [unintelligible 00:47:45.06] blaming the users, and stop blaming the poor Kubernetes operator, the human operator. No, for example in Kubernetes - and we have been very open about this, so this is not a secret… But the tooling that we have behind every release is massive.

[48:03] We are a self-serving team. That means that we don’t only click buttons to release things. [laughs] We build the software behind our releases. That means - for example, the most recent and tough battle that we have been fighting is with the supply chain security stuff, that has not recently, but it has been a work in progress, that a year ago it started blocking our releases, because we wanted to sign - if you’re familiar with supply chain security and all these new terms, newish terms… It’s like signing, and promoting, and all these things. The container images are a promotor, and Kubernetes was built independently for that. [unintelligible 00:48:47.21] But obviously, we didn’t want to redo it from scratch, so we tried to build the supply chain security stuff on top, or using the existing logic of the promoter. So that meant that at some point we needed the image to be signed twice. And the signature process, that had nothing to do – it was orthogonal to the release process itself. But it was a requirement in a checklist. So if the signature didn’t work for whatever reason, or because it timed out, or whatever, then the entire release was blocked, because all the artifacts were already created. You couldn’t just click the button and try again. No, it’s like, all the artifacts and - I don’t know, from the hashes, to all those little things were already created. But the way that our tools work, they were like “This step is already done… So why are you asking me to run it again?” But it was stuck in the image signature part, and it could not move forward, because we asked it to be like that. Like, “Do not move forward if the image signature is not ready.” But we did it, so that it was a secure process, not like – not that it depended on it. So we decided to get rid temporarily of that part, because it was blocking us, and a release that would usually take, I don’t know, five hours, would suddenly take a full week, or three business days, let’s say… Between the coordination of different time zones and whatnot. And it ended up taking like a full week of work. So we were like “Okay, sorry. We have to get rid of this”, and manually do things like that.

So that is a very specific part, but there have been times where – I don’t know, where the cloud provider has a bug, or an outage, or whatever, and it’s like “Oh, sorry, we cannot run this.” This is not specific to Kubernetes, but in general, and I think that most people have dealt with this.

But then on the industry side, the most common thing for me, and where I have found a lot of fun is when people, including myself, as a team, or combo of teams, think that something is ready to be released, when it’s not. And it’s ready to be released in terms of like it passed all the tests, and it doesn’t seem to have any bugs, and it works on my computer – or it works on cluster. [laughs] That’s the new “It works on my computer”, “It works on my cluster, and it works in my series of clusters.” But then it’s like “Yes, but it won’t work with this type of user, because this type of user has something that you don’t have in your 2, 5, 7 clusters.” Because they’re way more complex than you are. And then it’s like “Oh, right.”

[51:59] So the blockage is not like an operational, or like a button, or an algorithm, or something blocking it. It’s more like a human factor saying “It will be risky to deploy this as it is, for this reason.” But you have to be very attuned to that system. It’s not like you can see it just because you’re a Kubernetes expert. It’s more – you have to have a bit of that, but it’s more about how well you know all the components of that system, to be able to say –

Steve Francis: Yeah, lots… Probably the biggest one from a company point of view is the launch of Omni, which is our SaaS for managing Talos clusters. It’s a SaaS for bare metal, which sounds kind of a oxymoron, but you bring your own servers - they can be bare metal, they can be edge servers, they can be in the cloud… You boot off an image or an ISO or an AMI or what have you, and that’s basically it. Again, using WireGuard built into the Talos kernel; those machines will register with the SaaS UI control plane side, and show up as unallocated nodes. And then you can just go through, you can use a UI to say “These are control planes, these are workers. Go create [unintelligible 01:33:26.08]

You can template all that and do it in an API-driven way if you’re doing GitOps… And it just makes cluster creation declarative, it makes a cluster upgrades, operating system upgrades, ACLs, it solves authentication problems, it ties into your enterprise SAML or other identity provider…

So that got launched in end of February, early March, and that’s done really well for us. We’ve got people running hundreds of clusters on it, and it’s a great product. Talos itself has added TPM encryption for Secure Boot… Probably a variety of other things that – KubePrism, which is great…

Jerod Santo: OpenSignLabs’ mission is to democratize the e-signing process, making it accessible and straightforward for everyone. The software currently features secure signing, a user-friendly interface, audit trails, and an API for integration into other software and services. Host it yourself (React, Node.js, MongoDB) or use their cloud hosted version.

Erik Bernhardsson: The thing that I personally spend the most time on is probably figuring out the ergonomics of the SDK itself; like, in code, how do you express programs that execute in a distributed way in the cloud, and still making it feel like intuitive and easy to the user, without having to think about the fact that this function runs in a different container than this function.

We made that work reasonably well for online inference, but I think when you go to training and start dealing with file systems, there’s certain things that are still a bit like gnarly, and I’m working a lot on that right now. So making that user experience good and sort of intuitive I think it’s really important.

On a similar note, Modal right now is somewhat janky when you run it inside notebooks, for some particular reasons; I’m not gonna get into it, but it’s something I definitely want to make – the user experience, if you’re running Modal inside a notebook, I think should obviously be… You know, we need to fix that, too. It’s fine. It’s not like terrible, but I definitely don’t think it’s quite yet where it is if you run Modal in a script.

There’s always backend stuff… We definitely need to scale this up 10x or 100x the scale we are. We see a lot of demand… Modal does not have a publicly available sign up right now. Like, you sign up and you go on a waitlist. And part of it is that we just want to have a little bit more control over the scale; there’s a lot of work we need to do on the backend to build the foundational architecture running all of this stuff. It’s a very hard problem; it’s building, essentially, our own lambdas, our own Kubernetes.

There’s a lot of work we need to do on GPU support, and in particular, cold start with GPU models, and fast-loading of GPU models. So those are some – there’s a lot of cool work, we’re spending a lot of time on there, especially when it comes to like containers, and a general like isolation of VMs. It turns out that supporting GPUs in a secure way, in a multi-tenant environment is quite hard, so we’re going very deep - I’m reading about Linux Device Drivers, and CUDA, and trying to understand all those things.

Yeah, I mean, those are all the things we’re working on. I think, in a year’s time, I think Modal will see a lot more traction for other things than just online inference. We’re gonna see a lot of people using Modal for training, we’re going to see a lot of people using Modal for parallelization… I think we’re going to have much more customers on the enterprise side; right now we’re focusing very much on the startups, but we’re laying a lot of the security and compliance work to be able to go upmarket. Yeah, those are some of the things we’re pretty excited about.