Eugenio Pace: Well, there’s a couple data points there. One is at that time there were a couple companies that proved to me and to us that a business could be built around this notion of empowering developers to build better, faster, in our case more secure software, even with arguably boring things. So there’s nothing really very sexy about authentication. Identity geeks and engineers - some of them might be super-passionate about that specific problem, but it’s not something that everybody/the entire world wakes up every morning and says “Oh yeah, let’s learn how cryptography works, and digital signatures work.” We have other problems to solve.

But at that time, there were companies like Twilio and Stripe and SendGrid and Heroku, just to name four that were role models for me, that were taking arguably boring infrastructure requirements and making them easy to use and accessible for any developer in the world.

SendGrid solved the problem of sending emails. A pretty mundane thing; a pretty important thing, but not like an earth-shattering problem. Twilio was solving the problem of sending SMS messages, and doing that in a way that any developer on the planet could SMS-enable their applications. And they were very successful. So we kind of said “We are going to be the Twilio of authentication. We’re gonna be the Heroku of authentication, or the SendGrid of authentication.”

Derek Collison: [27:58] Yes, it can, but again, this is one where I think people still equate messaging systems with a single broker, and queuing, and back from the ‘90s type stuff… NATS is extremely good at routing and framing, and that’s it. But what is interesting now is that NATS is extremely lightweight, so a server can run anywhere. It can run on your phone, or whatever… And you can Lego-brick these together into any topology that you want. So if you think of it that way and you don’t think “Oh, I’ve gotta go through this queue”, where it’s “I’m just routing a request to the appropriate responder and getting the response back”, even just with that simple model, you’ve removed a lot of moving pieces in the transition from dev to production.

The other thing that’s really interesting to us - and again, this is where microservices has driven a lot of this, I think… All of a sudden with microservices it’s like “Wow, I’m doing point-to-point”, which you can argue for or against; I’m against, but I understand why people say it… But what people are interestingly enough struggling with is addressing a discovery in security. Everyone has their own security. And when someone raises their hand in the org and says “We can create a one org security model type stuff…”, and it’s usually painful; it’s hard to do that, and it affects the developers themselves, and you don’t have that really clean decoupling. NATS tries to preserve that.

The program that I write on day one, that goes into production, literally has one thing that changes, which is “What are my credentials?” So you say “NATS connect” and you give it a URL to a system. We have a global system that runs all over the world - every major cloud provider, every major geo. All you need is a single URL and it just works. We find the closest server, we do all the right stuff for you. But the only thing to go from dev to prod is “Hey, we need credentials to prove who we are.” But now, all of a sudden you have a consistent identity authentication and authorization system that has no private keys or passwords ever shared with the system itself, and it just kind of works. In other words, that pain point of “Oh crap, now we’ve gotta get secure, and locked down, and stuff” also goes away.

A lot of times, when you’re playing around, hobbying out, doing stuff, you don’t see that, especially for enterprise, or playing around with services that are usually exposed over the web. Where we have seen people immediately jump on something like this is IoT, and the IoT landscape, where people are like “Crap! We don’t really have anything, and the thing we have really doesn’t have a cohesive server back-end.” In other words, there’s a lot of decisions. If you’ve got an MQTT client on your device, but you don’t know what to do.

We haven’t landed it yet, but we’ve committed to it, promised it, and we’ve been coding on it… This notion that “Hey, you can just take those apps and connect them to a NATS server, too.” And this is a global topology, meaning your IoT stuff will work anywhere in the world. You can sell your gadget, your software to anyone, wherever they are in the world, and they’ll get a good experience, so to speak.

Steve Tuck: Yeah, your identity provider for auth. So again, in enterprise environments you’ve got usually a SAML-based auth environment, whether it’s Keycloak, or some larger, more unwieldy Microsoft products… And we were not gonna go try to replicate all of that. These are established authentication and identity validation mechanisms… And so integrating into that so that you have kind of a pretty clean workflow for being able to get that stitched together.

Now you’re the administrator, so what you are doing is setting up a silo, and that is kind of a boundary for – because one of the other important aspects of this is being able to operate in multitenancy. I know multitenancy gets thrown around a lot, but the necessities of having both delivering kind of quality of service guarantees to customers while having complete isolation is one of the very complicated and hard elements of running a cloud, and something that has been very difficult for extant systems providers to get right who are selling on-prem. Even some of the kind of hyper-converged folks that entered the market in the last 5-10 years, this notion of multi-tenancy is a pretty tricky one to get right. But in the Oxide system you’re basically setting up a silo or a number of silos, depending on your customers that you’re serving.

So you, Adam, have like two different departments, and you would have those departments in their own kind of boundary. And then it’s as simple as inviting them in, and those users can then come in, just like they’re hitting EC2 or AWS; they can set up their credentials and create a project, and they’re off and running. They can go deploy instances directly, they can do it via the API CLI…

Florian Forster: Yeah, I mean, it’s such a huge – authentication and identity space, in general, it has a huge depthness to it. And it always feels like you can easily do that. But as soon as you start poking into the space, you will see there is like a huge amount of time flowing into it. I mean, the OAuth threat framework is like 60 pages. I recently did something for the Swiss government, which will come out in a few months… It’s like 120 pages on just things you should consider when building something like that… And it starts with things like XSS, and CSPs, and… You need to care for that. It’s just – the depth is the problem, basically.

Phil Hawksworth: Yeah, and I absolutely think there are… But that moment of authenticating - that’s the perfect place to start to think about using JavaScript and APIs. And again, there are these services now that we don’t have to roll our own; there are identity services, authentication providers that you can use, with things like JavaScript web tokens, and different authentication methods that can then unlock either access to routes to things that have been pre-generated - that’s a common path… Because sometimes you and I might visit a URL and after authenticating get things that are personalized just to us. And maybe those have been generated at request time, or embellished at request time through some kind of progressive enhancement.

But you might equally find that you and I might visit the same URL, which is private, and maybe has targeted content for us after we’ve logged in, rather than very individual content… So in that case you can start segmenting the content, pre-generating that, and then all you’re doing - and I should be careful saying “all you’re doing” and using words like “just”… But what’s happening there is you’re doing the authentication that then allows people access to the URLs which would be pre-generated for them.

Matt Moore: That’s a great question. I would say signing is the beginning, it’s not the end goal. And I think that it would behoove most organizations to start looking at how you start signing things now, because it’s really a foundational thing. And if you can get to the point where you have that foundation, you can then start to do really interesting things with that, right? You sign things like provenance to say, “I produced this”, so think signed commits, but it goes so far beyond that once you start to talk about things like attestations, and all the other kinds of useful pieces of metadata you might sign with. And I think to touch on and tease Nforce a little bit, Nforce is really about the complimentary side of signing, which is, “I’ve now signed all my stuff. Why did I do that”, right? And it’s so you can actually have policy around the types of signatures you want to allow into different contexts, and things like that. Now you are really signing like a form of authentication. You’re saying, “This is my identity”, since now we have that identity element to this, right? And I am making some sort of statement about this thing that I have signed, right? So now what identities do you trust, in what context, to say what things, right? So these are the kinds of things where policy gets involved, and where you can actually start to leverage that metadata that we’re trying to make incredibly easy to attach, to actually then make judgment calls about these things. So yeah.

Steve Francis: Yeah, lots… Probably the biggest one from a company point of view is the launch of Omni, which is our SaaS for managing Talos clusters. It’s a SaaS for bare metal, which sounds kind of a oxymoron, but you bring your own servers - they can be bare metal, they can be edge servers, they can be in the cloud… You boot off an image or an ISO or an AMI or what have you, and that’s basically it. Again, using WireGuard built into the Talos kernel; those machines will register with the SaaS UI control plane side, and show up as unallocated nodes. And then you can just go through, you can use a UI to say “These are control planes, these are workers. Go create [unintelligible 01:33:26.08]

You can template all that and do it in an API-driven way if you’re doing GitOps… And it just makes cluster creation declarative, it makes a cluster upgrades, operating system upgrades, ACLs, it solves authentication problems, it ties into your enterprise SAML or other identity provider…

So that got launched in end of February, early March, and that’s done really well for us. We’ve got people running hundreds of clusters on it, and it’s a great product. Talos itself has added TPM encryption for Secure Boot… Probably a variety of other things that – KubePrism, which is great…

Michael Quigley: So zero trust started with this – I mean, I’m exactly sure when it started, but it sort of became popular with this concept that came out at Google called BeyondCorp. So they started this big initiative, and I don’t remember the exact timeframe. I want to say it might have been maybe – let’s call it a decade ago, or something like that… Where they had this BeyondCorp initiative, and it was about not considering the difference between LANs and WANs. Kind of like where you are in terms of physical security being related to how your network security is postured. So a lot of times a while back we would have configured access to things by “I’m in a corporate office” or “I’m in some physical location where I’ve got a specific network access to something.” Zero trust was sort of about getting rid of that concept, and going to “It doesn’t really matter where you’re physically located and what your underlying network is”, and treating the security posture differently than that. So using certificate authentication, and crypto, and things to strongly identify whoever’s using your network resources; like, actually securing them that way. So that was sort of the genesis of the zero trust concept.

Michael Grinich: Yeah, single sign-on is the first one. And specifically, authenticating with SAML-based identity systems. So the things you were mentioning around signing in with GitHub, or signing in with your Google account - these are often kind of social login experiences, or individual logins… So just kind of click your GitHub account, and then you get added to the team. But if you go sell to – I’m trying to find an example… Like Uber, for example; they have an identity system internally that they’re using as their corporate login, and they want to manage who has access to what, and who’s signing in from which device, essentially. There’s a lot of different things that these enterprise identity systems allow you to configure and manage.

And so they’ll say, “Hey, we don’t want you to sign in through GitHub anymore. We want you to sign in through Okta. Or like Azure AD, Azure Active Directory. Or this custom-built SAML system that we have. Or this thing from VMware, or CyberArk.” There’s all these different systems out there. But usually, the way people log in is kind of the first enterprise requirement that you’ve got to add.

Around that time also, sometimes they’ll ask about kind of compliance and security at the company. People often adopt WorkOS around the same time they start thinking about SOC2 compliance. So we don’t help companies with SOC2 compliance. There’s a bunch of products out there, like Vanta, or Drata, or Secureframe, which do this really well… But it’s around the same time. So usually, companies need to start thinking about their security policy, and positioning. Usually they don’t need to get it done-done, because it takes about a year for the compliance stuff to go through, but they’ll start asking for that, too.

And then the third thing companies will ask for usually is a thing for user management - automatic provisioning and deep provisioning. Because if you can imagine a company like Uber, that has hundreds, or I guess thousands of employees now, they’re not going to manually add or remove users. They want to connect it with their directory system, so when someone gets hired, they get access, when they leave the company, it’s removed… So they need to tie it into their directory system. And that could be like a payroll system, or some user database somewhere… And so a lot of the stuff is around connecting to their systems and really creating automation for companies. And that results in a more secure platform, and easier for them to manage.

Those are just a few of them. If you want to see the whole list, it’s actually pretty easy to figure out; just go look at what’s in Slack Enterprise, what’s in Dropbox Enterprise, what’s in Asana Enterprise. You’ll look at those bullet points on the pricing page, and they’ll pretty much be the same across different products. And those are all the things we’re building. That’s our roadmap. If you just go look at what do you get in Asana Enterprise, those are things that we hope to build, so developers don’t have to build them themselves. We can just put them into a platform.

Rob Barnes: I just wanted to kind of take it back to basics, just to kind of paint the context here. So when we talk about these credentials… So in your secrets management platform, you’re storing API keys, you’re storing database credentials, whatever it is, right? What did these credentials represent? And that’s the thing that we always need to get our head around - they represent an application identity. So if you have a specific service that needs to read and write data from a certain table or something in a database - well, the credentials that are used to authenticate to that database there represent that application’s identity. This is the payments application, for example.

So when you start to share that credential with other services that maybe need to read and write – I’m no database designer or anything like that, so maybe this is a bad pattern in terms of database design… But if you have another application that needs to read and write data, maybe to a different table within that database, it’s using the same application identity as the other one, right? So you’re sharing identity. That, in my opinion, is an anti-pattern, right? I think you’re always going to want to have the applications have an identity of their own.

A lot of the major cloud providers as well are putting in features that allow you to assign identities to workloads, because they understand this principle here. And essentially, how you implement your identity and access management strategy is always going to be using sort of the principle of least privilege. I just gave this example, the payments application needs to read and write data from a specific table in this database here. So that’s the exact permissions it should have. Nothing more, nothing less. If something else needs to do something with a different set of tables, then whatever permissions are required to perform that function, that’s what it should have. Nothing more, nothing less.

So we come back to the core constructed application, the identity of that - how do we test the identity of that, and then we assign the identity once we can make that attestation.

Dan Moore: Yeah. So I’ve been, for the last about four years, I’ve been working for an auth provider called Fusion Auth, and I’ve done a variety of roles there, and spent a lot of time talking to customers about how to implement auth, a lot of educational content… And when I say auth, it’s authentication, authorization, and user management. There are some other aspects of the authentication or user lifecycle that we don’t really focus on, like identity verification, or workforce-oriented stuff… We’re much more focused on customer identity access management. So that’s my expertise.

And I learned about OAuth, and SAML, and OIDC, and JOTS… Basically alphabet soup in terms of jargon. But I spent a lot of time decoding that and taking it – rewriting it, or rewriting my understanding in such a way that developers would actually be able to apply it in their day-to-day life.

Jacob DePriest: Yeah, I mean I think if we were just looking at the codebases and assumed that all of it was sitting unprotected on the internet, it would feel and likely be futile, to be honest. And I think this is where the rest of mature security programs come into place, and things that we can do as an industry. So I think that that’s where like zero trust and identity as a perimeter and those kinds of concepts, secure by design come into play. So if we have strong authentication in front of access to a lot of these systems, if we have network isolation between key systems, if we have role-based access control… So we kind of assume that parts of these systems will eventually experience some sort of security issue, how do we firewall those off from other parts of the system? I think this is where the rest of that comes into play. And I also think the other element to this is I think the industry does need to level this up.

[00:23:58.09] So the Secure by Design pledge that was announced in many companies, including GitHub, signed that RSA a few weeks ago, talks about this. It talks about needing increased commitment from key players in the industry to implement secure by design principles as part – not just as part of their internal programs, but as part of the products they offer to the world and to users, so that the settings that make things more secure are on by default, even if it causes a little more friction, or one more click for a user. And I think that’s really an important part of this, that honestly we do have to progress as an industry here, and I think it’s critical that that’s the other element of responsibility that companies, corporations and organizations take in this space.

Deepak Prabhakara: Yeah, so there’s always two aspects to it. One is more infrastructural, in the sense that what is the foundation there, right? Can you give folks the building blocks to say “You can now build, for example, an authentication policy around this”? So in our case, we do that with just purely enterprise single sign on, which is a way to get into any kind of system on the back of well-known established identity providers, like Okta, Azure Active Directory… And just to even do that, you kind of have to decide for all the protocols in between, how they talk to each other. There’s massive amounts of security vulnerabilities on the back of that simply, because of the protocols involved, and not necessarily knowing how to do it right. So even within authentication, to get authentication right is a lot of work. You can make things work, but then are you doing the right things in terms of being as secure as you can? Of course, there’s nothing like 100% security, but as much as you can.

Nara Kasbergen: That’s a great question… For Alexa, right now the only thing that you can do is use OAuth, and it requires someone to have the Alexa Companion app installed on their phone… And it’s very kind of ClearGate, where basically once you say “Okay, enable authentication”, the only thing that the user can do once they install or activate your skill is they have to go to that app and log in before they’re able to interact with it in any way at all… Which is not great for user experience, so I generally recommend to people not to use that authentication unless they absolutely have to. If you’re a banking app, then it probably makes sense, but for everyone else, I would say – we’ve seen an enormous drop-off with users; they’re not engaging with the Alexa skill if they don’t have an account already, if they don’t know what they’re getting into, just because there’s no way to provide a preview of what they’re gonna get.

Google is a lot more flexible, and I think it also makes sense because they are an identity provider. At this year’s I/O they announced a new feature, where you can actually log in via voice if you have a Google account already, and that Google account already exists on the other OAuth server. Or you can even configure it so that if it doesn’t exist, it just creates an account automatically… So it’s what they (or at least what we) call frictionless login, essentially.. So that doesn’t require you to take out the companion app, and that really helps. They’re also a little bit more flexible about – you can provide some kind of experience where they can at least start interacting with your skill without having to be logged in, and I think that helps a lot, as well.

Kris Brandow: I think part of it also comes from the fact that your backend code is within your security domain, and your frontend code is not, and you kind of have to treat your frontend code as if it’s hostile, in all cases. So I think that kind of lowers the necessity in some cases of needing to be as security-conscious about what’s going on in your frontend. And most of the time, a lot of the frontend vulnerabilities, like cross-site scripting, can be blocked or prevented on the backend side of things as well. But I do think the kind of lack of security awareness for frontend people, especially lack of cryptographic awareness, does hinder in some ways. I’ve noticed that there’s not a huge amount of these, of the cryptography APIs in the browser, where that could be a much better way to do authentication and authorization. Because the browsers had APIs for years now where you can just create a public-private key pair that is locked into the browser, where another script, even on your subdomain, can’t extract that key. And that would allow you to kind of uniquely identify that browser in a cryptographically secure way, and you’re not passing around tokens that can be lifted, kind of no matter what, and someone can run some JavaScript on your website.

[00:46:07.26] So it’s like, yeah, there’s a little bit you can do through cross-site scripting. You can still make API requests, but you can’t lift that key and send it to a server and start making a ton of requests, or anything that. Whereas you can do that with a [unintelligible 00:46:18.05] token. And I think some of the reason that’s not done is because there isn’t as much of a familiarity with these tools and these ideas in the frontend community, so there’s not a lot of support for this type of stuff. I mean, we’re starting to get it with things like passkeys, but that is super-late compared to what we could have had if we tried to start doing this much earlier.

Denise Yu: Yeah, it was really – it was hard to find those people. We didn’t do as much community outreach as I would have liked to. We just kind of ran out of time on that… But we did spend a long time going through the CFP.

Oh, actually - I wrote an application for it… [laughter] I was out of work at the time, and I didn’t like how PaperCall – like, no offense if anyone from PapelCall is listening to this, but I didn’t like PaperCall’s built-in mechanism for sorting through submissions… Because my requirements were I want multiple reviewers, and PaperCall caps you at five people… But I was like “Well, we have seven people, and possibly more, if we decide to bring on guest reviewers.” So I want an unlimited number of reviewers.

[28:22] I want the author, the organization and identify details to be not shown by default, and I also want the ability to edit the content of the abstract, in case someone said “Oh, by the way, I’m blah-blah-blah, and I work for blah-blah-blah” just in the abstract itself. PaperCall won’t do that for you; PaperCall just hides the name and the email something like that.

So I actually built an app in Node.js, from the ground-up, that you export all your JSON from PaperCall and you just put it into the app, and it displays every single talk as it’s own – like a submission form. I totally [unintelligible 00:28:55.11] the login process. I put it on Heroku, and I was thinking – I basically had a one-person hackathon for four days to crank this out… And the last night, I was like “Ohh, authentication… I really, really don’t want to think about OAuth right now. I really don’t wanna build a Twitter login, or something like that…” So what I did was I just put a map of keys and values into the Heroku app environment… [laughter] That was people’s usernames and passwords.

Marc Boorshtein: Right. And this is the second iteration. So they wanted to move to SharePoint Online. When you do SSO – so Azure has this concept of B2B, company to company, doing some kind of SSO, and B2C, company to customer. Well, we’re an SSO system, right? We’re an identity aggregation system. So in theory, what we would want - excuse me, you go to SharePoint Online, and they bounce you over to us to authenticate, and then we route you over to the right thing. That’s a problem on two layers. The first layer is if you’re a tenant inside of Azure, they just go straight to you. So I put in my FairfaxCounty.gov email address, they just bypass our system. IAMS is what it’s called. The National Capital Region IAMS program. Identity and Access Management. So that was the first issue, which wasn’t that big of a deal, because we still had stuff to do with provisioning.

[00:54:08.20] The second issue is the U.S. federal government. So I mentioned that we’re running in a commercial tenant. So is the SharePoint. Feds do not run in commercial tenants, and you cannot invite a guest into your tenant if you’re commercial.

That is government. So we had two issues that we had to deal with. One, we don’t just maintain a broker with other identity systems, we also maintain what are called external accounts. So maybe you’re a contractor, maybe you’re working for an NGO, a non-governmental organization, but that works with governments, that you aren’t part of our SSO party. So we maintain a credential for you. And so we have those credentials as well.

Now, we integrate with the federal government, so we can authenticate any U.S. federal user. So the question was how do we get Azure to recognize us as an identity provider? The other thing that we ran into that’s a technical issue is you can say to Microsoft “Hey, for tremolosecurity.com, go SAML to them.” The problem is, is that we have to prove that we own tremolosecurity.com, because they don’t want somebody proxy attacking. And I get that. But they also have no mechanism to work around that. Like, they have very specific exceptions… Okta, as an example, can get around this.

So what we ended up having to do was for U.S. federal users and these external users that don’t have jurisdiction accounts, we had to create an email domain alias that would be recognized. So we run an email service… We had to completely re-architect everything to make sure it was isolated. But we run an email forwarding service where you have to have an email, otherwise it gets rejected. So you have to know the email of the person that you’re sending it to. It has to exist. It’ll forward it to them, so now when we register the invitation with Microsoft, it’s at our domain, instead of their home domain. So they’re still using their home domain credentials, but they’re using our email forwarding service. So we don’t maintain email boxes. We have to forward it. But that was a whole other bit of fun, because we were originally doing it with just straight up SMTP, and it worked fine when we first went live, and then three weeks later people stopped getting emails, and we found out that basically if any email comes from any of the three major public clouds, it gets blocked by everybody. So if it comes out of Azure, it comes out of Google, if it comes out of AWS - it doesn’t matter, it’s getting blocked.

So we ended up going to AWS and saying “Okay, well, we’ll forward everything to SES”, and that solved the problem, because now we were able to rewrite it with all the right headers and everything, and then it went up to SES.