LIVE from Denver with Nora Jones!

Hello, and welcome to the lovely Oriental Theater in Denver, Colorado. Thank you so much for being here. I’m Jerod…

I’m Adam.

…and we’ve been doing the Changelog Podcast for a very long time, but we’ve never done this. So thank you for coming, this is truly a special time… And we wanted to have a special guest with us, a Denverite, and a really cool person… And luckily, we found Nora Jones. Let’s hear it for Nora.

Thank you.

Nora is the founder and CEO of Jeli, which is an incident management and learning platform. Then we had her on the show a few years back, in an episode called “Learning from Incidents.” That was February, 2022. Since that time, Nora and the Jeli team have been acquired by PagerDuty, where you now work…

Love that.

And your title’s really cool.

And long.

And long.

Wait for this.

So I’m going to read it for us all. You’re a senior director and the head of pricing and product strategy and growth.

That’s three jobs.

Yeah. Too many hats.

[laughs] So let’s start with the acquisition. So tell us about Jeli… You’re part of a much bigger organization now, but just briefly the Jeli journey, starting it, and then take us to the acquisition. I’m sure you could do that in 30 seconds.

Yeah, absolutely. So I started Jeli in 2019, and prior to that I had spent my entire career as an engineer dealing with incidents. I was working at Netflix, I worked at a company called Jet.com, which got acquired by Walmart… I worked at Slack for a bit… And I was kind of doing similar roles in each of these companies. And what I really learned through these companies was the power of post-incident reviews, and how they could really rally an organization to drive change, and focus on understanding other people in the organization’s perspective. And what I really started realizing was how much could be learned from how we speak to each other during incidents… So how we talk to each other in Slack, when there’s gaps in communication, who we bring in, why we bring them in, what they’re doing… And that can really be used to build more expertise within organizations, and really understand where discrepancies and misalignment lie in a productive way.

So Jeli was a product that we built based on real problems I was experiencing. It was really a tool I wished I had had in some of those situations… And so that was where it was born. And I’m very, very nerdy about incidents and that topic, and reliability, and I’ve been a PagerDuty user my whole career… And so the acquisition came up; it wasn’t something I was looking for, but I’ve always admired PagerDuty… And I actually was like “This could be really good for the industry, actually having these products together and interconnected.” So it made sense to me. And yeah, here we are.

How did you feel about the acquisition process? You said you didn’t plan for it… How did it come about? Did they reach out to you? Were they fans? Were they users?

Yeah, no, I mean… So the first two integrations that we built for Jeli, like before we even had customers, were Slack and PagerDuty. So we were like really tightly integrated. And the integration we had with PagerDuty was getting on-call schedules. And we were using the on-call schedules to figure out if people were actually supposed to be there during incidents, or if we were pooling in people that weren’t on-call. Because that can indicate there’s a knowledge silo, an expertise gap, if we’re constantly relying on people that the business didn’t actually plan to be there. So that was the first integration that we built with PagerDuty.

So I spoke at one of PagerDuty’s conferences in - I think it was 2020. It was like their first remote conference… And so we had always kind of been friendly, and maintained a relationship for a few years. And then they reached out and asked if like we wanted to have more serious conversations. And so I started exploring it, and it just kind of made sense. We had been doing Jeli for five years, and it was either we develop our own on-call solution and expand that way, or we join the most powerful player in the industry with that. And so it seemed like a really unique opportunity.

[00:08:12.21] When you looked at that acquisition, as a founder, you think exit, right? Because that’s a good thing. You built something, you want to profit from that, you want to see your work be recognized… Paydays are great, obviously. But how do you also think about the – one thing you said in your early part of it was how you thought about the product and what the industry could leverage from the fact that Jeli moves to PagerDuty. They’ve got a much bigger distribution than Jeli did, so it’s obviously going to be good for the industry. How do you weigh that choice between personal gain and the industry of people you know and love, and the product they need to succeed? How do you weigh the difference when you consider that acquisition?

I mean, I don’t necessarily think those are separate things, you know? Like I mentioned, I built this product because it was something I wished I had had in my jobs. And so that was really like kind of my North Star, was getting it out to everyone. And so after the acquisition, seeing the usage of something that I had built get automatically included in people’s toolkits when they start a job was what I had really dreamed about. And so I think it was mutually beneficial from that perspective.

Does Jeli as a platform still exist as part of PagerDuty today? Is it a tab, or is it a subsection?

Yeah, so it’s within PagerDuty. So one of the things – so I spent the first year helping integrate the platform, helping integrate the cultures… It’s very different bringing a 20-person company to a 1,200-person company. And some of these people had never worked at an organization bigger than 30 folks, so you know, there was some adjustment there. And then one of the things I noticed was within PagerDuty’s pricing and packaging there were parts of the incident lifecycle that were almost limited to the enterprise plan. And so they immediately put Jeli on the enterprise plan, but I started realizing there’s really more of the incident lifecycle that should go downstream, to all plans. And so I launched that project within PagerDuty, so Jeli is actually available to all plans in PagerDuty right now.

Is it called Jeli?

What’s that?

Like, do they opt into Jeli, or is it called something else now?

It is called Jeli in the user interface. It’ll probably eventually just be called post-incident reviews, but you know…

Well, you’re in charge of product, so you can change that if you want to, right? You can keep it or change it.

Just keep the name. Oh, you’re saying you’re in charge of product, so you can decide if that gets renamed.

[laughs] Yeah. I mean, I think at first it made sense to keep the name, but I think it evolves over time. I think if the name was something more relevant to what it did, it would probably make sense to keep it. But if someone’s in their PagerDuty UI and sees the word Jeli, they’re not going to know what that means.

Yeah, totally.

As much as I would hope they do.

We focus on the name, but I think it’s a proxy for this handing off of a thing that you built. And the attachment as somebody who built something, that you have for a company or for a product. And the name kind of represents like an era, or the thing, right? Jeli is the thing that you built… I ask these things because I’m always curious about the emotional attachment of a founder to the thing they built… And sometimes you have to like part with that. It’s like “I’m selling the thing, it’s gone now.” But it sounds like this is a good case, because it’s going to continue on, maybe with a different name, maybe not. But you didn’t really have to let go.

[00:11:57.22] Yeah, it’s integrated. There’s more integration that’s being worked on. It has a product manager managing it; I handed off my baby… But it felt like the right thing to do, you know? And we have a lot of plans for it too, so I’m excited to see how it evolves.

What was the response from PagerDuty customers? As you said, it became an enterprise first, and you’re like “No, it needs to be for everybody.” And then so you expanded to all the plans. So every PagerDuty customer has access to Jeli, whether you keep the same name or not… What was response to have this – I know it was already out there as a company, but you’ve obviously tapped into maybe an audience that never heard of Jeli, or heard of it less, or maybe took it less seriously… But now you’re acquired by PagerDuty, the biggest in the industry… I mean, we call it PagerDuty because it’s pager duty, but it’s also a company.

Yeah.

So what was the response like from the existing customer base of PagerDuty having this new tool?

Yeah, so when we got acquired, 90% of our customers were PagerDuty’s customers. And the very first customer I sold to with Jeli - giant enterprise company, which is very rare for startups… And so all of a sudden, most of the companies that were using us were large enterprise companies, which is a lot for a tiny startup. You’re kind of trying to act a lot bigger than you are. Someone wants to talk to the salesperson, I’m like “Let me go put on this other hat, and I will be your salesperson for a second.”

Ed Chambers?

Yeah, exactly. But yeah, and I just actually lost my train of thought.

Sorry about that.

Good job, Adam.

No, no, no. I forgot the question.

Well, you were saying putting on different hats. You were going from like founder, to sales to janitor…

Because of enterprise customers.

Oh, you were asking how customers reacted.

Yeah.

Because we shared so many of the same customers, those customers reacted really well. They were already using both of us… So it’s like a new integration, that will just make it work even better together, eventually like one piece of paper, no multiple contract cycles… And then the other thing is PagerDuty was originally created to page folks. And PagerDuty has done a lot more than that over time as well, but I think we were also filling a critical area of the platform that hadn’t been invested in much. And so a lot of customers were like “Oh, this has been on our user voice request list for years, and now, it’s finally happening.” So yeah, we’ve had a we’ve had a positive reaction.

How did it work once you were on the inside with regard to your personal drive and initiative? Because you’ve kind of been to the top of that mountain – like, starting a startup and building and scaling and getting enterprise adoption and then making a sale like that… That’s like a huge accomplishment. And now you’re inside the behemoth, right? The PagerDuty. Is it like “Well, now I’m going to climb the corporate ladder?” Or how did you think about it? Because you have a new title, I noticed, since the one that you got when you first joined. So you’ve climbed, it seems, a certain bit. But is that what you decided for your own ambition’s sake? “I now want to move up PagerDuty”, or “Now I want to have Jeli rolled out to more people”? Or how do you look at that side of growth for you?

[00:15:37.17] Yeah, I mean, I’ve been reflecting on this a lot recently, and there’s a lot of stages of grief involved in it, too… But I had been an engineer my entire career prior to starting Jeli, and so I actually had grief from being a CEO and not being in the code. I would want dopamine hits, so I’d go update our marketing site every now and then, just to feel like I was doing something, even though I was doing all these different things… But I also realized part of the reason I started Jeli was after incidents in an organization, a lot of the time every role is impacted in some way… But also, a lot of the time if there is a post-incident review, it’s just done by engineering. And they’re not going and talking to marketing, or sales, or customer support necessarily. Like, it’s usually done in a vacuum. Or even executives. And I started realizing, as an executive, as the CEO of Jeli, that I had my own sharp end. I had my own area of expertise that – you know, when I was an engineer, I always kind of felt like “Oh, executives don’t get it. They don’t get–” But then when I was an executive, I was like “Oh, I didn’t get it back then.” I was able to see a different side of the coin. And so I’m bringing all this up because now I get to be in a new role in a large organization, where I’ve worked at large organizations before, but always as an individual contributor engineer. And so from my perspective, I’m like “Oh, I’m learning another side of this coin.”

So it’s really interesting to me to have played all these different roles… So that’s kind of where I’m at right now, is like I’m learning a lot. I’m getting to run pricing at a large public company right now, and it’s fun, and it’s challenging, and I’m just kind of enjoying where I’m at right now. I’m taking it a day at a time.

So you have pricing, product, and growth.

What else is there…?

That’s three jobs, right? But I mean, you’re over it all. So you’re over three divisions, or three different silos…

Or are those just goals? What are those things? Are they goals? Are they divisions? Product, pricing, and growth. Are those mandates for you, or how does that manifest?

Yeah, good question. So the role I have is head of product strategy and growth, and I think that product strategies should really drive pricing, and drive how you organize yourself, how you ship your product… And so I’m really helping with the forward-looking roadmap, along with my colleagues, and taking input from them, and then leveraging that to drive pricing. So I do have those three different units under me, and I am wearing a lot of hats right now. I am hiring, so if you know anyone, please let me know.

And is Denver like a locale, or is it all remote and you just happen to be in Denver?

It’s all remote, I happen to be in Denver. There’s like 50 PagerDuty employees that live in Denver.

Really?

So there’s a lot of folks here.

Is there an office or anything?

There isn’t. We always say our office is the brewery. [laughs]

There you go. Let’s go meta for a second, and give you an opportunity to get a drink. I’ve seen you glance at your drink a couple of times…

Sure.

Lowercase m, meta.

Not the real Meta.

I still despise them for trying to steal one of my favorite words, meta… Grok and Meta have both been stolen by billionaires, and I’m not happy about that. In podcasting there’s an interesting psychology to it, because intellectually you know there’s going to be listeners later… And whether it’s 5, or 5,000, or 50,000, you come into it and you’re like “This is going to go out to people.” And so there’s that first five or ten minutes where you’re thinking about that… And there’s a really cool thing that happens, because in reality it’s just three people on effectively a Zoom call. And those 5 or 5,000 or 50,000 people aren’t there… And so you kind of just forget about it. And it’s neat, because the conversation becomes very relaxed, and you might forget there’s an audience… In fact, sometimes people afterwards say “Hey, I can’t believe I said that. Can we X-nay that, or whatever?” That doesn’t happen on stage. That’s my meta point right there.

[00:20:08.08] That’s the meta point.

I’m very well aware that there’s some people here…

Y’all are there, right?

We’re happy you’re here…

We can barely see you.

I’m reporting back that so far it doesn’t feel yet like just the three of us sitting in a room… I guess because we’re not [unintelligible 00:20:17.11]

In all honesty, I love this format. I’m loving it. We should do more of it. What do you think?

Yeah, I think so.

We should do more of it.

I didn’t want to be the first one to say AI…

I was going there.

Okay, go ahead.

Well, with you…

Okay.

Let me pull my notes, actually… I guess, first, let’s start with this - obviously, AI is the latest breakthrough for everybody…

He said it.

…and it’s almost a mandate for every organization to somehow figure out “How do we leverage it? How do we enable it?” And some of it has been generative, some has been simple summarization… What has been your position with being in charge of product to think about this new tech that’s burgeoning and infiltrating and welcoming? How are you thinking about it for PagerDuty?

I mean, I’m thinking about it in a lot of different ways right now. I think every single company in the world right now is trying to figure out how to use AI, how to develop policies around it… And for B2B SaaS companies, they’re trying to figure out how to monetize it.

Yeah.

And everyone’s in the early days of it, but also everyone’s kind of going through that, because that’s how the market is reacting. There’s expectations around that. I think especially within software reliability, my take is on focusing on when the agents fail, because that will be the case. And PagerDuty has been the one that you call in those situations… And so where my head’s at right now is how do we keep being that for folks, even in this new age? And how do we help them with the parts of PagerDuty that are tedious, or annoying? For example, we’re developing a shift agent that takes care of your schedules for you, because I know that is the largest complaint…

Beautiful. I love that. Shift agent.

Yeah, shift agent. So it’s things like that. I’m not a huge fan of AI that I feel like takes away the necessary human tasks, but it’s like the things that feel tedious or boring. And I don’t remember where this quote came from, but it was like “I don’t want AI to make my art for me. I want AI to do my laundry for me, so that I can make more art.” That’s kind of how I think about it as well. But I use it every day, multiple times a day, and it’s been huge.

Break: [00:23:03.03]

One thing you said recently, and this is on X… You said “The complexity of novel incidents”, which I thought was pretty cool, “will only increase.” Can you describe that?

Yeah. I mean, we are going to have incidents that we have no idea how to predict right now, and they are going to involve things that no one internally wrote. And so we also have to figure out how to debug them, and I think that is going to be challenging. I think the level of incidents we have are going to reach this rate where they’re very challenging for a little bit, we’re going to understand them a little bit more, and it’s going to normalize, but I don’t think we’ve hit that point yet. So I think we’re all kind of like bracing ourselves for it.

I even saw… Someone was using – I think it was Repl.it. And Repl.it deleted the entire codebase during an incident. It was like a new – and the agent’s apologizing, like “Oh yeah, sorry, I really shouldn’t have done that.” [laughs]

That’s bad.

That’s normally when you would fire somebody.

That’s straight up out of Silicon Valley, okay? Season six, that happened. The TV show.

Yeah.

So when we were in Seattle – when was that, May?

Yeah, May.

For Microsoft Build. There was a – I don’t know if it was a demo, or a video… There were some claims that I didn’t substantiate one way or the other, that they had agents on standby to resolve your incidents for you. Or at least to do the first level of triage on an incident, so that that pager goes off maybe later, maybe not at all. And I thought “That seems really awesome.” Like, I would not – it would be nice if nobody was on pager duty. Like, that’s a world that we all would appreciate. It’s not a reality yet, but there’s starting to be people saying that that kind of stuff is going on. From your purview inside PagerDuty, is that going on? Are there agents doing that kind of stuff?

Yeah. And I will caveat this, and just to put it simply, I think there’s a few different types of incidents. There’s the well understood, which are probably not even incidents anymore… And I’m sure all of us have worked at organizations where someone is urgently doing or fixing something that they’ve probably fixed before, and maybe there could be more automation around it. And so that’s where I think AI could really help, is like those interrupts that – and we’ve all seen it, we’ve all worked with someone where you’re like “Why are you still spending your time doing that? There’s better things for us to be doing right now.” And so I think that’s where AI can really help.

And then I think there’s partially understood incidents, where it’s like this kind of feels familiar, but not totally. I think AI can maybe assist there, but it shouldn’t take the wheel. And then with the new and novel ones, like I was talking about, that needs to be pretty human-driven. And so I think it’s just AI will really, really help with the kind of tedious, repetitive incidents and automation around them. That I think would save us all a lot of time and allow us to focus on the things that really need our focus.

Are you at a level where you’re still involved with the incidents? How much do you talk to the user, I suppose is the question? And two, can you give some details on the specifics of the novel incidents? Like, a terrible one, or just like when you’re like “Wow, that would have never been in my imagination before.”

[00:29:41.16] Yeah, so I’m not in incidents anymore, but I’m talking to our customers and our users every single day, and it’s still one of my favorite topics. I definitely do disaster tourism in our incident channel, just to see and understand… But I know better than to try to like hop in, you know…

But I think a lot of the new and novel is going to be around leveraging agents, leveraging generative AI in parts of production, in areas that we don’t understand. I think that’s really where a lot of it’s going to come up. I’m very curious about the company that had their codebase deleted… Like I would love to read that incident review when it comes out.

“Undelete that, AI. Undelete it now.”

Yes. “Undo, undo, undo.” Are there any – this could be historical as well, not necessarily new and novel, but any fun incidences that are no longer under NDA? You don’t have to name any names or anything like that, but certainly there are probably situations that have become legend amongst your teams over time, maybe internally, or maybe just like anonymously, externally, crazy stuff that’s happened. Can you think of anything? It doesn’t have to be like the craziest ever, but…

Like recent?

No, just whenever.

I mean, yeah, I have my favorite incidents…

Okay.

I always joke that when I’m old, I’m just going to be reciting incident numbers… Because 712 still sticks in my head, and 659. I remember those incidents, because I basically slept in the office that night, trying to resolve it. A lot of the incidents were trying to figure out in a very simple way how a piece of technology was supposed to work. And a lot of the times we had put the technology through the paces and almost leveraged it to do the things that it wasn’t meant to do. Consul, for example, Kafka… Those have been in some pretty hairy incidents in my experience. And a lot of the time it’s because we implemented them without having deep subject matter expertise in them. And then we also saw the power and the flexibility of some of it, and used it for things that it wasn’t necessarily meant to be used for. And then a lot of the folks that designed those things don’t work at the company anymore. So then you’re trying to figure out live what the intention was of this decision, and what the expectation was, and what’s actually happening… And those can get pretty complex.

Can you take us tactically into – I know you spent you said the first year kind of integrating… When did you begin this kind of newer role? Was it a year after you integrated and you started forming a team? Take us tactically into forming that team, strategy, rejiggering PagerDuty product, pricing… How did you rethink a lot of what was going on there?

Yeah, so when I joined PagerDuty, we had like two main product orgs. We had incident management, and then we had another product org that was doing automation, and AI, and it was like kind of focusing on other use cases… So like CS users, or FinOps, you know? So we obviously went into the incident management organization… There was no team around post incident reviews. And they did have an integration team that was in charge of a bunch of different integrations… So like ServiceNow, Datadog… That team was also in charge of the Slack integration, but it had been very underinvested in. I obviously had the team I came in with, and then I also had the chat experience team is what we called it, which was our Slack and Teams integrations.

[00:34:06.03] The very first task we had to do was – we were obviously selling Jelli standalone prior to the acquisition… The very first task we had to do was to get it into one of the existing PagerDuty plans, which was the enterprise plan. And working at a startup, I was using products that get things into skews really quickly, and I didn’t realize the complexities of it in a large organization. When that was deemed my first task, I’m like “Cool, that’ll take like 20 minutes.” That did not take 20 minutes.

Sure.

So I learned a lot about that process… And then it was also taking the existing PagerDuty Slack app and bringing all the Jeli functionality into that Slack app. So that took about a year to really fully integrate, and it’s live, so you should check it out… And then, midway through the year, that was when I kind of had that realization that our pricing wasn’t totally matching our value. So I just kept sharing it in meetings over and over again until I think I annoyed enough people that they were like “Okay, you can go do it.” And then I ended up in the head of pricing role after that.

You have a new job now.

I have a new job now. Yeah, exactly. So after launching it in all three product lines, we went through a reorg internally, and aligned our organization a bit more to the customer value, which was around six months ago… So my new pillar of the org is product strategy, growth and pricing. So I have pricing analysts under me, I have engineers working on growth, onboarding… I have folks working on insights and analytics, so really driving what are patterns we’re seeing, and how do we feed that back in so that we can help you with the well understood… And then - yeah, pricing analysts and product intelligence folks as well. So they are looking at how users are using the product, which areas they’re getting the most value out of, to really understand where we should keep investing in.

Growth at Jeli, OG Jeli was probably different than growth at PagerDuty.

Yeah. We didn’t have growth…

Can you go into some of the details of how it’s different, how it’s changed? Is it scary? Is it just way bigger? Do you have more money to spend? Bigger distribution because you have much more customers? What’s it like?

I mean, yeah, we have way more customers. At Jeli I had personal phone numbers of all of our customers. That’s not able to be the case at PagerDuty. I think customer relationships are so important, but obviously, I can’t have relationships with all of our customers now. So I also have to like use more discernment, and I try to form relationships based on where I’m seeing usage. For example, we have an incident roles function, or capability in the product… And we obfuscate a lot of data, but I could see people were creating a manual scribe role. And we actually have automation that, that does that for you, so that no one’s like typing the entire Zoom call, which is… That’s not a great use of time. Like, that would actually be a good use of AI. So I’ll reach out in those situations, and… That’s sort of really how I think about it.

I’m thinking about it from the customer relationship perspective, and I’m also thinking about it in terms of like an entire incident lifecycle, from like detecting, to triaging, to mobilizing a team, to resolving, to learning. If I see customers that are only here, I try to go talk to them. Because when you’re only in the detect and like the noise phase without having standard processes all the way over here, you’re going to end up in this hamster wheel… And you could actually drive a lot more improvement and save a lot more time.

[00:38:15.03] So that’s really how I think about it, is just in terms of customer value, which I think is where my background really helps. Like, I really love being in this role, because I’ve been a user. So it’s nice for that to come full circle.

How much code do you write?

Zero. I mean, just in my free time, but not at work. Yeah.

Is it because AI is writing it for you?

I was gonna say, how much code do I write? Way less than I wrote a couple months ago… If you set AI aside and imagine Open AI hadn’t done what they did with ChatGPT, November of ‘22, and everybody freaked out and it changed the course of the industry… What would the product roadmap look like? You’re looking at product at PagerDuty, right? Like, what – your initiatives certainly changed. I’m not sure how multi-threaded you guys are in terms of things you can work on at the same time. But everybody has to have an AI strategy, and that’s because everybody has to have an AI strategy.

Yeah.

So that drives product. We were actually joking about it last night at the meetup… It’s kind of like when one cool kid in middle school gets Air Jordans, whether they’re ugly or not, it’s like, everybody has to. What’s your Air Jordan strategy? How are you going to get the Air Jordans? And it’s like “I’ve got to figure something out. I don’t know. My friends are going to ask.” And that’s kind of AI right now, like where it’s like “What’s your strategy?” You have to have one. So that’s driving a lot of the product roadmaps, right?

Totally.

If that wasn’t something you had to think about, you would have more freedom to build kind of what you would like to, or… I mean, maybe you haven’t thought about that because you’re too busy thinking about what you have to think about, but in a world where you didn’t have that necessity, what would it look like?

I mean, I think it has driven some of our product roadmap, but it’s also made us faster in a lot of ways.

Sure.

I don’t spend a lot of time coding in my organization because that’s not my best value to the customers, or to the company, or to the folks I’m managing. But what I have found AI useful and is not just like driving the product roadmap it’s like actually aligning folks. If I have an idea or if I’ve been seeing a lot of customer data, I can mock up something really quickly, and have a conversation with our engineers, our designers, and our executives way more quickly than I could have otherwise. We don’t have to spend a lot of time in a lot of design cycles getting that initial wireframe; we can just start to visualize. And so I think it’s been really actually helpful with remote work from that perspective, especially when you can’t get in a whiteboard with someone… So I do think it’s, it’s made us faster. I think that we would be slower –

Sure.

I would certainly be slower without it. I think in terms of –

Just like Air Jordans. I mean, you run faster with the nice Jordans.

[laughs] What’s that?

I was making a joke. It was bad… Keep going.

You run faster if you have Jordans.

Yes. Yeah, exactly.

Sorry…

But yeah, in terms of the product roadmap, I don’t think – there’s never going to be a world where we have zero incidents. So I think those are some of the wrong problems to go after with AI, is like trying to get to like zero incidents. It’s more to handle the more well understood stuff, so that engineers can focus their time on things that only they can do, rather than things that agents or AI can help with. I think the whole industry is going to get it wrong in some cases. We are very new at this, and we’re all kind of like stumbling toddlers with AI right now. And so we’re going to mess up, and I think we just have to reflect on “Here’s a good use for it, and here’s a not so good use for it”, rather than trying to use it for everything. And so that’s my biggest worry, is that people are going to get overly dependent on it for things that they should not get overly dependent on it for.

[00:42:31.19] I agree with that.

Yeah. And earlier when you were talking about novel incidents, or like the really tough ones where you need that one subject matter expert to come in and connect the dots that nobody else could connect, right? Do you – I don’t want to use the word “fear” necessarily… Are you concerned that increased adoption of AI tooling reduces gradually subject matter expertise?

I’m definitely concerned about that. I think there’s ways that it increases subject matter expertise. I started Jeli before all of this was really happening, and my honest vision for Jeli was there is so much data and how you communicate that you can learn what Adam’s expertise is, and what Jerod’s expertise is just by how they’re talking. Like “Oh, we noticed we pull Adam in for every issue we have with this particular codebase.” And so we start making this map of Adam’s expertise, and we can start to download how Adam knows what he knows, and how he does what he does…

Or doesn’t.

Or doesn’t. And making that searchable. So I always wanted to put AI in Jeli, but then everything changed. So I actually think it could really enhance subject matter expertise, because it has a look into your whole organization… I think the challenge is with hallucination with that. Like, is it going to make up stuff that it’s not actually seen? And I also think that humans could stop trusting their own judgment sometimes. I’ll always test myself sometimes. Like, if I feel like I’m personally using AI too much, I will ask it a question that I know I have a lot of expertise in, just to see how it does, you know? And I feel like that helps me come back to Earth sometimes, just to make sure I’m staying grounded. But I think we all have that personal responsibility, but it’s also – it’s going to change the way we talk to each other. I think a lot about the language aspects of it, and I think about like kids that are probably learning with that now… How are they going to talk to each other in the workspace, in a certain amount of time? Anyway, those are things that I…

Yeah. To that point, there was a recent headline about changes in language, and how ChatGPT in particular is already affecting the way that we talk. For instance, the word “delve” is like skyrocketing in use, because it routinely says delve. Like “Let me delve into that for you.” And now more people are saying delve. This happens all the time in humanity. You hang out with people and you start to talk like those people, and we’re just hanging out with ChatGPT.

Right.

And so we’re going to start talking… “It’s not just X, it’s Y.” It always says that, too.

Yeah, exactly. And you know, I’ve always loved using em dashes, but I also find myself not using em dashes…

No more em dashes…

…and I’m so upset about it, because I’ve been using it for years…

Me too!

…and now I’m rethinking, “Are people going to think I wrote this with AI because I’m using an em dash?”

Right? So you purposely remove them, because you don’t want to look like this is AI-generated.

Same.

Yeah.

[00:46:02.01] Can we go into – I’m not sure how this manifests for you, but as a leader, it seems like maybe you all have some tooling that lets you glimpse into someone’s capabilities. I imagine that’s a great tool as a leader, because you want to spend one-on-ones with folks… Some of the team you know really well, because you brought them with you from Jeli, but some you’re bringing on… And as a leader, you want to know what their strengths and weaknesses are. Do you have that kind of insight? Is that what you’re talking about, where you can sort of like probe into your team to say, “Okay, Jerod’s really good at this, and Adam’s really good at that”, and you have the AI kind of like learn it for you, and you can sort of prompt whatever it might be to learn more about your team.

I haven’t been using it for that perspective, but I do – you know, as I build out my org, and I’m interviewing folks right now, and I’ve interviewed folks that are really great, and clearly experts in a certain position, but I’m also thinking about the expertise of the current team I have… And if there’s these four different areas I need expertise in with pricing, for example, if I already have someone in that bucket and I’m interviewing someone that also has the same expertise, that’s not going to maybe be valuable for me to hire that person, even though they’re great. And so I do think that that could be helpful in terms of just like creating teams that work really well together, and in looking at where someone’s strength is.

I have someone on the pricing team that loves to go into the data, data expert, but is not as proactive with talking to sales, and talking to customers… And so as I was looking for my next person, I was like “I really need someone that is doing that in the organization. I don’t necessarily need them to be a deep expert in the data, because I already have that. But I know that just because I interact with my team a lot. I think if you are running a large org, that might be helpful. I also – I use Granola, which is really, really great. I don’t know if either of you use it…

What is it called?

Granola.

Never heard of it.

It’s a note taker during meetings… And I have found that so fantastic, especially for one-on-ones. It just helps me remember them, it helps me actually be present, so that I’m not having to take notes during it… It prompts me of things to follow up on… And so I feel like it’s actually made me a better manager, because I’m able to be more present, and recall more.

How is – I hate to keep going to AI, man… But how is – I was just saying I hate to keep talking about AI, kind of, but I’m curious how AI is helping you not hire, or let go of people. Can you be candid about how it’s shrinking or growing your teams?

Yeah, and I know I probably sound like a broken record, but I really think it takes over the tasks that people were probably doing manually, that they didn’t need to be doing. So I think it increases roles, in that it is going to completely change certain jobs where someone’s entire job was manual data entry.

I spend a lot of time in like Tableau, and Periscope, and unfortunately Excel now… And when I get stuck, or break something, I actually use AI to help me fix my query. And that’s been really nice for me, in just terms of speed.

Break: [00:49:49.06]

What are you excited about, or what are you interested in, whether industry or elsewise?

What am I excited about…?

Well, she’s golfing now… She’s a golfer…

Yeah, I’ve golfed twice, so…

You’re a golfer.

I think I’m a golfer now.

Yeah, for sure. No imposters.

And you know, I was talking to Adam earlier. He was like “I mean, I just hit the ball. I don’t really golf.”

You said that?

I did…

And I really resonated with that. I can hit the ball, and it feels good… But yeah, what am I interested in? I’m really interested in how linguistics are changing with AI, and also just more remote-first cultures. We really lean on text communication now more than ever, and I think things can easily be misinterpreted over text, which actually can cause a lot of –

Sure. That’s a good response for people. In text, they would say like “Sure.”

Yeah. Or “K”.

And I’m like “What does he mean by sure?”

I say that all the time. It means yes.

When I joined PagerDuty, everyone reacted with the ACK emoji, which is a very engineering thing to do, but… You know, I had people that weren’t in engineering join PagerDuty, and they thought people were mad at them, because they’re reacting with the word ACK in Slack. And I’m like “Oh no, they’re not mad at you. They’re just giving you a thumbs up.”

Do they know there’s a thumbs up emoji they could use?

What’s that?

I said there is a thumbs up emoji they could use.

I know, exactly.

You’re using the at?

ACK, A-C-K.

Oh, ACK.

Which I think the history of that usage at PagerDuty is because you’re acknowledging an alert. And so people use it internally more than I’ve ever seen at any other organization to just be like “Okay, cool.” But they’re not upset about anything. Anyway, I’m thinking a lot about how these nuances – we all have our own perspective of how to communicate and organize, and it really impacts the organization, it really impacts how products get shipped and developed, it impacts how incidents get resolved… And so I’ve been looking more into that lately. I’ve been following this guy on Substack, Adam Aleksic… He’s a linguistics graduate from Harvard, and he’s been really just analyzing how the world is changing its communication, but I’m really interested in how that impacts organizations, and how it impacts products.

We are live before a studio audience, they are hearing this… But can you share what’s on the horizon? What’s something new for PagerDuty that you’re excited about, that you can maybe tease a little bit, or just spill the beans on?

Yeah, I mean, we are completely revamping our… I know this probably doesn’t sound as exciting for folks, but it’s very exciting to me. We’re completely revamping our pricing and packaging, so that you’re not going to have to worry about seats as much as you do. It’s a big transformation, and it’s really focused on value that we’re providing. That’s the little teaser.

And then other things I’m really excited about are seeing how Jeli evolves within PagerDuty. There are a lot of insights that we have in Jeli, and combined with the insights of PagerDuty I think can really drive change to the beginning of that incident cycle. And so I’m pretty stoked about that, and how that can help onboard new engineers, and how it can help ramp other folks up, like maybe ramp up a junior SRE because they can see how someone solved the problem. I’m excited about how it might level people up, and create this new culture of engineering. Yeah.

[00:56:23.12] Well, Nora, you are smart, kind, and fun to talk to, so we really appreciate you joining us… This has been really awesome. Thank you. Let’s hear it for Nora Jones.

Nora Jones.

Wanna do a little Q&A?

Sure.

We do have nine minutes, as I’ve been told, and he’s correct… So – we also have a microphone right here… And we have a little treat. Keychains. So the stickers are out there, free to have, but these keychains, these cost extra. To get a keychain, you have to come to this microphone and ask a thoughtful question for Nora. You can judge Nora, whether it was a good question or not, after you answer it.

Keychain-worthy or not.

So if anybody has a question, please come on up for a keychain. Please say your name, and what kind of –

Were there people that were what?

I mean, I’ll answer that as best as I can, but I think all acquisitions have challenges that come up. I think overall it was a very positive experience. I actually was – I love reading research, and I was actually reading a bunch of research on what makes a successful acquisition, and a lot of the time it’s if the founder stays. And so I also was trying to really focus on being happy myself, because I knew I was modeling that to my team, and figuring out those ways…

There were definitely people at the organization that hadn’t worked at a large company before, and it just wasn’t a fit for them, and so they moved on, and that’s totally okay. I think that makes sense in those situations. And then I think there might have been some people that were initially like “Oh…” There’s always – when you’re a 20-person company, it’s a really cool energy that you can’t replicate in a 1,200-person company. So I think there was grief involved with it from everyone’s perspective, including my own. I’ve spent a lot of time grieving not being a founder anymore, and it not being my baby…

I think the hardest thing is when I see a problem, and I’m like “That is not my responsibility anymore. I’m just going to say my words, and then move on.” But I think there were also some people that have been surprised by how much they like it, that I think will probably stay at the organization for a long time.

Thanks, Nabil. Anything else?

Thank you, sir.

That was a good question.

Good catch. Oh. Too soon. I thought you had it.

James.

Yeah, it’s a good question. I don’t think that the right approach after a gnarly incident is to set up automation. I think the right approach organizationally after a gnarly incident is to talk about it, a lot, and try to figure out how it happened. Not why it happened, but how. And have these productive conversations where we’re all sharing our piece of how we thought things were supposed to happen, so that we can learn from each other, and so we can learn about how this actually unfolded. I think until you have that understanding, automation is going to cause issues, to your point. I’ve been a big believer in that for a while.

When we were at Jeli, myself and Laura McGuire and [unintelligible 01:01:20.14] we wrote what we called the “How We” guide, which stood for the “How We Got Here Incident Analysis Guide, which is free online, but it talks about how to extract the most learnings after incidents, so that you’re better prepared in the future. And I didn’t mention this aspect of right when I came in, but one of the most important things to me when I came in was that we were using Jeli internally, and also practicing what we were preaching. So we use Jeli for every single incident now, and we are looking at how we communicate during those incidents, which is driving what we do or do not change.

I think with some incidents you might not need an action item at the end of it. It doesn’t mean that you didn’t learn from it, but the action that you take might be understanding your system more. So I think the best thing that we can do as PagerDuty is model that for folks, so that we can help them do it themselves.

Thanks, James.

Thanks, James.

There you go, James.

Hi.

Say that one more time.

And go do another company?

Are you itching to leave and do something else? When will you get the itch?

I am having fun right now. I’m doing things that are challenging to me, and I had never thought I would be doing pricing… But it is activating both parts of my brain that I really love to activate, which is psychology and math. And it’s very much an art and not a science, which is what I really – that’s where I really thrive and gravitate towards. So I think as long as I’m learning and feeling like I’m adding value, and feeling like I can make a difference, I don’t think I will get an itch. But I’m also playing with different technologies that I can’t necessarily always play with at work, and I… You know, I haven’t found something that really captures my interest enough yet. But yeah, I’m having a lot of fun, so…

Thanks, Kendall. We’ve got three more.

Yeah, good question. So like I mentioned, I have been an engineer most of my career, and it was within the reliability community. So most of my network was in reliability. And so I was really building out in the open with people I knew. And I also previously worked for Netflix, I was working for a company that got acquired by Walmart, and I was working for Slack… And so I really only knew enterprises. Looking back, I thought it was kind of an accident, but it was like “Oh, I was designing for something I needed in my organizations.” And so I think that’s just how it happened. And as we were building out in the open, we were looking for design partners, and we had slots for five. And I tried to get five very different types of customers, because I wanted to figure out who was finding the most value out of it… We did one with a startup, and it went horribly, actually. And then it went really well with this enterprise. And I remember talking to our investors about it, because if you – if you have an investor and you’re telling them that the first customer you’re signing as an enterprise customer, they’re gonna try to talk you out of it, because they don’t think you’re ready for it yet. And it was way more organic for me to build for enterprise, rather than to build for startups.

Thank you very much for that question.

Yeah, thank you.

Yes. Matt…

So if I’m understanding your question correctly, how can customers that are using PagerDuty best set it up so that they’re creating a learning environment?

Yeah. So one of the things I really appreciate about PagerDuty is how they’ve developed their integration system. We don’t expect customers to use us for everything, but you do get the most value out of it if you’re connecting it to all the other tools that you’re using. There’s over 700 integrations, and you can also build your own… And I think that is the best way to get the value out of it, because then we can understand all the signals that you might be seeing, and really help you prioritize if something is a really urgent signal that deserves human attention, versus not.

I think that’s where I see a setup go awry, is when folks don’t spend the time to set it up that way, and then it just becomes more noisy than it needs to be. But if you are really taking a look at your overall company, and being like “We use this product here, we use this product here”, and really connecting it to PagerDuty, it’s going to be a lot more valuable of a system that is not just learning from all that, but it’s also surfacing a lot of that to you as well.

Yeah, so…

Can you restate that question, please?

Yeah, so the question was “Do you see that just for engineering, or are you exposing it to everybody?” So that is something that was always on my mind with Jeli, is just like not keeping these learnings in a vacuum with just engineering. And it is challenging. It is not an issue a tool can totally solve. It has to be a cultural change. It has to be an engineer going and talking to the person that launched a campaign, and learning more about their world. And I think we really try to facilitate that. We show you all the people that were involved in an incident, even people that were just lurking… Which can be really interesting sometimes. Like “Hey, why did legal hop on to lurk?” Go DM that legal person and ask “Hey, I’m just reevaluating this incident. I’d love to know what your interest was in it.”

And I think what we really try to provide with the platform is a way to make that less awkward… Because it’s like you’re just looking at the data, and the data points say that these different folks joined. So it’s worthwhile to talk to them to get this holistic view of what really happened. And so I think that’s how we’re really facilitating it.

That’s cool. Thank you, Matt. Next?

I really started – I actually started my career in hardware. I studied computer engineering and I went directly to home security in a smart home company. It’s terrifying having that as your first job, as like a reliability person, because if you mess up, someone’s house might get robbed. So that was stressful.

And then I saw this job opening at an e-commerce company for a software engineer. That wasn’t technically my role. I was doing less web development and doing more hardware… But I looked at some of the traits that were needed. It was like a developer productivity and reliability role, and I was like “I do a bunch of this now”, and so I just applied. And I joined the org, and they were having incidents multiple times a day, every day. The company was growing so quickly that there wasn’t even a place to do our incident reviews, so we used to do them in the hallway every morning… And it was like the hallway next to the bathroom, so people were going like in and out of it… It was bananas.

And I was really trying to get creative. Like, my boss at that point was like “Do whatever you want. I don’t care. We can just try things.” And I found what Netflix was doing, which was called chaos engineering. And I started reading a lot about it, and I was like “Wow, this is like really, really interesting”, because there is a big organizational psychology to reliability, too. It’s not something that can just be fixed by fixing code.

And so I started experimenting with it, and I was really influenced by Netflix… I started – actually, I stumbled into the field guide for human error, which is in the aviation space… And so all of those things I think were really fundamental to my passion now, and what I ended up following to learn more about it. And someone brought up other industries earlier… I also read post-incident reports from other industries as well, because I find it very fascinating… And it is relevant a lot of the time, too.

Is your book still for sale?

Is my what?

Your book.

Yes, it is. Yeah.

Go buy the book.

There you go.

We’ll link to it in the show notes.

Let’s hear it one more time, Nora Jones.

Nora Jones, y’all.