Leading the charge on AI in National Security

Welcome to another episode of Practical AI. This is Daniel Whitenack. I’m CEO and founder at Prediction Guard, and I’m joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

Doing good. It’s a beautiful, almost spring day here in Atlanta. I walked my dog before the show…

Yeah, the sun is out.

It’s getting nicer out. I have to start with the weather, of course.

Yeah, and spring is breathing sort of new life into various ideas around this podcast as well… And before we hop into our really interesting guest interview today, I wanted to highlight something new that we’re trying at Practical AI. We want the show to be practical and useful, and part of that is actually helping you all get live, hands-on help with the latest technologies… And so March 14th, at 1pm Eastern, we’re gonna have our first Gen AI mastery class or webinar or tutorial, or whatever you want to call it. We’re gonna dive into all things text to SQL, data analytics questions with large language models, and we’re going to be joined by Chang She, who is from LanceDB, the CEO there, who was a recent guests on the show. So it’s going to be a lot of fun. Find out more at tinyurl.com/genai-mastery1, and we’ll include the link in the show notes as well. I’m pretty excited about it, Chris.

I am, too. Looking forward to that. It’s a new feature for us to do, and we hope folks in the audience enjoy that. I guess if it’s okay, I’m gonna dive right into our guest.

Go for it.

I will say ahead of time I have been looking forward for years for having this guest come on the show…

It’s a long time coming.

A long time coming, because I’m at Lockheed Martin and we have a lot of “Don’t cross the streams mentality”, I had to wait for this guest to retire from his previous position, so that there can be no crossing the streams concerns. And so I’d like to introduce Jack Shanahan. He is the only senior military officer who has been responsible for standing up and leading two organizations within the United States Department of Defense dedicated to fielding AI capabilities. One of those was project Maven, which was also known as the algorithmic warfare cross-functional team, and the other is the Department of Defense Joint Artificial Intelligence Center. And he was the founding and inaugural director of that. Jack, welcome to the show.

Chris and Daniel, thank you so much for having me on. I just want to – I’ll begin by just saying how much I love the title, Practical AI… Because I am a practitioner. This is not about research and development. Now, there’s lots of wonderful things happening in the AI R&D world, but that’s not my world. My world is how do you take this esoteric technology and turn it into a real product for the government, or for my case, for the Department of Defense? And a little bit for the national intelligence community. Because project Maven - you talk about crossing streams; we cross streams. It was both what we say the IC, the Intel Community, and the Department of Defense. But I’m really happy to join both of you for this podcast. Thank you

No problem. I will note, I follow you on LinkedIn, I encourage our audience to follow you on LinkedIn… You’re out there reading the scientific papers as they come out, and you often, when you’re sharing posts, will put a spin on the various papers that are coming out from that AI and national security context… Because those papers obviously aren’t necessarily in that way. Very insightful. I would encourage people to do that.

I wanted to start out - as a military professional, how did you get into this particular… I mean, you were the right person at the right place at the right time in the Department of Defense… How did you end up in this? What caught your attention and how did you get going on this story?

Yeah, I guess what I would begin by saying is I spent 95% of my career not working on this, because I started in actually fighter aviation for the first 15 years of my military career. I spent 36 years in uniform, retired in the summer of 2020, and then from the flying piece, I was charged to go run an intelligence organization. And then I did a

command and control wing, and then a flying wing out in Nebraska. And then did a command down in San Antonio, which was about intelligence surveillance, reconnaissance and policy positions and other positions in the Pentagon. So this was not my destiny, my path, as we say, from the beginning. However, when we got to that point where I came back as a three-star into the Pentagon, in something we call the Undersecretary of Defense for intelligence, I didn’t walk into that job with AI on my plate, as we say. I left that job with AI dominating everything I did, every single waking moment, in most of my non-awake moments; nightmares, dreams, whatever you want to call it. As I say, I cut a long story very, very short by saying we had an intractable problem. And we had intelligence analysts looking at full-motion video coming from drones. And there was more drone video than at any point in history, hands down. The intel analysts just couldn’t do it; they couldn’t spend enough time in a day.

[00:05:57.11] As I used to joke about at the time, they would have some sort of energy drink next to them, and probably some chewing tobacco, trying to just – this mind-numbing work of looking through this video. It was just – first of all, we’d kind of run out of people. But two, we were gonna get more and more collection; not just full-motion video, but from other assets, as we say, to include unclassified open source information. And it was a success catastrophe. More collection, from more places, at more classification levels than at any other point in history, period. So what could we do? And we could not find a technology that was ready to be fielded in the Department of Defense. Again, wonderful research and development work that was going on and goes on as we speak today… So we turned to commercial industry. In fact, we turned to Silicon Valley, and they said “Yeah, we’ve got something that could probably help you. It’s called Computer Vision and natural language processing”, and that’s where the journey began, to be honest with you.

We formally stood up in April of 20017. It’s hard to believe it’s been seven years now on that journey. And then we were off to the races… And I did that for just about 18 months, and then I was asked to stand up the Joint AI Center (JAIC), which was to take on everything that we weren’t doing in Project Maven. Project Maven was about intelligence, but the entire department needed to start bringing in AI in everything else that department does, and there was no mechanism for doing that. Each of the military services had something going on, but I would call them pilot project. Small-scale pilots that could not cross that valley of death. So the whole point of this new organization was “Let’s get going. Let’s get going at scale.” Fast would be very good; fast and at scale would be superior. And that’s what this was all about, it was speed and scale.

So that is how I ended up at the JAIC after having done project Maven… And we had to stand up an – as I say, I’ve decided it’s no longer tongue in cheek. I actually was the CEO of two AI startups in the Pentagon. But that doesn’t make sense to a lot of people; like, the Pentagon does startups? Yes, we did two startups, and all the challenges that you would expect from startups, I lived firsthand with my team in project Maven and then JAIC. So that’s a good starting point to tell you the journey that we took to get to the final of 2020, when I walked out the door… But I haven’t walked out the door, because I’m still very connected to what’s going on in government.

You mentioned in part of that discussion the words “at scale” a couple of times, and I know as even now I’m building my own company, you encounter problems at each level of scale that you try to achieve… But when you’re talking about at scale at the government level, or worldwide, where all of these organizations are operating, I’m sure there are things that go beyond the technology element… So saying “Oh, this is a great model for this application”, whether it’s a computer vision, or natural language processing, or Gen AI now… I’m mostly curious because I don’t really have a good window into what does it mean to actually scale one of these applications, AI applications in the national security space versus in an industry context, where you might be scaling to this many users or something within your company… What’s different, and what’s the same, and what’s kind of unique of that at-scale component in the national security world?

Really, all germane points, Daniel, and there’s so much to talk about just on that alone. I think I’ve now spent enough time, now that I’m out of uniform, I’m out of the Department of Defense, working with venture capital companies, inside partners in particular, and I get a chance to go and spend time with CEOs of everything from small startups to pretty big companies that are still in that venture capital business somehow, whatever seed round of funding that they’re getting. And the journey is remarkably the same. Actually, surprising to me… I would sit there and cry in my beer, so to speak, and say “Oh, nobody understands my problems”, and then one day I’m driving into the Pentagon, just despondent about how I’ll never get this thing called the JAIC built, because I have no people, I have no money… How are we going to get there?

[00:10:09.16] I’m listening to Guy Raz, How I Built This, and I was listening to the CEO, the founder of Belkin Routers. It was a fascinating story. It was exactly my story. He says “I built this thing in my parent’s garage, and there were days when I was just ready to throw in the towel and give up, and there were other days, 24 hours later, where I would suddenly - some technological breakthrough or some big contract that we got, and all of a sudden it looked right again.” O lived that. I said “Well, if he could turn it into that company, then hell, I can do the same thing in the Department of Defense.”

And I want to come back to something you said, Daniel, because it’s so incredibly important to this discussion. The technology, of course, was fundamentally at the center of what we were trying to do. But everything else was even more important to “How do you get to scale?” So I would say maybe the differences are - well, they’re used for military operations… But I would say that if you look at any big commercial company that was not built as a digital company, it was built as a hardware company in the industrial age, we have the same experiences. I’ve talked to enough people that say “Well, yeah, it’s the same thing we’re going through.” It’s getting that pilot project scaled and built in a way that you can then put it in one - for a company, let’s say for a medical or a financial business, it’s for that company. But for a lot of other places, it’s “How do I get this thing scaled to the rest of the world?” But in terms of industries, very similar to what we were going through.

And I talk about this all the time, I have eight things, and I don’t have to go through all of them, but they have nothing to do with technology… It’s mandate, vision, alignment, obstacle clearers, bureaucratic enablers, resources, authorities, and then sort of talent management. All of those are not technology-focused, but they have to exist, or you can’t get there. We all want to focus on “I have this cool new widget/gadget, and this thing is going to change the world.” Well, it might… But if you don’t have all those other pieces in place, which if you understand the bureaucracy known as the federal government, you have to have somebody that understands how to navigate in that world. Whatever I didn’t know about technology - and that’s a fairly long list of things at the time I started all this journey - I sure knew a lot about how to work in the federal government. I commanded six different levels in the United States Air Force, I had been in all these places where I had to do this day in, day out… So that part I was very comfortable with. I wasn’t nearly as comfortable with technology, so I had to – it was a hockeystick learning curve. No if, ands or buts about it. I still feel like I’m a novice today, by the way… And that’s seven years in the past when we started. It’s actually eight; we got started in 2016.

So I think there’s more similarities than there are differences, at least for the non-digital companies, because they have the same problem that a CEO is experiencing today, saying “Is this AI thing real? Should I really invest millions of dollars of this company’s money? What is the return on investment?” That return on investment question is an unsettled discussion, and I had lots of those with people very skeptical about what we were trying to do in the Joint AI Center.

That’s actually a point I’d like to extend a little bit… You talked about the federal bureaucracy, and the challenge… And the US military is not just a single organization, it’s a lot of large organizations, with their own structures and their own cultures, in each of the services… In a sense, that’s almost a much harder thing, I would argue, than a lot of Fortune 500 companies; not only from size, but just you have so many different things at play.

When you’re taking something is new and as hyped as AI is… So you’re having both the potential of the technologies that are being developed, but with a lot of hype thrown in on top of that, and those skeptical people in different organizations within the larger… How do you navigate that to make – you talked early on about the velocity being so important, velocity at scale… How do you navigate that in a large organization? What are some of the lessons that you learned on that? Because normally, we see such slow change – the American people will see such slow change, and other nations too, in their governments. At least that’s the perception. You did a lot in a very short amount of time. How did you navigate that?

[00:14:16.00] Very carefully. It was hard. It was hard to do it. But Chris, I’m going to come back to the word that you said, and to me, I would put this at the core of everything I was trying to do… And that’s culture. You’re trying to change the culture of an organization. And the thing about the Department of Defense - I’m not going to say it’s completely unlike a lot of big companies, but I think there are big differences. Because when you talk about the Department of Defense, there are cultures within cultures within cultures. There are a military unit cultures, there are service cultures, there’s an Office of the Secretary of Defense culture, there is a Pentagon culture; foreign as that culture is, it is a culture unto itself when you walk into that building every morning and leave at the end of the day, dark on both ends.

So this culture piece is difficult to figure out how to change that. So what you have to do is just persist for days, months and years at a time in saying “Join me. Come on this journey with us.” And there’ll be a lot of resistance to that, because not surprisingly, when you talk about war fighting, the Department of Defense is still largely a risk-averse organization. It has to be, because lives are at stake. We’re trying to take a culture and change it in a way that people are just not comfortable with because of the hype, because they don’t understand what AI really is. To them, they keep hearing Miracle Whip, Miracle, Miracle Whip. This thing that we’re just going to take out of the jar and spread on a couple of pieces of bread, and all will be well with the Department of Defense. It just doesn’t work that way.

So this idea of changing culture from a bottom layer, top layer, middle management layer - it all has to be done simultaneously. And part of this culture is - and this is why when I talked about what you need is not just the bureaucratic enablers, but the people who are obstacle clearers. So I put them in three categories: the classic disruptors, which the Marine Corps Colonel that ran Project Maven, Drew Cukor, was absolutely positively one of those. You also need the people that clean up the broken glass that comes from the disruptors. And we had a lot of people that were capable of doing that.

And then I have the networkers. These are the people that a lot of times are what we would consider middle management. They get disparaged, but to me, they’re the key to success. They’re the people that know how things get done in the federal government. They go have a cup of coffee with the budget person, they go have a cup of coffee with the service general officer, and they work the networks that they’ve established over the years. And if you find somebody that can do that effectively, you’re going to do much more than people that just think sheer force of will alone will change culture. You really need some ratio of all those three types of people. And the ratio will change over time, which I’ve found - in Project Maven, we need a disrupter, period. We had to force-feed this down the throat of the Department of Defense. Hard, hard, hard to do. But over time, as I got into the JAIC, I needed more of the people that could do the networking, and that would do a little bit of disruption, but more it was about cleaning up some broken glass, but really moving faster and faster. So culture eats strategy for breakfast. It always has, it always will. You have to put culture at the center of any technology.

[00:17:27.19] Well, Jack, I really love how we’re getting into a lot of these culture, strategy, talent sort of subjects, which is really key and practical across a lot of organizations that are trying to adopt this technology… One of the things that you mentioned which kind of caught my ear is - you know, there’s people involved, and particularly in the context that you’ve been working in, that do have a true, valid concern around risk of these technologies… Like, “Hey, people’s lives are on the line, and we care about that”, which hopefully, they should care about that. Right…? So after working in these environments for a while and bringing new technology to sort of risky situations, for a lot of those people out there that are also navigating - maybe it’s not people’s lives, but maybe it is like real impact to people’s lives, and integrating AI systems around automation, or really sensitive subjects like finance and healthcare and other things… Do you have any learnings or thoughts from your experience of AI in risky situations, or with sensitive data, that you’d like to highlight, and how that played out in your situations, and what you learned over time?

Yeah, Daniel, I would say this is something we talked about pretty much every day. Now, Maven was a little different, because we knew the problem from the beginning. That problem could not be solved in any other way, so we knew we were going. But when I got into the JAIC, the conversations we had were generally along this line: we’re going to start with lower risk, lower consequence use cases. Solve those, learn from them, and then slowly move up the ladder. Maybe even fast up the ladder, depending on what we were talking about, to get into those higher-risk use cases.

There is a lot of talk, of course, about the dangers or the risks of so called killer drones. I’ll tell you the thing I wasn’t working on in the last five years of my career was killer drones. I had nothing to do with those, because they were too high risk, too high consequences, and we had to understand how to actually do AI before it started getting to that.

So we started with things like predictive maintenance, or there were some medical initiatives that we were taking on. There were some things that we could put on individual intelligence platforms or sensors. And so we learned from that, and then based on those lessons learned, began to sort of edge our way into more consequential use cases. And to me, there would be a clear analogy with any business that’s out there. Because again, as CEO, the risk will be different. It might not be life or death, but it could be tens of millions of dollars, and that’s very risky for a CEO.

So the idea of proving success at something… And the reason we stood up the JAIC is because we showed enough success in Maven. As difficult as it was, we did show real success, and put AI models out in combat operations within a year of standing up the organization. Okay, let’s go try some other things. And so what you’ll see now is, okay, now that we’ve learned all these lessons, they’re going to a little bit more consequential use cases. Maybe AI-enabled autonomous drones. Maybe not lethal drones, but autonomous drones. And then you’ll start seeing this. In fact, you can kind of see some of this playing out in Ukraine, as you look at the headlines what they’re doing with drones on both sides right now.

So to me, it’s like starting with something that’s manageable. Get your arms around it; learn what it means to build a data management pipeline. Because what I’ve found in the DOD, hands down what stopped people cold when trying to start AI projects was data. Much better now than it was seven years ago. Much, much better. But it’s still hard. So if you can’t get over that data hurdle, then people get despondent, they can’t reconcile what they’re hearing about the hype, with the challenges of doing it for real. And I’ll tell you, I’ve been around long enough now to know when somebody has never done a real AI project and just talks about AI… Because there’s a big difference in those two things.

One of the things I wanted to share with the audience is that you have recently submitted a paper with a lot of guidance to the US Senate. They had the AI Insight Forum on National Security, and you submitted your document… And this was very recently, on Wednesday, December 6th of this past year. And it’s great in that it gives a fantastic kind of oversight into AI in a national security context, and you offer some recommendations, which I’d love to go through in terms of that. But you mentioned Ukraine, and you also mentioned that in the document. I was wanting to, as a transitional question, kind of - what have we learned that you can share from Ukraine?

[00:22:01.25] So as you’re racing through getting AI gradually integrated into our armed services, and we’re watching and supporting the Ukrainians against the Russians, and we’ve had this real-time, real-life learning process that’s come out of that, and seeing what they’ve done with drones and stuff… Is there anything that stands out in your mind, that was either great learning, or something unexpected, that came from that real-life application that’s being developed right now?

Technology is at the center of this fight that’s going on, this war that’s going on between Ukraine and Russia. And what you’ve seen in Ukraine in the first year of the war was how quickly they adapted. It’s an amazing story. It’s actually an amazing story, that what you have is some people that might have been born in Ukraine came to United States, got educated, went out to Silicon Valley… But when the war started, went back over to Ukraine, and then focused on how much faster they can bring technology into the fight. Things that we’ve been talking about doing in the US Department of Defense from many years, they did instantly.

People don’t maybe understand this part of it - moving their entire government to the cloud. If they had not done that, it would have been a disaster. They would have lost everything. But then just on the military technology pieces, how quickly they were able to bring in what I say is this really an example of what we’ve been talking about with software-driven warfare, is moving that fast. And it’s not to imply that technology, of course, is no longer relevant. It’s as relevant as ever. But technology that is software-defined is a different kind of technology, and that’s how Ukraine has been gaining an advantage, is by moving faster than the Russians and adapting much faster. And the drones are the best example of this. It’s crazy to see what they’ve done. The idea of bringing in 100,000 drones, some of which are first person view, and you’re watching somebody wearing these goggles drive a drone with explosives on it into a Russian tank and blow it up.

So there are a lot of lessons. Now, no two conflicts are exactly the same. So we’ve got to be careful about the fungibility of lessons learned. But I think there’s so many lessons that will apply to conflict for the next 10 to 20 years. And this idea of being smaller, smarter, cheaper, attritable, networked, and even swarming. It’s playing out in Ukraine as we speak. Now, is that going to make the difference between winning or losing the war? Well, I think it’s made the difference in not losing the war. It’s much harder to win the war, because you’re up against Russia. It’s a very difficult fight that they’re up against. But what we’re seeing is technology used in such imaginative ways. And it’s this weird juxtaposition of World War One-like trench warfare, with AI-enabled systems, Maven-like capabilities that are being used to find targets, spot targets, and send artillery against targets. So you’re seeing what a lot of us thought was coming, and it did happen, and it is happening, about how quickly you can adapt to the changing conditions of the battlefield, or I would say battlespace, because cyber is part of this as well. Electronic warfare is really, really important right now, because it’s killing Ukrainian drones. The Russians are very good at electronic warfare. So now the Ukrainians are trying to adapt based on what’s going on.

So we have to listen to those lessons, and apply them to the US Department of Defense. And by the way - and I’ll stop here - I think what you’re seeing is the replicator initiative, which is buying thousands of drones of various sizes and capabilities, is in a large part, I think, based on the lessons that they’re absorbing from what we see in Ukraine.

I wanted to go back for a moment to software-defined warfare. The paper that you just referenced, that you wrote with Mr. Mulchandani, if I’m getting his last name pronounced correctly… Who is your CTO at the JAIC, and I believe he’s the CIA CTO at this point…

He is.

[00:25:47.02] That was quite a landmark paper. Ironically, for those of us who are in industry and maybe you’re not military related at all in the audience, you’re used to Daniel and I always talking about “AI is still part of the software. It’s all bound together, you can’t do AI without the hardware and the software and the systems written together to make it practical AI that’s usable.” And you really went there in software-defined – it’s called Software Defined Warfare Architecting, the DoD’s transition to the digital age… And it was quite a landmark paper for those of us in that industry, because it really laid out the future of how software needs to be integrated. Do you have anything that you wanted to comment on about that, just given – I thought it was very important to anybody concerned with AI, in the DOD.

And it was really written by Nand Mulchandani after his experiences in Department of Defense with the JAIC. But then he left, and before he took to CIA, he had all these ideas germinating, and he said “You know what? I’m going to write about this.” And we’re co authors, but he’s the author. I’m the editor, and I placed the operational imprimatur on that report. And it’s so important, because it represents all the commercial software industry’s best practices that need to be brought into the Department of Defense. And what’s sad about this, Chris, is - to your point, if you’re in the commercial tech industry, and you were to read this report, you would be flabbergasted that the Department is not already doing all these commercial best practices, which are just the way of doing business. You know, microservices, platform as a service, software as a service… It’s just so foreign to the Department of Defense. It’s getting there, it’s come a long way, but that’s why he wrote that paper. It is a blueprint, and saying “We can talk about –” This is what Nand and I used to have these conversations toward the end of my time there… He said “We could talk about AI till we’re blue in the face, and it’s a wonderful conversation. But unless you make the Department of Defense modern, and digitally-modernized, which includes data best practices, and do all the things that’s in that software-defined warfare report, AI will be meaningless. You’ll never get there. At least not at scale.”

And by the way, I am now a member of the Atlantic Council’s Commission on Software Defined Warfare. So they’ve taken that mantle and are making it the centerpiece of all these recommendations they’re going to have. So it’s a really important piece, and I thank Nand, because - as I’ll say, for as long as I talk about this, Nand changed that organization the day he showed up, because he understood there was a different way that we had to be doing business than the standard traditional Department of Defense way.

One of the interesting things that I think - I haven’t seen the specific paper you’re referencing, Chris, but I like the ideas that are being discussed here, because one of the things that I’ve seen… And I’ve kind of tried to parse through and figure out how to put words to is when I’m on customer sites or interacting with people, it’s something about this AI technology that seems to disconnect people’s mind from the fact that there’s still a need to have error checking, or… It’s not just that you have this AI, and you send something over and it kind of automates things, and then you move on. There’s still the idea of a software application, and there’s still a call, at the minimum, to an API that maybe you need retries around, or some sort of health checks, or backups, or this sort of thing. Does that perception hold water in terms of what you’ve seen at JAIC as well? And you were talking a lot about culture… Any recommendations for those kind of dealing with this fact of swapping software for AI versus embedding AI in software?

Yeah, it completely resonates. It absolutely resonates, because this is – I even say, maybe this is going a little bit too far, but I don’t think it is. And Ukraine’s validating this, by the way. The next conflict is going to be an API-driven conflict. If you get that piece right, and you can update faster than your adversary on this idea of software-driven warfare, you will have a competitive advantage. Because the battle speed – let’s face it, you put out an AI model and you never update it, then you might as well have never done it in the first place.

[00:30:00.14] The idea of – the battlefield is going to change, the battlespace will change, but also models will drift, and all the other things that happen, they get exposed to different data, new data, whateverc… You’re going to have to do a continuous integration, continuous delivery or deployment. That’s part of this. But you can’t do that in the traditional way of doing, which is you have an entire weapon system that you have to pull apart, completely pull apart, rebuild, and put back out again. That doesn’t work. You’re going to have to break all this apart and just focus on those little bits and pieces that have to be updated.

You mentioned API calls.. There’s so much more that can and should be done in the Department of getting those status updates in two-way feeds. One feeding those updates of higher headquarters down, but two, putting all the things that the end user is seeing, feeding that back uphill. That’s what we have to be thinking a lot more about, to be able to handle what I think is going to be a very, very chaotic environment.

Now, this AI, howevergreat it may eventually become, does not change the fact, as I said in my testimony, that warfare is a very chaotic, nonlinear, dirty, ugly place. And it’s horrible that we have to go to war. But this technology could provide that competitive advantage that makes a difference in a future fight.

Break: [00:31:13.20]

To extend what we were just talking about right before the break, you kind of finished talking about the study of warfare being asymmetric technology advantage, you talk about that in your paper, and just the challenges… And you’ve talked about integrating in these latest, greatest technologies. One of the things that you address a fair amount is you talk about teaming between AI and a human being in the context of national security, and adjacent with that, which I know Daniel asked you a little bit of a question about that earlier, kind of having to do with safety and the security of the AIs, and stuff. Can you talk to us a little bit about – that’s obviously a question that many people have in their minds that have nothing to do with this space, the military or the industry around it, just concerned about what does teaming mean? How far in your thinking, does autonomy go? Is there a point, just to infer something, that as the pace of warfare speeds up - you know, we have drones, and we have different types of autonomous technologies and stuff…But as the pace of warfare speeds up and you look into the future, versus kind of what we’re looking

at in the near term, how does that change the human AI teaming equation as you’re trying to keep up, and maybe having humans in the loop can become a problem or a serious logistical issue? How are you looking at the future in terms of where we are today, and where that’s going, and what are some of the challenges that we need to overcome to get there?

I think so strongly about this idea of human machine teaming or human systems integration, that if we don’t get this right, we’re in trouble. And what I mean by that is, there are people have been writing about human-machine teaming in some form for 50 years. No different in terms of the concept of “How do you get the most out of human machines?” But to me, we’re at a period where it is changing. And it’s changing because the machines are going to be so much smarter than we’re used to talking about, that it will be a different view of human machine teaming than we’re used to. So I think there’s a whole new research field. There are already people working on this, and I’m amazed at some of the work that’s being done. But we just don’t know how good some of these technologies are going to be.

Let’s just take the example - very brief, the example of sort of ChatGPT, or the equivalent. There’s this idea of prompt engineering. I look at that as almost like the elevator operator in the ’30s or ‘40s, whenever it was, and it goes away eventually, because people get comfortable with it. But they’re not as comfortable with it today. You need help figuring out what that interface looks like.

But the other thing I’d say - and this is where I come back to - I really do believe there will be situations… I call this the bell curve of military operations. Some cases, you really do want that human making that final decision. Maybe to launch nuclear weapons as the President of the United States, and nobody but the President of the United States. On the other end, you need the machine to do what the machine does best. There is no time to get in the way. Everything else in between - I don’t know what that is. 80%, 90% is human machine optimized for both. That Centaur idea. And that we need a lot more work working through. And here’s the example I would give from my time in the flying world… There were a lot of times the machine did not operate as intended. The human was there to make sure that everything went fine with the mission. “Oh, that broke again. We’re going to have to pull that circuit breaker…” I started in the F4. Circuit breakers popped all the time. [unintelligible 00:37:12.27] hydraulic fluid out the backend, whatever. So the human would have to make up for a lot of the machine’s mistakes. I’m not sure that’s going to be a luxury in the future. You’re going to have to let the machine do what the machine does, and the human do what the machine does very well. What do humans do well? They do reasoning. They do inductive reasoning. They do deductive reasoning, they do inductive reasoning. They put things in context, they deal with other human emotions. Machines don’t deal with emotions, they don’t understand emotions, and I’m not sure, despite what some people claim, they ever will.

So what does that look like? I think there’s a lot that has to be done in this area in experimentation. Play around with it, see what works. Because if you don’t get it right - and there’s a couple of examples where it’s went dramatically badly, and I think the 737 max is one of those examples. “Well, we’re gonna put this software in. Trust us, you don’t need to be retrained as a pilot on this. Just listen to what it says in the cockpit.” That was wrong, and it has proven that it was dramatically the wrong thing to do.

[00:38:11.12] So I think this is an area that needs a lot more explanation. There are people writing about this, and really good writing, in various places… But we have to figure out how to get this part right. Because the machines are going to be smart enough that they have to be allowed to sort of run when they should run.

We’re seeing both modern models are getting so impressive in certain areas, in certain capabilities. There’s a lot of hype around them, they can’t do everything… But what they do, they tend to do quite well, recognizing that there are hallucinations and other technical issues to be worked through over time. But we’re seeing kind of a rapid increase in capability in a lot of these areas. So one of the things that I get asked all the time, just kind of on the side with people, is as those capabilities go to some level of increase in the future - whatever that is, and whatever the tasking is - and you’ve kind of alluded to, it kind of changes the balance in human teaming… Is there a set of metrics, or some guidelines that you have in your mind on, as you see the technology progress - maybe not today, maybe not this year, maybe not next year; but maybe 5-10 years out. And you see these capabilities in specific areas increasing far beyond what a human can do for a given task… How do you rebalance? How do you assess that? You talk about assessments in your paper as well, so that you say “This is one of those moments where you let the model do what the model does really well, because it does it faster, it does orders of magnitude better than a human could do in the same amount of time…” How do you make those assessments? Because I think that for us humans with emotions around technology and warfare, I think that causes a lot of concern. It seems to be the foundation of many questions that I get asked… How do you make that metric judgment on those adjustments to the culture of how we interact in that way, kind of the way we think about it?

Yeah, a few thoughts on that. And while I remain at AGI skeptic, I do believe these machines, these AI-enabled or smart machines will get so good, the human interfering with the machine could be a worse outcome than human-machine together. And the example that I’ve heard - I think was Gilman Louie raised this about AlphaGo and the Move 37. If a human was there to override the machine, it would have lost the game. The machine said, “No, you really do want to make this move. Leave it alone”, and it won. So that’s an example of - humans are going to get to the point where they’ve got to be more comfortable with this technology. And they’re not necessarily today. Why? Because just like when I use ChatGPT, whatever version of a large language model, it does get some things still very wrong. Maybe the ratio is only 10%, versus 20% just a year ago. But if that 10% is dangerously wrong, then do you really want that in military operations? Not yet you don’t. So how do you do that? This is the core of your question. To me, it is testing evaluation, it’s experiment, it’s putting it into the hands of users as an MVP, a minimal viable product… Which again, is a little bit different, or a lot different than the way the Department of Defense has fielded systems in the past, where you field any system when it’s as close to perfect as you’re going to get… Which may have not been perfect, but close enough for government work, as we say.

But in this case, you need to put things in the hands of users sooner rather than later. This is why I’m a big fan – despite my reluctance to say “We should not be using large language models for putting items in the presidential daily intelligence briefing”, you should have all sorts of experimentation being allowed in all these federal government organizations and agencies. In fact, the White House executive order on AI says “Go try this out.” It does – it basically says “Go do it.” So I’m a fan of the experimentation, what we’d call sandbox experiment; put it in users hands. Because we don’t know yet. But part of that is core testing evaluation. We do not want a short-circuit testing evaluation. And why is this so important? Because I think that’s where we’re gonna see the biggest risk, at least initially.

[00:42:10.24] Some risks are going to be hard to figure out until you use them in an operational world. We just know that, just like in commercial business. Some things are going to surprise you. But that’s why you then update the models, because you learn. But in that early stages of design and development, before you get to the fielding part, there is this thing that we have done really well in the government for many, many years, especially on the hardware side; it’s called Test and Evaluation. That does not go away when we talk about AI. In fact, AI is still novel enough, and I think it’s more important than ever to get spend a lot of time on the T&E. It doesn’t mean you’re going to go slowly relative to sort of putting out other things, but I am a little bit cautious about moving too quickly on some of this, because we just don’t know what those risks are yet.

There’s a lot of work going on, including some things I’ve been working on on a risk management framework for AI systems in the military. Because this risk - it is a hierarchy, one, and their AI-enabled weapons are really, really bad. On the other end, there’s process automation for a finance system that touches nothing having to do with warfighting. Negligible risk, move really fast on that one. And then a lot of things in the middle. You’ve got to do test and evaluation to determine how many risks there are, and then come up with risk mitigation strategies.

I’d love to weave a couple things together from what you mentioned in your testimony, one of those having to do with a concept which I thought was really interesting around techno-economic net assessments. And I think there’s a lot of people - so I’ll ask you to maybe explain what you mean by that. There’s probably a lot of people that see news articles about “Oh, this country is ahead of this country in this type of AI, whatever”, and of course, you see articles about “Oh, China, whatever company in China bought up this many GPUs”, or other things like that… And depending on where you’re at, you get concerned… But it’s all sort of very amorphous, and it’s hard to really grasp where countries are in terms of the AI stack, and their capabilities… And the other thing that you mentioned in the testimony is encouraging the DOD to take bigger bets. And I imagine that that’s also connected with sort of making sure that our techno-economic net assessment is sort of on the upward trend. Could you talk a little bit about those ideas, and maybe how they’re connected, and how you would love to see those kinds of big bets going forward?

Yeah, if anybody was not in the government and skimmed through my testimony, they probably would have went past that paragraph and not paid any attention to it. There’s a reason I put it first. It’s that important. It really is the center of attention. And that is - let me put it in terms of, say, one commercial company competing guests another commercial company, in the same general business space. CEO of one is always looking at CEO of two, saying “How much faster are they moving? What special sauce are they bringing into their product that threatens our market dominance?” This is going on every day in commercial industry. Well, it’s also going on between states, in this case in AI.

So the problem with state is you can get a lot with industrial espionage, but it’s a little harder to do when you have a nation state like China, and both sides, the United States and China, are talking about how much they’re each doing in AI… But how much is reality, and how much, again, is the hype? Well, you need a lot of intelligence assessments. And I’ve found this out from my earliest days in Project Maven, where I would ask these questions about “Okay–” And I’ll stay very unclassified here. The People’s Liberation Army - what does their AI stack look like? The intel community is not spending any time collecting - or at least they weren’t at the time, because nobody told them to. What is a GPU? Okay, well, when we know where we have to start this conversation then is “What does their compute look like? What do their models look like? Are they using open source, or who’s building them for China? What does their talent base look like?”

So this idea of not just technology, but then, equally important on both sides, US-China or US-Russia, whatever you want to say, is what are they doing with the technology? What are they building new operational concepts? Are they actually reorganizing? When you start seeing bureaucratic reorganization, they’re far along. We don’t see that yet. We don’t see it in the Department of Defense, because we haven’t figured out exactly what this new technology is going to do for us yet. So that’s what I mean by this.

[00:46:26.12] And by the way, the US government used to have this thing called the Office of Technology Assessment. I think Newt Gingrich managed to kill that as part of the revolution in government, or whatever. Now there’s some serious work to bring something like that back. Now, I’m just talking about within the Department of Defense and the intelligence community, on the classified side, how do I bring in all this information, both unclassified and classified information, to give us a relative net assessment? A net assessment, us versus them where do we stand. And it turns out that’s pretty difficult to do, because technology is not so easy to take a picture from a satellite of a GPU, and decide how much farther they are ahead. So to me, we’ve got to do better at understanding United States vis-a-vis China, or Russia, or anywhere else around the world, what they’re doing in this area. It’s such a big concept to get right, and it’s hard, because it’s just not like collecting against tanks, or nuclear weapons, or something. I can’t see it anymore, and as I say, there’s no fluid coming out of a building somewhere to tell me that they’re working on a particular project there.

Well, Jack, we’ve covered so much material here, and time that we’ve been talking, it’s flown by. I think we’re gonna - if you’re willing, we’re gonna have to have you back on the show, because there’s a lot that we haven’t been able to address yet…

As we finish up, I’m kind of thinking in the background about culture and what you were talking about, and just the massive organizational, and even national change that we’re doing here in the United States, in government, in military, in intelligence… It’s changing the way that we are looking at the future of warfighting. You mentioned in your paper the joint warfighting concept, which I’m very familiar with, and I believe there is an unclassified version out there that we’ll add into the show notes… But it’s changing the way we think about all that we do with the military and intelligence, and I think AI is a big part of that. As we wind up, do you have any – we often ask guests to kind of take the last question to be whatever you want it to be. Paint a picture of the future with whatever you want to be. Can you give us a sense of kind of what you’re excited about going forward, how you think some of this may evolve, and what that means to the United States’ military and intelligence communities, so that the listeners of this mostly not involved in that directly have a sense of where things are going? Any thoughts you want to finish with today?

Yeah. Thanks, Chris, and thanks, Daniel, for allowing me this time with both of you today. And it’s a big thought, and it’s something I’ve been thinking a lot about… And when I retired, I went back and got another master’s degree thanks to the GI Bill… And I did get a chance to think big thoughts about technology over the course of human history. I mean, really, back to the very beginning I took some courses about looking at what we would call - maybe not technology today, but it was back then; [unintelligible 00:49:07.11] and how it just sort of diffused globally. What I say in my testimony, I do believe in my core that we are going to be in the middle of, at some point, a third revolution. Sort of agrarian revolution, industrial revolution… This is different. It’s not the Fourth Industrial Revolution, it’s some kind of digital revolution. We don’t know what it’s going to look like, because we just haven’t been there yet. I say in my testimony, “The future is, to a large extent, both unknowable and unpredictable.” Why? Because it’s not determinism, and it’s not technological determinism. It will be dependent upon the decisions by many, many, many, many thousands of people, from leaders to citizens of countries deciding they like or don’t like the technology. It’s going to take maybe 50 years to 100 years to play out, and historians will be the ones that look back and define when this revolution began. But to me, it is fundamentally different, which means warfare will be different. The character of warfare is going to change. I will not say the nature of war is changing, because the nature of war is a human-centered decision. Like why we fight, and why one country fights another country, and so on. So I don’t believe the nature of war. But that character of warfare is going to change dramatically, as we’re seeing in Ukraine, and we have a chance to be on the right side of that, as I say in my testimony, asymmetry equation. If we’re on the wrong side of it, we risk losing. And we’re not used to losing… And this is a very serious risk. So this idea of it’s playing out as we speak; get involved in it. Don’t wait for it to catch up. You’ve just sort of got to dive in and start working these big projects in the government, wherever you are, or anywhere else in the industry.

That’s a great call to action to finish up with. Jack Shanahan, thank you so much for joining us on the Practical AI podcast. Really, really interesting conversation. Thank you for your insights, and hopefully, we can get you back on the show to cover things going forward. Really appreciate your time.

Thanks, Chris, thanks, Daniel, and of course, I’ll be glad to come back.