Courtney Barno of Anthropic moderates a panel on what has to be true for AI adoption in government to produce mission value. Bharat Patel walks through the early days of Project Maven, where bounding boxes meant one thing to a GEOINT analyst and something else entirely to the fires community, and the lessons he carried into standing up the Army's first AI program. Jazmine Furtado brings the GenAI.mil delivery view, where use cases are being defined at the edge rather than handed down. Thomas Roderick covers the anti-patterns that sink pilots — starting with technology instead of a problem, skipping the baseline, trying to solve everything at once, and tying someone's promotion to the outcome. The panel closes on what to demand from vendors and what to do next week.

Courtney Barno:

Well, good afternoon everyone. Super excited to be here with everyone. My name's Courtney Barno. I am on the Go to Market team at Anthropic and I focus on making sure that our government customers realize value from their AI investments. And that makes me really excited to open up the conversation that we're having today on AI adoption. And first, I just want to frame this by saying this is not a conversation about model comparisons. I know that we have representatives from model providers up here today, but this is really a conversation about how adoption intersects with mission value. And the goal for this conversation is really to leave you all with a clear roadmap on how to go from access to adoption to mission value measured, made defensible, and then making sure that your AI capabilities scale. So with that framing, the first thing I'm going to do is ask our esteemed panelists here to just do kind of a rapid fire intro.

You've got names and titles, but what I really want you to hear is each of your perspectives. And I'm going to ask each of you to give us kind of a one-liner on what AI adoption really is in government. And if the goal is mission value, what does it really mean? And Tom, I'm going to start with you.

Tom Roderick:

Well, hey everybody, Tom Roderick. Very happy to be here. Appreciate Prodacity setting this panel up. So my entire career has been in machine learning and AI building as well as governance and really just setting up the infrastructure to be able to support it. I've been doing it for over a decade now, and I'd say AI is adopted when we no longer call it AI. It's now just something that's fundamental and part of what you do in your day to day.

Courtney Barno:

Okay. I appreciate that. Jazmine, I'm going to go to you next.

Jazmine Furtado:

Hello everyone. Jazmine Furtado. I am at Google. I've been there for the last three years. I am part of the delivery team that's delivering Gemini for GenAI.mil, which is the DOW's AI platform many you may have heard about. So my vantage point comes from delivering enterprise scale AI solutions to the Department of War. I did doing that at Google. I did that while I was in the military active duty and I'm continuing to do that while I'm in the reserves. My perspective on what AI adoption is, when you have a capability that has repeated users coming to the platform, you have use cases that actually create mission impact, those wins are highlighted and you start to see that spread across the whole organization, that's when you know that you're on the right track.

Courtney Barno:

Yeah, I love that. I love that framing. And Bharat, I'm going to go to you next, have you give us your one-liner.

Bharat Patel:

All right. Hello...fine American nerds. I'm excited to be here. I started my journey in the Defense Department. I was in the Navy, I was sailing around the world with 200 dudes, realized real fast that that wasn't for me. I got out and I started supporting the Army, working at the Program Executive Office, Intelligence, Electronic Warfare and Sensors. Did some really amazing things, fielding capability to Iraq, Afghanistan through OCO quick reaction capabilities, helped the Army modernize their entire Army intelligence layer from ground to air, to a lot of their foundational intelligence programs. Had the opportunity to be the Army's technical lead deploying AI into operations with Project Maven. And the last role I had was standing up the Army's first AI program. I was employee number one. I left it with an organizational structure and building it out. So adoption for me is several things, but I'll try and stick to one line or maybe two.

One is definitely having the appropriate organizational structure to scale it. And then the other part is working what I always call the bureaucratic jiu-jitsu, the things that you got to do in the Pentagon to work through the mess that's not really the technology.

Courtney Barno:

Yep. Yep. That's the bureaucracy hacking we all know and love, right? So I want to stay on this theme of adoption in general and really go on this journey of, okay, if I'm first trying to drive adoption in government, what does that look like? What are the different patterns that I can use to drive adoption? If you've described what it looks like when you have it and why it's important in realizing mission value, Bharat, maybe I'll come right back to you.You've had some interesting experiences with the early days of Project Maven in particular and what adoption looked like there and why it didn't quite resemble what you thought it was going to.

Bharat Patel:

Oh man, that's fantastic. And I've been at Accenture for about eight months, so the government stink hasn't come off. So excuse me if I assume that when I say the royal we, like we in the government are doing this, so I apologize about that, but we are all one team. So way back when Project Maven stands up 2016 - ish, it actually started with the Intel community. So all of the Intel community had to support pilots. I got lucky enough to be the Army's technical lead. And of course, when you put bounding boxes in front of a Geo Int analyst, the first thing they're going to say is like, "Yeah, I know that's a tank because I know that's a person, but I need second and third order effects to it." It's like, "What does that mean for me? Because I understand the basics, but the technology, that is what was the state of the art of the technology, being able to identify things in imagery.

But for Geo-Int, it wasn't that important. As that kind of evolved, the fires community picked it up because fire bubbles, they'll fire on anything. So if you put a bounding box in there, it's like," Yeah, that's a kill zone. I don't care what it is. "Obviously the Intel folks are going to be a little bit more precise on what they want to send off or call for fires. So as that happened, I also started seeing the shift from what Project Maven was intended to be was the DoD's accelerator for AI adoption. And then I was in the Pentagon. It was Army Day. We're all there. And then the leads for Maven started to shift the conversation to AI-enabled mission command. And I'm like," Holy ****, this ************ just changed the course of Project Maven because the other part to it is it's really hard. Actual AI is really hard.

You got to understand the data, you got to understand the use cases, you got to figure out how to bring that data into a relevant environment to do training. You got to understand your test evaluation, your risk frameworks, and you got to figure out how to deploy it. Very complicated.

Courtney Barno:

No, that's interesting. So there's this idea of finding the right problems and ensuring that the AI is focused on solving the right problems. Jazmine, I'm interested from your perspective with all of your work with GenAI.mil and your work over the last few years at Google, what patterns in Bharat's story resonate for you or how do you think about it differently when you think about more enterprise rollouts?

Jazmine Furtado:

Yeah. So I mean with GenAI.mil, I think we've all seen, or just generative AI in general, this democratization of these AI capabilities. Yes, there's different types of AI, there's different flavors of it. Some are a lot more sophisticated and hands-on, really require more of that mission use case or development experience. And others can be more generalized, administrative, things that can help more day to day that you don't need to have a deep level of expertise to be able to see value and impact from. So I think what we've seen with gen AI and GenAI.mil is that people are adopting, people are taking it and running with it in ways that we can't even comprehend. These use cases are not necessarily being defined anymore when it comes to AI from higher ups, leadership all the way up top that's being brought down to everyone's level to be able to define what use cases work for them.

And this decentralization of all of these AI capabilities is, we're seeing it in all these different ways. It's not these huge big projects that require all these people to be a part of that we're seeing a lot of value in nowadays. It's a lot of these smaller things that day-to-day just stack value on top of each other and we realize great gains from that.

Courtney Barno:

No, that's fantastic. There's this theme of access, giving people access to a tool and allowing them to experiment with it, drive that muscle memory and therefore drive adoption. Tom, I'm interested in your perspective here. How do you, from your vantage point, go from access to now we're adopting to this idea of adoption starts equating with value, right? You've spent so much of your career focused on measurement. How do you think about it?

Tom Roderick:

So it's a really fascinating question, right? And no organization is exactly the same. There's some common threads that show up. We heard from Project Maven where they were building something and ended up not being the right case, and that can turn into what you can think of it as a supply push where somebody from the outside, a team or somebody trying to accomplish an objective is saying, "Well, you must go use these tools." But then you want to flip the script and turn it into a demand poll so that people are like, "No, this will be vastly useful to me. This would be vastly useful to my team, to my agency, whatever the case may be." And so we saw the same thing with GenAI.mil, right? They tapped into the group that needed it. So some of the fundamental things that you see happening there is you see people that are willing to champion the integration of some type.

Maybe it's bringing in the external tool, maybe it's taking the time to learn about the tool and saying, "Oh my gosh, this would really help this group over here. Let me go talk." And it's hard to formalize that exactly, but turning it over to measurement and thinking through what adoption means from a measurement perspective, it really comes down to are people using it? Is it helping to accomplish the mission? And if it's not, do you have a way to stop it?

Courtney Barno:

And I'm going to ask maybe each of you to grapple with that question live a little bit of, okay, we're using it, but how do I know if it is helping my mission get better? How do I create evidence that I can put in front of my senior decision maker or my budget authority and either scale the pilot I'm already working on or get additional funding for it? Bharat, maybe I'll start with you.

Bharat Patel:

And I am more of not necessarily a generative AI. I'm definitely a fanboy of generative AI, absolutely. My interaction with AI is more on the sensor side. I want to be able to identify things in sensors much quicker. So sensors are constantly collecting information. The more that we can understand the battle space, the more faster and more effective we get to emitter mapping across the globe, understand what our adversary's intents are by triggering these models that can help us do that. I think that is one of the most areas where we could identify some success. I think from a generative AI perspective, a lot of it is business and can we improve our day-to-day productivity based on some of these generative AI tools as they get rolled out into the mission? I'm also curious of how we should do that.

Courtney Barno:

Jazmine, how about from your perspective?

Jazmine Furtado:

I think there's a couple interesting things that come to play when you talk about pilots in the generative AI space. There's a lot of great ideas that come out when it comes to sustaining it, maintaining it. I think that's where you see a lot of the pitfalls with pilots. So being able to not just show initial value, but then also show how it can be maintained over time, show that it has the connection points that it needs to be able to deal with the right amount of data. And then not just that, but how do you scale that beyond yourself? Do you have the right people on your team to be able to collaborate on this tool, this agent that you created, or is it all dependent on you to make it successful or not? Are you able to take it to the next level and scale it out to even more users?

How much can it handle? What are the performance metrics or limitations of the thing that you're creating? So when it comes to the pilots in the generative AI space, I'd say you have to really take a look at how you're able to take that to the next level and not just show success in your small little bubble. And with the generative AI tools that are now, you expect self-service, you expect to be able to connect to things. I think that demand signal is really coming through to all the people that are providing this and the expectations are becoming more reality a lot faster.

Courtney Barno:

I think that's great. It's really about being responsive to user demand, right? And that's a huge metric in terms of demonstrating value. I'm interested, Tom, maybe coming back to you and what other, from your career and your perspective, what other anti-patterns you've seen emerge in scoping pilots, right? If my desire is for my pilot to go off swimmingly and to have the best evidence to get this thing funded for production, how do I set myself up from day one to be successful?

Tom Roderick:

That is a fantastic question. And the playbook to make that happen is meaningful. So it's something in your organization you should look into. But the biggest anti-patterns would be starting with the technology before you start with the problem. The solution space has exploded in terms of what's available and what's coming in. And all of y'all are familiar with how hard it is to get things through ATO and to bring in things from the outside. All that aside, if you're coming at a pilot and you don't have a well specified problem that it's solving that's direct and relevant, that's typically a failure point that you're going to see right away. Now assume that you start with a problem, it's well scoped, you understand it, the stakeholders and the people championing for you understand it. A lot of folks still jump into solutioning too fast and don't take a step back to define what they're going to measure first.

You need a baseline. That baseline could be zero because this thing has never been done before. That's totally fine. Zero becomes the baseline. But if you have been doing a process or something to that effect that AI is going to be helping to enhance, you want to make sure to define evaluation before you define the product or define what the workflow will be. The third thing really is lack of composability. A lot of people want to go solve all the problems with one small pilot. Really keeping that pilot focused so that it can be deployed faster. Your speed to market in a pilot is getting the pilot completed, not necessarily getting it fully adopted. So that's one of the core items. And then the last one that I would suggest would be, be honest with the pilot. There's so many people that tie up their promotions, their rank, whatever the case may be, however they see themselves even to some degree into the pilots themselves.

And so they're like, "Oh, it must exceed at all costs." That's indicative of an organizational failure where the organization needs to be able to provide that coverage so that people can experiment without bad outcomes.

Courtney Barno:

Great. And Bharat, I was actually thinking about your background as Tom was talking here. I'm interested in the context of those three things that he just outlined. What are your reflections on Maven? And those early days. And then what were you measuring?

Bharat Patel:

What were we measuring? It was how many models we could put in front of Soldiers in units and they were interacting with those models and the feedback and how quickly can we get updated models back to the Soldiers so they can, again, interact with them. That also kind of reflecting on that to ensure that we're doing the right things. When I launched my project, it was formally called Project Linchpin. And then before I left the government, converted into product manager AI ops and services under the Army's next gen C2 umbrella. One of the first things that I did was I needed to figure out what everybody needed to do and wanted from an AI perspective. So I hung a link on the Army CIO website saying, "Hey, if you have an AI need, hit this link and we'll collect your use case." We started collecting a whole bunch of use cases.

A lot of those use cases were non-AI related. It was like, "Hey, if you got better training, man, you forgot your buttonology, go back to net and do that." And then there was a lot of just, again, non-critical AI, but we also found common themes. These common themes also helped build a POM. And if you have ever built a program from scratch, man, it's very painful, but you got to figure out what you need. So this helped me create a budget line item to work through congressional forums and everything. And then that helped us to focus on the key areas. And a lot of them, believe it or not, in the mission is still imagery. Everybody loves pixels, whether it's coming from the satellite or some kind of full motion video. People love pixels. So it's trying to build a pipeline around building an AI capability and models around pixels.

The next thing was really electronic warfare. There's tons of things that are just being omitted from a day-to-day perspective. It's trying to figure out what those are. But those are two different sensor types that will have necessarily the same type of companies and standards associated with them. So you can't treat them the same. So kind of divergent and figuring out how to do that. And then there's the how do you productize that and then bring it into an environment that you could actually use it. All this was very complicated and all lessons learned from Project Maven.

Courtney Barno:

That's awesome. So there's this element of trying to find the right problem to solve and racking and stacking. Hang on to that. I want to come back to that. Jazmine, I want to ask you from, again, pulling on really the massive scale with GenAI.mil. When we think about starting to look at metrics and measurements as indicators of value, what were you thinking about and looking at when you think about GenAI.mil?

Jazmine Furtado:

Yeah. My colleague Josh had actually brought this up during his talk earlier, looking at not just the number of users, the number of people have logged in, the number of people have logged in the last X number of days, but also the repeat users, monthly active users. Also looking at from a model provider perspective, how quickly are we able to get features out to the user community? How quickly are we able to take those commercial features and make sure they're configured with the security that's needed in order to get them out? So how can we get that done in 48 hours, 24 hours? Are we gathering feature requests from the field and turning that around and making sure that we prioritize those? There's so much feedback that comes back because people know what to expect from these tools and capabilities now. So the expectations are very high.

And so how well can we respond to those requests is definitely at the top of the list.

Courtney Barno:

Yeah, that's great. Tom, I'm going to come back to you. You laid out really nicely for us the ingredients that either make a pilot succeed or fail. If your pilot is successful, how do you know that once you are doing something at scale, it's still working? How do we start to think about this idea of mission value at scale?

Tom Roderick:

Absolutely. So you have to understand how to value your mission first. In addition to working with a lot of government agencies, we've worked a lot with healthcare and others. In healthcare, it can become much simpler. You have the notion of, well, we wanted to detect this disease. Are we detecting it? And it becomes a little funny because usually when you're looking for something, when you're like, Hey, we want to go measure this and the measurement should go up or down depending on how you're measuring it. And we did this intervention and it's happening. In these types of situations, you're hoping that your model gets it wrong more often because you're intervening because of what the model said. So the ultimate prediction starts to separate from what you're actually looking at. So this is a common ML Ops issue, but the core of it is starting with an understanding of what the value is to the mission and then going and measuring the components of that to make it matter.

So in the context of say acquisitions, maybe you're looking at, well, how many things can we review or produce or generate within a day? And that used to be two, three, five, six, and now it's 30. Well, that's great. You're at 30 now. Who's reviewing that to make sure it's accurate? There is no free lunch. You're not going to be able to just say, well, AI is going to do it and it's magic and I love it. It will do part of it, but you still need to have the human involved in the process to make sure it's actually what the mission requires. And so I think acquisition becomes a great subject to kind of frame that around because you do have a lot of documentation, you do have a lot of compliance, you do have a lot of review, you have a lot of people trying to enter it in.

And it becomes so funny on the acquisition side when you can tell that folks have generated and not very carefully edited what got submitted. And so now you get into, we call it AI lupidity where it's like, well, I had three bullet points, I generated the email, somebody got the email reduced to the three bullet points. So when you see the waste and excess coming out of that type of information or that type of flow, it's something to be critical of and to watch for. And so DevSecOps is, in my mind, one of the best frameworks to think about AI monitoring and deployment so you get full observability going forward. And that's really something I would say that we would probably want to see from providers as well. So because if you don't have good observability, you start to see things start to break. You have model versions that might come out and you built a whole workflow around one specific version of Gemini, Claude, OpenAI models.

And now all of a sudden that one's no longer available and upgraded to the next one and it changes the whole workflow and everything's breaking. So watching, monitoring, building up the system to be able to support it, that's the real key to

Courtney Barno:

It.That's great. Bharat, anything you would add?

Bharat Patel:

From a pilot perspective, I'm going to take a little different twist because twists and turns are awesome. To make pilots successful, everyone from industry, the first thing we all ask for is like, "Hey, what's the environment going to look like? What's the data going to look like?" One of the biggest struggles with pilots is showing up with a system that's going to relatively work. In order for you to do that, months and months of months will go into creating that prototype that'll work there. The struggle right now is trying to get unclassified data that's representative to industry. When I was in the government, I was lucky enough to use the SBIR program. I was the first, well at least within the Army to use the direct to phase two. I gave about seven awards, $1.7 million, 18 month period of performance. Four of those was to create synthetic data for commercial satellite imagery because I was starting to support the Titan program, tactical intelligence targeting access node, which has direct access to both the TS SCI layer and the commercial layer.

And we want to start building models. So I'm like, "All right, let's build synthetic data." The first question they asked was all four companies, "Do you have sample data?" I'm like, "Oh crap. Now I got to go figure out how to give them sample data." And we ended up doing that. I partnered with the Army Geospatial Center and had a couple of nerds keep looking at the screen, trying to get relevant images to do that. But again, a lesson learned is having relevant data in order to enable your pilot, but then also not over optimizing on that to be successful because what we learned with Project Maven, the first models, once Ukraine popped off, models did not work because it was a different biome, it was a different look and feel, different angles. It took a little bit to course correct and make them relevant.

But again, not over optimizing on exercises is also one of those key lessons learned.

Courtney Barno:

Yeah. I think in general, there are some government specific constraints that make some of these things more challenging, not insurmountable, but more challenging. Jazmine, from your perspective, I'm interested, what were some of those uniquely government challenges or constraints that you encountered? How did you navigate those?

Jazmine Furtado:

Yeah. I think the approach that Google took with GenAI.mil was to try to really embrace that commercial architecture, embrace that infrastructure that enables for those features to be pushed to production really quickly. Obviously we have to deal with compliance. We have a lot and we've talked about, we've heard from a lot of speakers, a lot of leaders are now leaning forward and accepting risk where necessary, understanding the trade offs and really understanding what this new architecture provides. Modern capabilities require modern infrastructure, modern components, making sure you have zero trust, granular access control. You have the logical separation over physical separation where appropriate, making sure you have the data meshes in place. Now we're looking at agents and NCP servers. And I know I'm saying a bunch of buzzwords, but when you actually take a look at all these characteristics, you really have to ask yourself, okay, is the solution that I'm providing to the government now, what is my answer to all of those characteristics?

Because everyone's going to ask about it now. That's what people expect now when it comes to modern software and modern solutions is that it complies with modern architecture and modern components. So it's really working with the government and overcoming these conversations. It takes multiple conversations. We continue to try to push the boundaries of what is safe to bring into this government space. And it's a long journey, but in order to really take the benefit that we see in the commercial space and bring that into government and go at that speed, we're going to have to continue having them.

Courtney Barno:

Yeah. And let's flip the script, I guess, for a second. Bharat, Jazmine, both of you have spent time as government employees, right? Bharat as a civilian, Jazmine as active duty, now as a reservist. What's your advice on what we should be asking providers for when we want things that are commercially aligned from an architecture perspective, we want feature parity, we want to drive adoption, we want to drive mission value. What do we need to demand of our vendors? And I'm putting you in the hot seat.

Bharat Patel:

Let's see. I look at the names across all of this. There are so many partners and it's like, how do you become a good partner that doesn't lock in the government to a particular product? So it's how to create partnerships and ecosystems that allows the government to evolve with technology and really not get stuck with, again, one partner. So for all of us to come together and to support the government with their limited budget, because that's still what it is, even though it's a trillion dollars, but it's still going to be a top line and everybody's going to get a little bit. So for us, we need to come together to figure out how we are going to work a little bit more effectively to enable the government to accomplish their mission because one of our objectives is to accelerate AI and to accelerate AI, it's going to take all of us.

It's also going to take the government to realize that they need more investments in actual AI, right? The infrastructure, the high performance computing center. I was the number one user of the high performance computing center at Aberdeen Proving Grounds. I had access to 260 GPUs. Why was I the only one hitting it? Because nobody in the government knew that you could actually use it for training. Insane. So the government also needs to communicate to the left and the right side because there's a lot of investments that are just being dumped everywhere. And to maximize that, again, it takes people on the five side of the building to stop playing Game of Thrones every once in a while and be friends and come together and achieve a mission.

Jazmine Furtado:

To add on to that, we've heard it all throughout the day, speed, the commercial parity, security, the integration with other platforms and tools. So you're not looking at those 17 different screens, really forcing that interoperability and that scalability ultimately upfront is definitely something that should be asked in the beginning from any model provider. How are you not providing another bespoke solution or another tool? How are you making the user experience something that'll work for the masses?

Courtney Barno:

And Tom, bring us home. If you're a program manager or an acquisition professional sitting here and you're trying to square the circle of like, "Hey, I know I want and need commercial parity. I need these various features and I want to deliver a solution that's actually going to give capability to my people, but I also have this compliance weight or this governance regime that I need to meet." How do I square that circle? What can we leave people with?

Tom Roderick:

Yeah. So I'm sure a lot of folks have thought of the idea before about automating your governance, but there's a key to that that does help. You won't automate everything, but if you can encapsulate what your business requires or what your mission requires as unit tests on the system, then that takes the behavior and understanding of what's happening both at the training and validation stage, as well as ongoing performance assessment and watching and seeing when those automated alarms start hitting. A single alarm by itself is not a huge concern, but once you start seeing a lot of degradation, then you are looking at potentially having to hit that shutoff switch on the system and reverting and you want to avoid that as much as possible. So you want to tollgate it and phase it when you are looking at potentially having to go to something else.

Don't be afraid to rebuild. Don't be afraid to build back.

Courtney Barno:

No, that's a great point. And I know we are coming up on time here, so let's do a lightning round here to end. Leave the audience with one to two sentences, one to two, Bharat, on what you would like to see people start to do next week to drive adoption for the purposes of driving mission value, right? Not just adoption for adoption's sake. Tom, I'll start with you.

Tom Roderick:

Understand the value of your mission and write it down and say, how do I fit in and where in my system blueprint or my service blueprint do I know what the bottlenecks are that I can start to attack?

Courtney Barno:

Love it. Bharat?

Bharat Patel:

Try and be quick. And I think it's give honest feedback to the government whenever they release an RFI or call for solutions and every once in a while call them out and be like, "Yo, bro, that's dumb." Maybe not use that, but just be honest. Things are complicated. Help them out because that's exactly what RFIs are for. They're not necessarily, the government's not looking for a solution, they're actually asking a question. So just give honest feedback and help them shape acquisitions moving forward a little bit more effectively.

Jazmine Furtado:

I would say we're all here as practitioners, thought leaders, leaders in and of our own right. I'd say make sure that you're training and enabling not just yourselves, but also the people that you work with. A little goes a long way. Just seeing examples of what's right, explaining the bounds, that's how we drive adoption little by little. So making sure that you have those opportunities for people to come to the table, have honest discussions, talk about the pros and cons and make it a two-way conversation.

Courtney Barno:

That's great. Well, I think we're at time. Thank you to all of our panelists here. We really appreciate it.

‍