Bryon Kroger: Prodacity & Outcomes in Prod - Shipping Software in the Age of AI
Summary:
Bryon Kroger opens Prodacity 2026 by laying out the scaffolding for the whole three days. He starts with how he got here: seven years of targeting operations using software so bad he watched missions fail, a move into the program office that built it, and co-founding Kessel Run, where the software factory name he chose took on a life of its own. His argument is that no factory, platform, or tool chain was ever the point. Continuous delivery of valuable software users love into production is the point. From there he walks the mission logic model adapted from the Kellogg Foundation, the four-step mission improvement framework the curriculum follows, Wardley Mapping as a guide to method and team design, growth boards and the value equation, and the elite DORA bar he believes GovTech should hold itself to. He closes on what AI changes: software generation has sped up more than tenfold, which moves the constraint to identifying the real problem and breaking it into outcomes.
Transcript:
Thank you so much for coming out. Oh, if we could just fill in. Come on, fill in, fill these couches. All right. So I know the talk title on the agenda was Shipping Software in the Age of AI. That was meant to be my subtitle. I'm actually not going to talk a lot about AI. Maybe that's good, that's bad. I don't know. You're probably tired of hearing about it at some conferences. But what I like to do to open every single year is talk about why we're here, prodacity and putting outcomes into production. And then I like to provide the scaffolding for the curriculum. So we treat this like an education event, and there's a syllabus that we're following. And I'm going to give you the scaffolding to fit every piece of knowledge that you're going to learn throughout the three days into so that you can remember it and take it home.
The subtext of that this year though is that everything is changing with AI and not in a hype cycle sort of way, but in a very real way. And throughout the conference, we'll be talking about what is changing and what we're doing about it. But before I get into that, I want to share how I ended up here and why we decided to build this community. So a long time ago in a galaxy far, far away, I joined the Air Force as an intelligence officer. And for the first seven years, I did mostly targeting operations and I used really terrible software to do critical missions. And I watched missions fail and people die because of bad software. I got so fed up that I applied for acquisition intel exchange and I pulled a bunch of strings to get assigned to the program office where they built my targeting software.
And then I called Defense Innovation Unit, had a little X at the end at the time, Defense Innovation Unit Experimental, and started this series of events that would lead to me and my team co-founding Kessel Run. And it's a name that we chose because we felt like we literally had to smuggle DevOps into the bureaucracy. It's a smuggling route if you're not familiar with Star Wars lore. And what seemed a good idea at the time, we decided to call this thing a software factory, a choice I maybe regret because it's a term that took on a life of its own. Over time, it became synonymous to some people with the platform or the tool chain. For other people, it got conflated with organic blue suit development. But we defined it in the Toyota production sense that you see here, the combination of people, process, and technology that provides the ideal conditions for continuously delivering valuable software that users actually love to production with minimal waste.
And that still matters. So I'm going to come back to that in a minute, but ultimately we were very successful in continuously delivering valuable software that warfighters loved halfway around the world onto the secret network in an armed conflict. It was far from perfect. I know all the warts because I'm responsible for a lot of them, but it sure beat the death star. Now, around this time, I started to notice that this really good thing was going really bad. And my friend Dan Ward, who you'll actually hear from in the agenda later, he pointed me to this discussion where Charlie Rose asked Warren Buffett, "Should wise people have known better?" Buffett replied, "Of course they should have, but there's this natural progression for how new good ideas go badly wrong." And he called it the three I's. First come the innovators, then come the imitators, then come the idiots whose avarice undermines the very innovations they're trying to exploit.
And this is a force that we all fight against in this community, that we start a good thing and it quickly gets taken south. They were again far from perfect, but the innovations were definitely undermined. And I'm not telling you all of this to lament about the past or throw Darth Vader under the bus. I do know who it was, but I'm not going to tell you, and it wasn't my father. But the idea still matters. That idea that I mentioned before, that we should find the specific combination of people, process, and technology to continuously deliver valuable software users love to production in government with minimal waste. But that got lumped together with a lot of failure. Again, some of my own, I'm responsible for, and it made it really hard to carry the movement forward. So I think we have to think about that because the empire struck back and I'm sure every single one of you has experienced the empire striking back.
And I think the rebellion has taken a serious hit over the last decade that I've been doing this. In the Star Wars arc, it's like maybe we're the last Jedi or I don't know, the resistances and shambles. But I want to emphasize that neither Kessel Run nor any other innovation program was ever the thing. No software factory was ever the thing. No platform, no tool chain was ever the thing. Organic development was never the thing. Continuous delivery of valuable software that users love into production is the thing. And that's the movement that we have to keep learning how to do better, faster and cheaper. And that's why we started Prodacity. It's a home for that revolution and what we're doing takes boldness. So we put out a call every year. This year you all answered. So you're part of this community of GovTech change agents and innovators who have Prodacity, which is the audacity to continuously deliver valuable software users love into production in government.
So we want this community to be one that keeps it real about outcomes in production, right? No innovation theater, no reading our own press releases, just serving those missions. And the goal of any software delivery organization, whether we call it a factory or a refinery or just a good old fashioned IT project should be to increase mission capability and we measure that by one mission metric that matters. I'll get into how we determine that later, but what we know from the research is that when it comes to software delivery, the highest performers are twice as likely to meet or exceed their organizational performance goals. This applies across industry verticals. It includes nonprofit, it includes government. And high software delivery performance is measured by lead time and deployment frequency, mean time to restore and change fail rates, and overall site reliability or simply stated speed, stability and reliability.
And there's no trade off in this space, in the software space between speed and quality or speed and reliability. And together they actually correlate, as I said, with higher mission performance. So today's agenda is going to focus first on mission. So let's start there. We've adapted this tool from the Kellogg Foundation. We like to call it our mission logic model. Impact represents a positive change to the operational mission value stream. So not talking about IT metrics here. I'm talking about real changes on the ops floor or in the Veteran care portal or whatever mission you serve. Senior leaders tend to focus here, right? Desired impact. The problem is that execution teams then tend to focus on allocating the resources they have to activities and outputs, software features. And Josh Seiden points out that in software, there's a big missing focus area for most teams and that's what people actually do with the software we give them.
And so a mission outcome, the way we define it then, is a change in user or system behavior that directly improves the mission value stream in the desired way. And so we should work backwards from impact and outcomes, not forwards from the resources we've been given. To better do that, we have this mission improvement framework. And first, you understand the mission charter and the ideal state. So if I use that old Kessel Run example, we wanted the air operation center to become Uber for air power. Kind of seems lame in retrospect, but that was our charter and our ideal state was on demand air power.
Step two, we go map to current state. And if anybody's familiar with air operation centers, it is a 72 hour air tasking cycle. So third, we have to set the next target state and create a roadmap to get there. And that's going to be one of many target states between where we are today, our current state and that ideal state that we're trying to achieve. So most of day one of the conference is going to focus on steps one through three. Day two is about step four and day three is getting hands on with one of the practices that you chose. And right now, today, I'm going to give you the 30,000 foot view of how we work through this. And then over the next few days, as I said, you'll be able to go deep on as many as we can in the conference agenda and you'll know where to fit it in to what you need to go do.
So for mapping the current state, you're going to learn about mapping the mission value stream, identifying pain points, and then mapping the mission processes around those pains, pairing that with mapping the technical architecture underneath those processes using domain driven design, including a technique called event storming. I think we got a whole session on that and I believe a workshop as well. And then mapping the user journey through that tech enabled process.
Together, those give you a current picture, a picture of the current state with measurements, and that allows you to identify where the mission constraint is in the mission value stream. And then when you set that target state, you're doing so to unlock flow in the mission value stream. And for those of you that have been really struggling with OKRs or OGSMs or some other goaling framework in your organizations, this might be why they go here. And if you haven't done all of the other work that I just talked about, these are very hard to come up with and chances are the ones you come up with won't produce the desired impact that you're looking for.
Once we have this though, now we can map the impact. And if we need to increase benefit approval at Veterans Affairs by 30%, we look at the various actors involved, what outcomes we could create for them and what outputs we believe we would have to build to create those outcomes. For the government folks in the room, if you want to put outcomes on contract, everybody's talking about outcome oriented contracting, firm fixed price outcome contracts, this is where they should come from. But don't put the outputs on there though, because those are almost always likely to be wrong. We don't know what we need to build up front. We have to discover that through continuous delivery. And so instead we recommend thinking about this like a bet. We make a bet that to achieve the impact, the desired impact, we need a subset of the possible outcomes and the related outputs.
And that's going to form the basis of a hypothesis statement that we can now go test with software delivery. So in this example, we believe these outputs will drive these outcomes and lead to said impact. And usually there's a one to many relationship between impact and outcomes. So each output and outcome like this one might only move the needle on the impact part way, right? So it might be a one week iteration or a two week iteration. And so in this example, maybe a plain language evidence checklist for veterans, wish we had more of those, will increase first pass completion rates by 15% and lead to an 8% overall faster benefit approval. Again, not the full 30% goal. There are many other outcomes we're going to have to ship, which includes many other outputs. But once we have that, now we can build an outcome oriented roadmap, a roadmap that's sequenced by these outcomes instead of a list of features.
And that allows us to derive project or experiment backlogs for agile software development. And yes, I just said project. Did the guy talking about agile and DevOps and AI just say project? Oh my God. I did. Depending on the nature of the problem, you need the appropriate method. So another great tool, Simon Wardley created Wardley Mapping. It's a great tool for here you see the value chain organized on an X and Y axis. The X axis is the evolutionary stage of a component of your value stream from genesis to custom built to product and commodity. And as it turns out, the things on the left lend themselves really well to agile. That happens to be most of the things this community works on and what we're going to spend the bulk of the three days talking about, but worth noting that the things that land in the middle lend themselves better to lean and the things on the right, Six Sigma and more traditional project management.
So project management's not dead, it just has a place alongside agile depending on what you're working on. But again, we're going to focus on agile software delivery and figuring out how to get those new nation capabilities into production. And so for those of you that are running mixed portfolios like this one, it also gives you an idea of where to outsource, where to use off the shelf solutions, where to build in-house, and that aligns really nicely to what kind of people you should place in charge of each team leading those with your pioneer types on the left, like your real first line innovators, your settlers, the people who are really good at taking their crazy ideas and making them work in the enterprise, and then the town planners on the right who make it really operationally efficient and reliable. But regardless, we believe that small balanced teams of product design and engineering produce the best outcomes with the least risk.
And that's because they account respectively for the mission, the users and architecture. So product is always asking, will this help the mission? That's a product manager's role in this enterprise, not will it make my company money, not just to be a backlog monkey, it's will this help the mission? Design asks, does it solve a real user problem? And engineering asks, is this feasible?
And then you might say, but what about my culture, Bryon? I don't have those people. Well, when the first teams ship outcomes, everyone else sees the results of working in a different way and that's what changes values and attitudes and that's what changes culture. So computer-based training, I was in the military, lots of computer-based training to try to teach people how to think to hope that changes behavior. And then agile certifications and the like, that's not going to cut it. We have to change behavior to change thinking. And so that's the order you should think about it in. As you do all of this and you start scaling in it, you drop in some new governance. Something we recommend a lot is growth boards, and this is really great for portfolio managers. Now you can manage your investments on an ROI basis. And speaking of ROI, cost schedule and performance still matter folks, but we think of them as one side of a value equation.
Value is not price. Value is the performance per unit of time per dollar.
And so the more we can increase the numerator and decrease schedule is actually the biggest lever, the better your value is going to be per dollar, not by relentlessly focusing on price or cost. And we know, based on what I just talked about, that performance is mission impact. And then we also should factor in likelihood of success. We also know that when the world's moving at machine speed, critical missions, like the targeting one that I served, they have a really high cost of delay. Sometimes the cost to delay is measured in lives. So we should think in terms of time delay, not just schedule. And then finally, cost is about more than just the dollars you sacrifice, effort and then other sacrifices factor in there as well. So this is a more comprehensive view of the value equation, but really at its core, we're always running into these situations where the innovators and the bureaucracy are talking past each other or coming to blows over these things.
Cost schedule and performance is not something to fight. It's just something to better articulate. Now, you may have noticed that I haven't even really talked about technology yet. The important thing is understanding the mission and then building the right thing efficiently. But you may remember these, software delivery performance becomes really important in optimizing that value equation once we have that desired impact nailed down, right? Huge implications for cost and schedule. It also, as the research shows, increases the likelihood of success of achieving that mission impact.
And there are benchmarks for this in the research, and I think in GovTech, we should hold ourselves to a much higher bar. When I show the DORA metrics, a lot of times people would be like, "Oh, it'd be really great to be a medium performer." And our war fighters, our veterans, our citizens, they all deserve so much better than that. I think we should hold an elite bar where we're deploying on demand with lead times less than a day without any trade-offs and stability or reliability. And to do that, we need to address the software delivery value stream that's underneath the mission value stream that I just talked about. And what we're aiming for is continuous delivery. Now that does not mean taking on more risk. I wish my fellow innovators and change agents would stop saying we just need to take more risk. This is actually an exercise in risk reduction.
It increases the likelihood of mission impact while lowering both cost and schedule risks. And again, we don't mean go fast and break things. So in the book Continuous Delivery, the authors actually define it as getting changes into production safely, quickly, and sustainably. Anything else is not continuous delivery. It's more akin to maybe continuous breakage. And when we say prod, we mean it, right? It's the setting where software is put into operations for its intended use by intended users. So I want to touch briefly on the technology side. And again, this is a 30,000 foot view. We're just building out the scaffolding for the rest of the conference here. A structured and opinionated PaaS is a prerequisite for continuous delivery and software mission outcomes in government. The more opinionated, the lesser app teams have to choose, configure, maintain, operate, secure, and get compliant. And so overall message here, high level, don't build what you can buy, don't buy what you can rent.
Once you've got that really good production platform in place, your software teams will need continuous compliance capability as part of a paved path to production to achieve those low lead times and high deployment frequencies for elite performers. So you call this the so-called continuous ATO that I so poorly named nine years ago. Between that and software factory, I just really missed the boat on naming things. But if we zoom back out, we need to understand the mission and the ideal state, map the current state, set the next target condition and roadmap it and then execute. So there's a lot of tactics and execution underneath all of that and we're going to get into some of that, but zooming out, it's simple enough conceptually, right? I mean, that is the purpose of framework. That's what they're for. Now, if I layer in a government procurement cycle, it would look something like the right.
So again, conceptually simple, map the mission, go acquire the outcomes, deliver software to satisfy them via a path to production and gain operational acceptance, rinse, repeat. So today we'll talk about the mission, a little bit about outcome acquisition. Tomorrow, mission mapping, more tactical outcome acquisition, delivery, the path to production. Day three, you get to do some of it for real. Awesome, right? So it sounds like we have it all figured out, but everything that I just talked about, I was very confident about at the last Prodacity 18 months ago, but we're in uncharted territory now. AI is disrupting everything in a very real way. The speed at which we can generate working software has more than 10Xed. And then the question becomes, how do we maintain that stability and reliability that I talked about? And assuming you can do that, the bottleneck or the constraint just moves to our ability to identify the real problem and break it down into outcomes and outputs in a way that aligns to the overall mission that we're trying to support.
And that sounds easy, but that's actually, in my experience, the hardest part. And now we have to deal with code coming at us 10, 20 times faster. So we have to apply AI to this whole framework, not just the software that's in software delivery that's inside of box four.
Enjoyed this talk? Subscribe for more insights from the brightest minds in GovTech.