Bryon Kroger, founder and CEO of Rise8, builds on his Day 1 keynote with the next evolution of the software factory: the Mission Outcome Factory. He starts with the problem. Eighty-seven percent of large government IT projects fail as measured by mission outcomes, and the procurement model rewards activities and outputs while the vendor gets paid either way. He walks through the options for building custom mission software, argues that much of the commercial software sold to government is single-use rather than dual-use, and lays out what outcome-based accountability requires, including a conditional performance guarantee and a first outcome in production in 180 days or less up to the secret level. He closes with how AI changes the front end of delivery, from Gherkin-format user stories to value stream and impact mapping, and a path toward near real-time outcome delivery.

Welcome back everyone. I got a little hot yesterday, so I wore a short suit today. So today I want to build on what I talked about yesterday. When you talk about what went wrong in the software factory and some of the lessons we need to carry into the future in the past, in fact, I even wrote a white paper called Software Factory 2.0, and I think that was the wrong way to think about it. More recently, as we've been seeing all of the ways to apply AI, not just to the software delivery process, but actually left of that with all the things I talked about, about how do you actually define what you need to build in the first place? And then the last mile of getting it into operations, the observability, working with users to iterate on it. I think that what we're moving towards is an outcome factory.

And I'm going to talk about what that is. First, I just want to remind everyone of the definition of outcome that I used though, which is a change, and I'm going to do a mission outcome, a change in user or system behavior that directly and positively impacts the mission value stream.

The problem right now is that the model we have for procuring software in government is broken. You all know this problem really well. 87% of large government IT projects fail, not struggle, not go over budget, they fail as measured by mission outcomes. And what nobody's talking about is that the entire system is oriented around the activities and the outputs instead of the outcomes. And so I've seen programs where teams worked for several years and where hundreds of story points got delivered or slock if you're still measuring that. The demos looked great in the demo environment at least or the laptop. Leadership across the board said, "Hey, we're on track. Things are looking good. We're going to deliver." And then nothing made it to production. Or it made it to production and users wouldn't adopt it. Or it got adopted, but it didn't actually move the mission needle.

Or in some of the cases I experienced over a decade ago, it actually made my mission workflow worse. I ended up fighting against my system. I would have rather worked on a whiteboard or a pen and paper. And so the programs that fail aren't failing because vendors or government folks didn't work hard. They're failing because our whole model rewards effort as Kent Beck talked about or activities and outputs instead of outcomes. And so at the bottom here, you see some of the problems that I come across. I mean, no matter what, we're talking about outcomes, but I still see hours go on contract, build hours, not outcomes. So traditional contracts, they pay vendors by the hour one way or another. Even if it's firm fixed price, it's probably firm fixed price LOE - level of effort. That's not bad. I'll talk about when you can use those, but in general, we're rewarding effort, time spent, people assigned, meetings attended, not mission impact.

And the vendor gets paid whether we succeed or not, whether the mission improves or not. And then the second thing that I see a lot is that the ATO risk still ends up sitting with the government. Most of our traditional vendors, even though they've been working in this space forever, everybody's always pointed at the ATO process and blamed the ATO process. I think we proved for the last 10 years now that that's not an actual barrier if you know what you're doing, but usually it's like, oh, that's out of scope. Our job was to deliver you a disc with a software on it. And either way, whether it's out of scope or that's a change order, I'd like to charge you more, they don't do it well and delivery gets stuck, you don't get outcomes in prod. And then adoption also ends up being on the government.

And so user adoption, I'm talking in the software space, it's actually different in the hardware space usually, but user adoption, operational integration, mission impact, ROI, it either gets handed off to another vendor to do O&M, which I don't know what that means in the era of cloud native software, or it gets handed off to a government organization to do. And so vendor one way or another is gone. And so when you run into problems, there's nobody to go back to that actually built the thing in the first place. And so whether or not it works in your operational environment, it's your problem. The truth here is that we're traditionally, and there are exceptions to this rule, there are some people that are crushing it. In fact, you're going to hear from some folks later today that have done really great outcome oriented contracting, but in general, we're buying hope.

We're paying vendors and just hoping, hoping that we get an outcome in production and that it turns into mission value. And every time we're like, this time's going to be different and in three years we're going to have it. And then when it doesn't, vendor still gets paid. And for the government folks in the room, you're left explaining to leadership why you spent millions or the last person spent millions on their three year tour on software that isn't being used. And so there is a better way. Yesterday I mentioned Wardley mapping and I want to highlight something here because I talked about how where it falls, where the component or product falls in this evolution axis determines what methods you should use. Most of the stuff that we're all working on does fall into either genesis or custom built. And so agile is the appropriate method, but there's something else to add on here, which is what about contract type?

People ask me, "Oh Bryon, you talk about firm fixed price outcomes all the time, but I know you have a couple TNM contracts. We don't anymore, but we used to work on them and I sometimes recommend to folks that they go do this." Why? Because here you're doing exploratory work. You don't know what the outcome is. And I can go do mission thread mapping with you and all the things that I talked about yesterday, but the reality is there's some exploration that we have to do on the ground with development teams to figure out what we need to go build. And so even if I go back to my early Kessel Run days, our acquisition strategy was we're going to do one to two years of TNM until we feel like we can put a firm fixed price outcome on contract. And that's something that I would recommend to any one of you as you're trying to do something that's truly new or in its genesis phase or firm fixed price level of effort. Where you apply outcomes is once you know what needs to be built and it needs to be custom. I'm going to talk about how you know when it needs to be custom or not, but obviously in the product space, we're looking primarily at cots and when we get over into the utility side of the house or commodity side of the house, we're looking at utilities. AWS at this point is a utility. Compute and store in general, power, all of these things, they're utilities and that's how you should think about buying them. But when I showed the value equation, there's all this stuff on the bottom and I said, we've really got to break out cost into all of the effort it takes and all of the other types of sacrifice. And I call these things below the mission value line because they're not directly related to the mission impact.

Mission impacts up on the top, that's like the user facing capability that you're building. Most of the time, certainly infrastructure, but even the paths that I talked about yesterday, that is the prerequisite for mission outcomes in the government space with software is below the value line. The government should not think of themselves as a platform company. No organization in the government should think of themselves as a platform company. Building platforms is really freaking hard. You can ask, there's a couple platform vendors here. They have hundreds and hundreds of the best engineers in the world, thousands in one case, building these things. That's not just a lot of money and people that you're applying to a problem. That's your focus. That's your mental bandwidth. You need to be focused on the mission impact, but you're focused on how to build a platform and make networking work.

That's not great. It would be like, I don't know if anybody's ever heard of the toaster project. Simon Wardley turned me onto this, the same guy that did the Wardley map, that if I asked you to go make toast, you wouldn't go first to try to build your own toaster. So this toaster project's a really great exploration of just how crazy it is to build a toaster. And they spent all of this time and all of this money to manufacture a toaster by hand and then they turned it on and it exploded.

And so when it comes time to build custom software, there are a lot of options on the table, but most of the ones we consider are not good ones. So traditional systems integrator, even one that's like, "Hey, we're going to do modern traditional systems integration." The problem we run into there is that got these four to five year delivery timelines, very waterfall approach. We all know that's not the right answer. Then there's internal build, do it yourself. For a long time back in my early days when I was a captain, I pushed this idea of Airmen coders. I do think in the long run we need to invest across all of the services having uniform software developers, but at the end of the day, the DIY, very difficult until the Services decide to make a real investment in that. Then there's staff augmentation, but I already said TNM is really good when you're doing exploration in the genesis phase, but it is actually not very efficient or effective way when you know what the outcome is that you need to build.

Put the outcome on contract. COTS, well, when you're in that genesis space, custom mission software, again, I want to point something out about COTS by the way. COTS should be defined. I wish it was defined as dual use software when we're talking in a government context, because a lot of times we say COTS, but we're talking about single use software, which has none of the advantages of COTS, primarily economy of scale would be the first thing I would point you to. And so just something to be aware of, but COTS is really good in that product space where I said, where the problem's really well defined, we know exactly what we need to do. But when you're doing it in an area like, I don't know, planning a bombing campaign or air refueling campaign or something that is inherently unique to your mission, that's a single use problem.

And so the COTS never quite fits and you end up spending a ton of money on config costs. A lot of times it exceeds custom builds costs to begin with or it just doesn't work and then you're locked into the vendor's roadmap. And then of course the last option is just to maintain the status quo what you've been doing, but what good looks like is outcome-based accountability. So first we have to define the outcome. I talked a little bit about that yesterday. I'll talk more about it today, but we need a process for defining what the outcome is and setting metrics around it. There's a lot of work that sounds really easy for me to stay on stage and I put the OKRs up there. Even if a lot of you I know have experimented with OKRs, a couple of people came up to me like, "I did." When you said that, I was like, "Oh yeah, we spent a year trying to do OKRs and it goes horribly sometimes." And so there's a lot of work that goes into being able to identify and measure the outcomes and hold people accountable to them.

There's stuff on the upfront side. There's things that need to go into your contract if you want to be able to hold a contractor accountable in proper ways. Speed to mission value. So first outcome, so assuming you have an existing path to production, first outcome should be delivered in weeks.

So when we work with customers that have an existing path to prod, we can get first outcome, not MVP, but real mission value in weeks, not years. There's a lot of things that have to be true on the government side, not just the contractor side, but that's what we should be chasing. That's the bar. So real impact, real result. Something that I don't think gets talked about enough, because a lot of times as we're doing this, we're looking for non-traditional vendors and talk about Silicon Valley a lot is you need the elite technical talent, yes, but they also need really strong domain expertise. We're talking about mission outcomes. No amount of understanding of how Kubernetes works is going to help you understand the workflow of a war fighter. And I think this goes wrong, especially in the consulting space. A lot of really consulting is, I don't mean it in a pejorative way.

I actually think that there is some value in consulting, but when you look at a lot of the big box consultants that go into the Fortune 500, it's really easy to walk into someone else's domain when their domain is shopping because you all know what it means to shop. And even though you might not be an expert in retail, it's pretty easy for you to grasp that domain and build for it. If you've never seen the inside of an air operation center, there's no analogies I can give you. And I have tried, I've done many experiments with talent mixes and how to get this right. And there's just no substitute for having one, usually I would like to have half of the people on the team really understand the mission domain in a real way, including one person that actually lived it.

And then this one on the last mile piece, still seeing a ton of this where we saw, again, I feel like this is a solved problem and some people are going to talk about it today and some of the even better advancements we're making in how to build security in from day one and be able to deploy on demand with full compliance packages, but this has to be done. And I think if you get these things right, you should be able to offer a performance guarantee. So this is like a soft announcement. We're going to do a real announcement of this in September, but we're going to launch a performance guarantee and I want to challenge the entire community to do so. And I would say if you're about to go on contract with a vendor for a firm fixed price outcome and they say like, "Oh yeah, we've got the best talent.

Oh yeah, outcomes, outcome, speed to mission value. Yeah, security built in from day one." And they'd be like, "Okay, can you guarantee it?" And if they say no, they're lying about the other four. So you should be able to put your money where your mouth is. Now, the guarantee has to be conditional. And for government folks in the room, I'd be willing to share the draft SOW that we've written up on what firm fixed price outcomes should look like in software. So if you want that, feel free to reach out and I would love your feedback on it as well, but you can take it, share it, use it. You don't have to use it for Rise8. I want to build this out as a community of everybody should be doing outcome oriented contracting. But for us, we want to say, assuming there's no existing path to prod, so I said, if there is a path to prod, you should be able to do it in weeks.

But even if there's not, at least up to the secret level, top secret and special access programs get a little hairy, but all the way up to the secret level or FedRAMP high for the Fed civ folks, you should be able to get the first outcome in production in 180 days or less. If not, something is broken. That is the bar. People have done this over and over again. I know we have personally, we're not the only ones. There are people that are doing this. And so if we can't get that guaranteed, something's wrong.

And that brings me to the Mission Outcome Factory. I said earlier, the next evolution of the Software Factory isn't 2.0. It's thinking more holistically. I want to look to the left, mission need. How can I get into that space and start automating versus we're going to do workshops on day three. I love doing human interactions in these workshops, but there is a lot of work to go from that, synthesizing all of that data and turning it into the roadmaps and the backlogs that I talked about yesterday. So automating that upfront piece too, not just starting with the software delivery process and then on the back end of how we integrate it into the mission. But if we do this right, I think that we should be able to guarantee time to first outcome and then guarantee continuous delivery of outcomes thereafter. That should be the new bar for this community.

So if I just expand that, some of the things that I think have to be true, the first is prebuilt infrastructure. So I'm going to go in reverse order here. Number one I think is the most important, but I'm going to start with pre-built infrastructure. This largely is another solved problem I would say, but the brutal truth about government software right now is 60 to 80% of the effort on most programs go to those things that I said are below the value line. Undifferentiated heavy lifting, like infrastructure setup, security scanning, ATO documentation, compliance artifacts, CI/CD pipelines, environment config. I mean, sometimes just getting access, you get on contract and they're like, go use. I shouldn't name them. This agency's enterprise cloud or that agency's enterprise cloud or everybody's got some platform. And sometimes it takes two, three months just to get access to resources and have them provisioned.

Those are the things that we've got to fix, but a lot of people have fixed these problems. So we'll see many cases where there's 12 to 18 months of work on major programs before they can even start building mission capabilities. And so we want to eliminate all of that.

Now I mentioned earlier that. Sorry, let me. These top three, even though they're not the important thing, the mission metric is the important thing, they're like the table stakes that you have to achieve. They're the thing that make the mission metrics possible because they give you the feedback loops, the rapid feedback loops you need to discover mission value. And so I said the bar should be elite delivery performance, on demand deployments, less than one day for lead times with no trade off on fail rate or stability and reliability. And so we want to achieve continuous delivery and to do that we need this structured and opinionated path. So one of the things that's changing right now, in this space, is once we have... So they're going to talk about continuous ATO later on today, I think at the end of today's agenda. But if you look on the right side of this, I mentioned on the left, everything that is highlighted is something is provided to you and then app teams have to build or configure the rest. And so on the right hand side, you see what a compliance view of that is. Once we have established this maximization of inheritance that allows us to move into a continuous authorization to operate, we can now bring these modern dev tools into our IL-4 development environments much easier. So everybody is struggling right now to get AI into their development environments and government. And I would say the people that are doing it well or having an easy time with it are the ones that already understood continuous ATO. And if you haven't done that, you should set up your environments to maximize inheritance and have more of a continuous ATO process.

Then you'll be able to get your dev tools, your AI augmentation tools into your development environment much easier and the outputs of that, the software that comes out into production much easier. So I don't want to spend a lot of time on this, but when we talk about how AI is changing things, this is a huge area that we have to focus. I'm going to talk about elite teams next. I said you have to pair that technical talent with government domain expertise, but another thing I talked about these small balance teams yesterday is those teams need a new area of expertise besides product design and engineering, or maybe it would be better to say everybody, regardless of their specialty, needs to become a full stack builder now. Once I offer you those development tools, expect not just engineering is using them, but also product and design. There's something I want to say about this because there's a lot of skeptics right now and people are like, "Oh, it's not good. It's not working." It's kind of like if I gave you a bicycle, you don't know how to ride a bicycle. It's like, "Hey, here's this really great tool to get from point A to point B faster and you hop on and you fall over and you try four or five more times, you're all bruised and you've got cuts everywhere and you're like," Bryon, bicycles don't work. "All of you think that's absurd, and that's how the people that have invested the time learning how to ride this bike of AI feel when we hear people say," Oh, the AI doesn't work. "Generally speaking, now there are things that we're still learning, new methods, how to employ the AI in different ways to solve some of the stability and reliability challenges that I talked about.

So everybody's still learning, but we have gotten to a place now where we can confidently say that this gets us from point A to point B faster without those trade-offs in reliability and stability if you do it right. And so most of the use cases that I'm seeing where people are saying," Oh, the AI is not working, "is just because they haven't, and I was this person, Mike had to just get me in there, start coding. They haven't invested the time in building the skills it takes. And once you do, you'll see the payoffs there. Again, still learning, still time and improvement to do, but that's a big shift that has to happen. All right, now I want to get into the big part that I wanted to talk about today, which is the outcome-based delivery system. So we've developed this thing and we put it out into the community. It's free. It's not proprietary. We call it Mission O/S, but you can go on our website and I have videos explaining all of it, and we've put out articles and white papers and everything for the community on all of the things we believe it takes to continuously deliver valuable software that users love into production. Now the task is automating all of that and using AI to augment on the front end and the back end.

So Kent yesterday talked about futures versus features, and he said one of the ways that we maintain our optionality, that green line, is by pausing before we write the next feature. I think that we'll probably get to a place where we can do a lot more at once, but right now, some of the things that we used to always do as humans, like the Gherkin format for writing a feature is based on behavior-driven development, like what do I need the software to do? Then I write a user story. As a user, I want to be able to do X, so that X, and then I give a scenario given, when and then.

If you ask AI to do this before it writes the code, now there are people that are arguing that this is maybe inefficient in a future AI world, maybe we won't need to do this and it's a waste of tokens, but right now I would still advocate for this. So before you have it start writing features, have it use a Gherkin format for user stories and scenario. And then what's really cool and something a lot of people are doing, we're seeing a lot of success with it, is doing our user interviews and recording them, or you can do this live, and having the AI turn this conversation into this, a hundred of these. So I walk out of a one hour user interview and before I can even finish my bathroom break, I already have a hundred user stories queued up that we can start iterating on.

And then it's like, okay, well now. And by the way, I should say those user stories, we can also write tests from those and I still think that we should be doing a lot more unit testing and integration testing than I'm seeing when people are generating AI. So just say that. But now the bottleneck moves into all of this other stuff that I talked about of, well, how do I even know what user to go talk to, which actor? Well, I talked about value stream mapping and a whole host of other things to map the current state, and you're going to learn about these today. In fact, right after me, we've got mission value stream mapping and event storming and how we're applying AI to be able to do just like that user interview example where, yeah, we still got to go into the room in person usually and facilitate these discussions, but we can take all of that and what used to take us a week or for a complicated problem weeks and turn that into something that's ready to go conduct those first user interviews.

And so we're setting the target state, same thing. It's really good at identifying where the constraints are in the mission value stream in ways that humans have a hard time doing because we're all biased, especially if we do come from the mission space. That includes the impact mapping, generating hypothesis statements, the actual outcome oriented roadmap itself and what order you should do things in. Now I'm not saying that you should automate all of these and not review them as a human, but it is really good at all of these things, so much so that I think we're getting close to the point where we can go end to end in a week.

So if we put all of those things together and then we go to bed for the night and we wake up in the morning, we'll have generated what used to be the MVP that would have taken us six months that probably has first user value that can be deployed to production. And I know for some of you that seems really farfetched because you've heard all these horror stories and there's a skeptics like, "Oh, everything you're reading on Twitter or hearing in these conferences, most people are actually struggling with AI." That is true. There are still some real struggles that we're all having, but these things are real and I think we can do them today. And so if I go back to this mission outcome factory, what I want to be able to get to is, I mentioned the guarantee today should be 180 days.

Again, that's if you don't have an existing path to prod. If you do, it should be way shorter than that. But what if it's not 180 days? I think we can start pushing it down to 120, 90, 60. I think it is very feasible to get to almost real time where I can walk into a warfighter environment with my delivery lead, Becca, who I see here, and we can do a quick value stream and event storming workshop and the next day have software in production with a full ATO package that's ready to be tested by users. And hopefully, now humans get this wrong, the first iteration wrong, and the machines probably will too, but there will be many times where we actually deliver the first outcome with that. And that's the challenge I have to you all. Stop being skeptics. Let's figure out how to apply AI, not just to the software delivery part, all the testing that's around it, generating the user requirements, the stories, all the way up backlog, all the way back to understanding the mission value stream.

And then on the back end, which you're going to hear about at the end of today, making sure that all of the compliance and everything that we need to actually deploy to production safely and sustainably is there too. I really believe that this is possible and that we're there, and I'm looking forward to working with all of you over the next year to two years to make real time outcome delivery a real thing. That is the outcome factory. Thank you.

‍