Episode 03

Bryon Kroger (00:03):

In the last episode, we laid out the new goal. We talked about the strategy of risk reduction and the metrics that prove it works. But to make this real, we have to get crystal clear on our definitions because if we don't, the bureaucracy will use our own language against us. It will co-opt our terms and drain them of all meaning until they represent the very thing we're trying to escape. It starts with the most important word in our vocabulary: production. [00:00:30] A lot of people think production is just a label that you put on a server. They'll say, "Oh, I shipped to production." But what they mean is they moved code to a sandbox environment that no real user will ever see. So let's get tactical. If your software is running on a server rack in a lab and the only people using it are the test team, that is not production.

(00:52):

If you've handed off to another government program office, but it's not in the hands of the end user, that is not production. [00:01:00] Production is the messy, chaotic, real world where the mission happens. It's the cockpit, it's the operations floor, it's the analyst 's desktop. Production is where your software is put into operation for its intended use by its intended end users to solve a real problem. Several years ago, there were software factories standing up all across the Department of Defense. And every single time I would hear these unbelievable stats about like, "Oh, we've got 10 apps in prod, 15 [00:01:30] apps in prod." And I heard one platform claim that they had over 80 applications in prod. And this was just unbelievable to me because I know what that takes. And I just started digging in and come to find out they had an IL-4 or an unclassified environment that was labeled as production.

(01:50):

And every time an app was put in there, they said, "Oh, the app is in prod." But most of the apps that were there were not destined for an unclassified [00:02:00] IL-4 environment. In fact, most of them were destined for IL-6 or a secret level environment where most of the mission happens. And so you had all of these apps that were using IL-4 production as a staging environment before going to IL-6. Less than 20% of the applications that were "in prod" had actually made their way over to the mission floor. This isn't just semantics. This is the heart of the matter because our ability to deliver real outcomes in real production environments [00:02:30] is directly tied to our ability to succeed. The work done by DORA, which is detailed extensively in their book, Accelerate, established a direct causal link between what they call software delivery and operational performance or SDO performance, and an organization's ability to achieve its goals.

(02:49):

High performing DevOps organizations are four times more likely to achieve their business outcomes. And this is true across every industry vertical, including government. Why [00:03:00] does this causal link exist? Because organizations that are good at software delivery performance have built feedback loops into their DNA. They have automated testing that gives them instant feedback on code quality. They have monitoring that gives them instant feedback on operational health. They have deployment pipelines that give them the ability to get feedback from real users instantly. High software delivery performance isn't the goal. It's the output of a system designed to learn. This brings us to one of the biggest dangers any transformation [00:03:30] faces. It's called the alignment trap. When large organizations decide to go agile, their first instinct is to focus on alignment. They create endless committees, hold massive planning events and implement complex frameworks like the scaled agile framework or SAFe.

(03:48):

The alignment trap feels productive. The meetings are full. The PowerPoint decks are beautiful. Everyone is aligned, but it's a mirage. It's the illusion of progress without the substance [00:04:00] of delivery. It's like having a perfectly planned cross-country road trip with every stop mapped out, but you're still sitting in the car with no engine. They spend all their time and money in boardrooms debating what's valuable. The problem is they can't actually ship anything. So they get stuck in this trap where they're perfectly aligned, allegedly, on a plan that they have no ability to execute or validate. Their costs go up, their growth goes down. We've all seen this. My favorite example of this is a large ACAT 1 program [00:04:30] that was going on at the same time as Kessel Run in the Air Force, and the juxtaposition couldn't be more clear. You see, they had spent years implementing Scaled Agile Framework.

(04:41):

And as we were shipping our first apps, they were still debating theirs. They had agile release trains and architects and all of these personas. And I kept asking, where are all the software developers? Who's going to build all these things? By the time that I left Kessel Run, [00:05:00] they still hadn't delivered anything to the field. Nothing. The problem was even once they started building code, they had no way to get it into production and to validate it with real users. And so they failed. This is why our approach is different. We start with a path to production. We establish that well-oiled IT machine first. I don't even care what you ship initially. Ship a Hello World app. In fact, that's super valuable. You can prove that you can get code from a developer's keyboard to a real [00:05:30] user on a classified network in a war zone, because once you've established that path, once you've achieved continuous delivery, now you can have a productive conversation about value.

(05:40):

You have your first real feedback loop. If you and I disagree on what's valuable, it doesn't matter. We can run an experiment. We can test both ideas and see what the data says. We lose two weeks, not two years. This is the core of the Mission OS way, but when you start talking like this, you will inevitably hear the four [00:06:00] most dangerous words in any enterprise. It won't work here. They'll say, "That's great for startups, but we're the government." Or, "Our compliance rules are too strict. Our bureaucracy is too unique. It's a story that people tell themselves to avoid the hard work of change. Let's break each one of these down. Our compliance is too hard." Well, guess what? We've built a continuous ATO, not just once, but over and over again. It's been done. Our networks are too [00:06:30] slow. Well, we've deployed and disconnected low bandwidth environments.

(06:34):

Our people aren't ready. Well, Toyota once took the employees from the worst performing auto plant in the United States and made them the best in three months. One of the most frustrating versions of the "It Won't Work Here" arguments that I came across in the DOD is that we don't have the talent for this. Let me be very clear. There is no talent shortage. There's only a shortage of environments where [00:07:00] people with growth mindsets can learn and thrive. The data from the state of DevOps report covers thousands of companies, including nonprofits and government agencies. The principles are universal. The physics of software development don't change just because your building has a flag in front of it, and we have the ultimate proof. At Kessel Run and at Section 31 and at Bespin, they were told every single day that it wouldn't work. We were told that we couldn't get continuous authorization to operate.

(07:27):

We were told we couldn't ship code every [00:07:30] four and a half hours to Warfighters. We were told that we couldn't build a high performing culture inside of the DOD. The bureaucracy is the challenge we are here to solve, not the excuse that we use for inaction. The principles of Mission OS are not a Silicon Valley fantasy. They are battle tested reality, proven in the most complex environment on planet earth. It's worked for me and it'll work for you too. But to drive real transformation, we need to scale something more challenging than continuous delivery, and that's leadership. [00:08:00] That's where we're headed next.

(08:12):

One of the reasons that I'm so adamant about defining production very specifically is because I saw it lead to a ton of waste in the DOD. There was a period of time that I call the great platform wars where Kubernetes platforms were popping up everywhere across the DOD and they were all fighting [00:08:30] for resources. And how did teams go about getting resources? Well, they had to claim they were the most successful platform. How do you claim that? By claiming that you have the most number of applications in production, which isn't even a good metric, but nevermind that. This became the metric that the DOD was focused on, and the Air Force began pumping tons and tons of resources into a platform that was built by the government instead of a platform that was bought [00:09:00] from commercial enterprise. And I am a big believer that we are not in the platform business in GovTech, at least not yet.

(09:08):

That could change over time, but today we are in the business of producing mission value. And the value line on the tech stack, to me, is everything below the user facing applications and data. And so building platforms was a really expensive and [00:09:30] really wasteful initiative, and it was driven entirely by false claims of the number of applications in production.

(09:43):

When I left operations and arrived at Hanscom Air Force Base, the thing that I kept being told over and over again was going to crush all of my hopes and dreams was the ATO process. The Risk Management Framework. Probably one of the most gratifying things was finally being granted [00:10:00] an ongoing authorization for continuous delivery. What was so great about it is that we didn't need any special authorities or any waivers. We didn't have to violate any policies or, God forbid, law. We actually did everything the RMF says to do. It's like a masterclass in bureaucracy hacking. And we proved everyone wrong. You can go fast and stay compliant with Risk Management Framework.

(10:30):

[00:10:30] For a leader who is genuinely afraid that these methods are too risky for their high stakes environment, the first, most concrete step that you have to take is to reframe risk. What I mean by that is that the way that you're likely doing business today is actually risky. All of the bureaucracy and the process was built to reduce risk, but it's not actually doing it. It's just passing on risk to operations. [00:11:00] And what we can show with all of these methods is that they actually reduce risk in concrete ways. Every single aspect is an exercise in risk reduction. Do them one at a time in the order Mission OS gives you, and before long, you'll see that you have reduced risk far beyond what anybody in current operations is doing today and at the same time producing more value.