Are You Measuring What Matters—or Just What's Easy to Count?

Most government software programs can tell you exactly how many features they shipped. Far fewer can tell you what changed because of it.

The honest answer for most programs is that no one has agreed on what "done" actually looks like—in terms of user behavior, mission capability, or measurable change in the field. Features get shipped. Demos get applauded. And the mission stays roughly where it was.

The fix isn't more process. It's a better question: not "what did we build?" but "what changed because of it?"

In this Mission O/S Live, Rise8 VP of Delivery Max Reele and Josh Seiden—author of Outcomes Over Output and Lean UX—bring that question to the table from two angles. Josh has helped top global organizations stop optimizing for activity and start delivering measurable change. Max has done the same inside some of the most constrained delivery environments in the federal government.

What they'll cover:

• How to shift from output metrics to outcomes your team can actually act on
• Why discovery is an ongoing delivery practice, not a pre-work phase
• What gets in the way of outcome-thinking in government programs, and how to work around it
• How to know if your AI initiatives are contributing to your outcome or just creating activity

If you're leading or delivering on a government software program and you're tired of measuring progress in features shipped rather than mission moved, this conversation is for you.

Max Reele (00:37):

Hey, good morning everybody. Hopefully everybody can hear me. Audio is working just fine. Really glad to be here today with Josh Seiden. He's helped shaped how I think about delivery throughout my career. I'm Max Riel, VP of delivery here at Rise8. Excited to be hosting this Mission OS Live series event with Josh. Josh is the author of several notable books. I'll say most notable for me is Outcomes Over Output and Lean UX, two books that I have on my short list of required greeting for anybody, especially us in the gov tech space that's trying to get to measuring what really matters to demonstrate your team's success. We are going to talk extensively about that during this time. Josh has spent an entire career helping organizations, specifically with this type of measuring problem and helping teams really use it as a point of leverage to getting to doing the right work and focusing on the right next problems to solve.

(01:36)
So with that, thanks Josh for being here. Much appreciated. I know you haven't done a ton of work with the defense sector. We just kind of caught up on that a little bit, but it hopefully is somewhat flattering to you that within GovTech, we've used your bodies of work as a foundational element to some of our thinking. Specifically here within Rise8 and the frameworks we follow within our Mission OS core practices are really built off of what we learned early on from LeanUX and outcomes over output. So welcome. Thanks for being here.

Josh Seiden (02:12):

Thanks for having me, Max. It's great to be here.

Max Reele (02:14):

Yeah. I'll say for everybody in the audience, please feel free to drop questions, whichever forum you're viewing from. Please use the link that you use for registration to be able to drop questions. Hopefully we'll be able to get to some of those. These MissionOS live series talks tend to go very fast and we don't get to as many questions as we want to, but we're going to do our best. So with no further ado, Josh, I'm going to jump right into some of the material and to learn as much as we can from you. I'll start out right away with the output trap. What we all get stuck measuring. This is probably what's top of mind for anybody who's joining this to learn from you. Fundamental confusion between activities and outcomes. That's kind of what we get stuck on all the time. So I had to say, what's your diagnosis for why output thinking is so sticky?

(03:09)
Why do we get stuck in this way of thinking? And what is a key hint to help us know that that's where we're stuck?

Josh Seiden (03:18):

Yeah. I think that part of it is it's more natural to think in terms of the stuff we're making. It's so much more concrete. I mean, I've been working with this idea of outcomes for a long time now. And most of my ideas still start not with outcomes, but with outputs. You're on the bus or you're in the shower, you're driving and you think like, "Oh, you know it'd be cool if we made blah, blah, blah," whatever that thing is that you think it'd be cool to make X. I just think it's kind of natural in the way that we think about things. I mean, I think there's other sort of systematic incentives in the way we structure our work. But just to kind of begin with, an outcome changing someone's behavior is, I just think it's a more abstract concept and the stuff we make is concrete.

(04:26)
So I think that's the first thing we need to overcome is just to say like, okay, most ideas start with an idea about the output and it's our job to work upstream to get to like, okay, what's the point of making that thing?

Max Reele (04:41):

Yeah, I couldn't agree more. And we work really hard internally. It's part of our Mission OS framework, as I kind of mentioned at the intro, to make sure that we're working backwards from the impacts we're trying to achieve. And we ask our teams to work that way in their software development practices. So hearing from you specifically, what should that feel like to teams, this kind of outcomes over output and how outcomes create a fundamentally different relationship between the teams and the work that they're doing, what will change in how the teams are talking about their work or planning their work?

Josh Seiden (05:25):

I think that one of the things that changes when you're thinking about outcomes is because they're more abstract, they sort of put us in an uncomfortable place. And let me tell you a story about my breakthrough about thinking about outcomes. And I've done a lot of work with kind of mission-driven organizations. You said, it's true. I haven't done a lot of work in the defense sector, but I've done a lot of work with mission-driven organizations. And I was doing a strategic planning session with a nonprofit based here in New York City and they were a team that operated a service and they'd been given the strategic priority for the year to improve the customer satisfaction scores of the service that they operated. And so they sat down with me and they said, "Look, this is our goal for next year is to improve customer satisfaction in this service and we don't know what to do.

(06:33)
" And that's an incredible starting place to be able to admit, to look around the table at each other and go, "We don't know what to do. " That's actually the key. And it's super hard to do because we have all of these incentives in our work to appear smart and to have the answer and to never admit that we don't know the answer. But the fact of the matter is, the more strategic our work is, the less likely we are to actually know the answer and the more likely it is that we have to discover the answer by doing the work. And so what we said was, "Okay, when we have increased customer satisfaction in our service, what will people be doing differently?" In other words, what are they doing now that causes a lack of satisfaction and what will they be doing a year from now that causes improved satisfaction?

(07:27)
And that behavior change, like framing it in terms of what are people doing now and what will they be doing when we're successful, that's the sort of question that opens up outcomes. But I think admitting that we don't know the answer and then living into that question and saying, "Okay, the thing we're experts at is not having the answer. We are experts in finding the answer." And that's a really big mind shift, but I think that's the kind of fundamental change in the feeling to answer your question, Max.

Max Reele (08:01):

Yeah, I love that. It kind of gets right to the kernel of be less certain and continuous learning, right? Making sure that we recognize that as we're approaching any problem for any of our clients or any product we're trying to build, that we don't have certainty on what we're building. And so the discovery process itself is not pre-work. Sure, there's part of discovery and framing that comes in before you start building, but then discovery is continual process.

Josh Seiden (08:29):

Yeah.

Max Reele (08:30):

Yeah, I appreciate that framing of it. I'm going to jump in a little bit here now to kind of high compliance and how do we operate this way in a world where specifically within GovTech, we deal with clients, shall I say, they're subordinated to the congressional budget and that I'm sure many people maybe wouldn't love to hear it discussed that way, but it's just objectively the facts of how we handle annual planning, specifically in the defense sector, but really for all gov organizations. And so when we're foundationally built on requirements-based contracts that are foundationally built on expending funds on an annual basis to make sure that the budget is being used, I'll say effective use of the budget, but potentially suboptimal, what is your best way of instructing teams to say, "You're in this environment where you're on a requirements-based contract, but you care more about delivering outcomes than you do about checking boxes on those requirements." What's your best way of suggesting this is the daily motions your team should really engage in to make sure that you're thinking about that correctly and not getting lost on a hyper fixation on what requirement is due this week or next week?

Josh Seiden (09:53):

Right, right. Well, so I want you to think about the experience of getting a set of requirements. The experience of getting a set of requirements, you look at these requirements and you never have any questions, you're just ready to go. That's never happened. It's never once happened, right? It's never once happened. You get a requirement and you're like, "Oh, okay. Well, now I have a million questions. When you want me to build X, should we make it like this or like this or like this or like this or like this? " There's always uncertainty even within a requirements driven process. And so one of the clarifying questions then is to sit with stakeholders and say, "Okay, look, I understand this requirement. I have to produce a system that meets whatever this set of requirements is, but there's five or six or 10 or a hundred ways I could do that.

(10:52)
" So what's the outcome we're trying to generate?

(10:55)
When we've built this system correctly, what are people doing differently? And so there's always uncertainty. I just believe that. And maybe there are cases where there's no uncertainty. Fine. If there's no uncertainty, don't use outcome centric thinking. There's no value there, right? But when there's uncertainty, and there's always uncertainty, clarify by saying, "Well, what's the outcome here?" And then drive towards the outcome. And that has two effects. One is it helps guide the work of the team, right? So instead of pursuing path A, which is all optimized for this behavior, which no one cares about, we're going to follow path B, which is optimized for this behavior, but it also creates stakeholder alignment so that if there's a conversation about requirements and if there's an opportunity to clarify or adjust requirements, we've had that conversation with requirements and we can have a reasonable conversation that says, "Look, we have to go back to these requirements and we actually have to adjust them." And so depending on the parameters of the project, like that adjustment, I understand it's subject to different kinds of governance, but that's the starting question, right?

Max Reele (12:16):

Yeah, kind of a universal truth, regardless of the high compliance industry that you're operating within. We got a pre-question in that I think is fairly relevant on this line of talking that I'd like to, I'm going to have to paraphrase it, so I apologize to whoever submitted this just because it's fairly long-winded, but they're working on a zero to one product build to replace a legacy system and their requirement basis is to get to a full automation of the legacy system. And stakeholders are completely focused on that for the final automated finish line. And so the question really boils down to where do you start and are there specific outcomes or metrics that you think these types of teams that are doing legacy modernization or rebuilds of what were kind of legacy business logic into a modernized tech stack? How would they go about successfully measuring the progress they're making there and talking about it with an outcomes orientation?

Josh Seiden (13:15):

Yeah. So I don't know the specifics of the project or the kind of specific contractual situation here that we're operating under, but I think it's really important when we're thinking about creating changes to programs, right? We're going to plug in an automation, right? People were doing this thing manually

Max Reele (13:50):

And

Josh Seiden (13:51):

Now we're going to be replacing that manual process with an automation. The question is, how do we know the automation is working? How do we know that it's successful? It's not just that it's passing QA, right? It's that there's an adoption metric, right? To me, that seems like I've got the automation running, but people are using it. People have migrated from the old manual process to the new automation process. And it would strike me that if there isn't a conversation about adoption, that we haven't structured the program correctly. So that's the first thing, right? We often talk when we're doing enterprise work, people say, "Well, we're building an internal system for internal users. There's no competition."

Josh Seiden (14:54):

And that's just fundamentally not true. Your competition is always Excel. And so you can roll this system out, but people are still going to have ... There's always shadow IT to worry about.

Josh Seiden (15:09):

The question is, in this scenario for this questioner, what's the shadow IT version of we finished the automation, but people aren't using it and do stakeholders care about that?

Max Reele (15:21):

Yeah, fair point. And particularly in the defense sector, a lot of times we're talking about operational users who are in some pretty austere environments and they're very happy to go back to pen and paper or a whiteboard. So you may not even be competing with Excel in that case. They're going with the simplest implementation possible. So you really have to build simplicity into the tools that we're providing for them. So I think your point holds really true for the use cases we're talking about for most of our client basis and probably most of the people on the call here. I'm going to jump in to the world of AI. I'm going to talk about the elephant in the room. How does the output trap become exacerbated in the world of AI? I've seen you've posted recently on this topic. I know it's something that you're addressing with your large scale clients.

(16:16)
Can you just give us a bit a snippet of the wisdom that you're seeing as this environment morphs as we're all becoming builders in the world of AI?

Josh Seiden (16:28):

So look, these new AI tools are really cool and they're really fun and they're really seductive and they make it possible to do things that just weren't possible just a few months ago. I'm building all kinds of side projects and it's just really fun and it's really powerful. It's not just fun. It's a real amplifier, but I think it's an amplifier without any judgment. So it amplifies the things that we're good at, but it also amplifies the things that we're bad at. And so if we tend to kind of think in terms of outputs, this amplifies that tendency. If we tend to skip discovery and validation, this will tend to accelerate or amplify the tendency to skip discovery. I don't have to go talk to users. I'll just ask the AI, does the AI ... Create some fake users for me, take 10 minutes, and then let's ask the fake users.

(17:31)
It's so much easier than talking to real humans, and it's so much faster. The problem is, we know AI is kind of essentially 100% hallucination. It doesn't actually understand meaning. It's just getting better and better at hallucinating. And as long as we like the results, we don't call it a hallucination, but it's all hallucination. And so it's our job as the humans to validate all of these things, and it's our job as the humans to bring our expertise. So if we're doing a responsible job of product management, program management or design or engineering, we are bringing the things that we know about being good engineers or product people or designers, we're bringing that to the table

(18:20)
And we are guiding the AI to help us amplify those things or speed those things up. But my concern is that these tools are only as good as the foundational knowledge of the operators. And so for me, my first few personal projects were all output based. And I've just recently started, I've built a system for myself that I'm sort of testing more broadly that is outcome based that starts with ... I've got a context file in my system that says, okay, when we start a new project, what's the outcome? And the LLM needs to interview me to determine the outcome. And then together, we're going to brainstorm our way through outputs as a way of achieving that outcome. And we're going to build the measurement framework into these artifacts as we go along so that we can track our progress towards the outcome as we build.

Max Reele (19:26):

Yeah. I love that. I mean, I think it's something that we're all wrestling with right now, especially as engineering leaders or technical leaders, is that everybody is building a lot more and they're prolific and they're fast. And so if we're building in the wrong direction and we're building a ton of it, we're just building really fast and with great quantity in the wrong direction. So I do like the way that you're kind of suggesting that we simplify that logic to say, make sure that you're consistently checking in on tight iterations, following the core principles to buy down risk along the journey and making sure that you're consistently checking in on what is the impact you're trying to have end game. I think that was a good way to dovetail into kind of another foundational principle that we've really adopted from your early writings about balanced teams.

(20:22)
You kind of touched on the specification that you would need out of all the different experiences, this diverse set of experiences across a balanced team. And we've been kind of hearing more and more in the tech sector that the people are becoming more generalists with the use of the tools. Engineers can begin to potentially start to PM their own projects. We can create agent personas for the users, forget talking. These types of things are starting to creep into the ways of working, especially when you're trying to move very fast and you want to reduce the burden on communication costs with the other humans that used to be involved in more traditional small size teams. So if you could shed some light on it, I know you've written recently that you don't necessarily believe that it should be a requirement for PMs to be builders, but you qualified it by saying they should be very well versed in the technology of their domain, but they don't necessarily have to be builders.

(21:21)
Do you care to perhaps expand on that one a little bit?

Josh Seiden (21:25):

When I became my first job as a junior product manager, I was given responsibility to manage the software engineering team at the company that I work for. This was in the '90s we were building ... I was working at a company called Kensington that made track balls and mice. And these folks made device driver software, kind of nerdy stuff. And I said to my boss who gave me the job, I was like, " I don't know anything about software engineering. How am I supposed to manage a team of programmers? "And he said," Ask them good questions.

(22:09)
""Okay, I'll try that. " And so I just made them explain everything to me and they knew I didn't know anything. They were like, "Oh my God, who's this guy?" But I learned about it, right? I don't know how to write device drivers, but I learned what they're worried about and what their choices are and explained to me the trade-offs and all of that stuff. And so I learned enough to have a good conversation with them, but not enough to do the job because it wasn't my job. The analogy today, I think, the thing that I've been thinking about is, and I read Christina Woodkey, who has written some great books on product management. Christina wrote a wonderful book about OKRs called Radical Focus. Christina said, "We define roles based on people's activity and based on their knowledge." And one of the really interesting things is, a year ago, if you went and you looked at what the activities of a designer versus a product person versus an engineer were, they were very different and the knowledge was very different.

(23:29)
AI is kind of changing the activities so that the activity that I do in Claude, for example, as a product person and the activity I do when I'm wearing my design hat and the activity that I do when I'm trying to build something as an engineer, which I'm very bad at, it's the same activity. I'm prompting the system, right? What's different is that I have this store of knowledge in some of those roles that I don't have in other roles. And I think that's going to be the challenge for us is not to confuse the activity with the knowledge. And so if I have a really good designer, really good product person, a really good engineer working together, they're going to bring those three different sets of knowledge and those three points of view together. And so even if the prompting looks very similar, the activity looks very similar.

(24:22)
In other words, the knowledge they're bringing to that role is critical and critically differentiated. And I think our job is going to be to try and maintain, to continue to bring those specific expertise areas into our work.

Max Reele (24:40):

Sure. So there is a place for the specialization and the depth of experience that comes from a product background or a design background entering into this heavily world now that everybody's building. It's critical. Yeah, I appreciate that. And I think that there's, I mean, we still operate with balanced teams within Rise8. And I think all of the people that we work with firmly understand that and we try and make sure that we're still instilling the value of that. But I think it kind of goes without saying that the makeup of that team might be morphing a little bit. With the amount that everybody's able to build and how quickly everyone's able to build, perhaps you don't anymore need a PM, a designer and a six pack of engineers, if you will. What do you think that the ideal team makeup starts to look like on a balanced team moving forward with the use of agentic development?

Josh Seiden (25:44):

At the risk of seeming old fashioned tiered. So first I'll say, I don't know.

(25:49)
I don't know. And I'm not sure anybody knows right now. The agentic development, its capabilities are changing so fast every month we're seeing stuff get more powerful and more and more powerful. So I think anybody who's certain about what this is going to look like in a year is either lying to you or very lucky if they get it right. But it's a problem that we need to solve, which is how do we bring this knowledge? When I sit down and I talk to my old friend who's a CTO about the way he's pursuing agentic development, he's thinking about dimensions that would never occur to me to think about. But similarly, on the design side, I'm thinking about dimensions that it would never occur to him to think about. And so how do we make sure that those dimensions are represented? I don't know the answer, but I think that in the absence of an answer, it's very interesting to watch teams try to figure it out.

(27:03)
And I think the collaboration, the continued collaboration piece is going to be the critical way that we figure it out.

Max Reele (27:16):

Yeah, that makes perfect sense. I think we have a lot of people that are a bit anxious about it. So everybody's diving in and learning and everyone's trying to be early adopters of the tools and maybe now we're belt curve adopters of the tools. But still, yeah, what you said, it drives a bit of anxiety that it's very uncertain. It's very uncertain what the new team structure is going to look like. And so we're all trying to fight to make sure that designers can stay true to core design practices and PMs can stay true to core PM practices. Meanwhile, as a leader of many delivery teams, I'm trying to instill within everybody that you're all technical now. There's no such thing as a PM anymore. Everybody's a technical PM because I'm wanting the mentality to exist that I will become fluent in the tools that are being used here, even if I personally don't have to be the one that's building and shipping to production, I should understand how this changes the magnitude of the work that the engineers are going through or how to prioritize the stories that the engineers are going through.

Josh Seiden (28:17):

I think if we think about it as activities and knowledge, I think that gives us a head up. We have no idea what's ahead of us in terms of activities, right? But what we do know is that the knowledge I bring as a PM or an engineer or as a designer is knowledge that doesn't live in the head of my peer, of my colleague. And so really standing on the value of that knowledge and saying, "We're going to bring that knowledge to the team and together we're going to figure out what these new activities look like. " To me, that's the platform.

Max Reele (28:54):

Yeah. I love your ability to distill down things that are hard to execute in practice, but you distill them down into these ways that are very, I should say, linear for us to be able to understand and then be able to actually put into daily exercise. You've done that before in your writing, and in fact, you kind of referenced yourself as potentially old fashioned earlier. I beg to differ. I think some of the ideas that you're pushing out here, there's some beauty in the simplicity of it. And I think that actually makes it something that is easier to adopt. In fact, you can even look at our website, which we fancy ourselves as very much we're pushing the edge of adoption on these tools for gov tech. We're pushing ourselves into new market segments as a result of what we're doing with the adoption of tech and getting everybody in our company to be able to become fluent in AI tooling and building.

(29:49)
Yet, foundationally at the core, our website is fraught with things that came out of your early writings, to be able to work backwards from impact to bite off small chunks of ... Iteration to reduce risk, balance teams as a core foundation so that activities and knowledge, as you put it, are represented all wholly on a balanced team. Josh, we're about at time here. I just wanted to ask any particular closing remarks about this move into the world of AI and how to get to outcomes orientation in this world?

Josh Seiden (30:25):

I would say there's two things. The first is we are all out here in the industry figuring this stuff out at this moment. I've been doing this for a long time and I've seen a lot of disruption and this one is crazy. This one is really like it's blown everybody up. So I just want to encourage everybody publish your experience reports. What you're experiencing now is super valuable for everyone around you. So publish your experience reports and go back to your fundamentals as product people, engineers, designers. That's what you know. And don't be fooled. The machines aren't going to replace that knowledge.

Max Reele (31:12):

Yeah,

Josh Seiden (31:12):

I love that. Those are my two call to action.

Max Reele (31:16):

Yeah, for sure. Call to action is exactly what I was going to say. Stick to your fundamentals. Make sure that you understand at the core that you have to stay true to your fundamentals and then publish the experience reports. Understand what you're going through. It'll force you to think through what your experimentation cycles look like. Yeah. All right, everybody. We're at time here, Josh. Thank you so much for spending this time with us, imparting some of your wisdom on us. And more over, thank you for your career's body of work that we've all been able to leverage to kind of do digital transformation and advancement and modernization within DefenseTech and GovTech writ large. Everybody who's listening in, thank you for joining us. Please follow Josh on LinkedIn, check out his books and his website. And please stick with us. We're going to continue to do these MissionOS live series.

(32:03)
I hope that each of these episodes have been really beneficial for everybody who's tuning in. We're going to stick with it. We love it. We love the core learnings that are coming out of these Titans from industry that we're getting to hear from directly. Lastly, I'll plug coming up in late August is Prodacity. We hope to potentially get Josh on stage and get to speak to us in a large venue, Nashville, Tennessee, so you know it'll be a good time. That's August 25th to 27th. Josh, thank you again for joining us. Thanks for having your time here today, and thanks everybody who attended online.

Josh Seiden (32:35):

Thanks folks. Thanks, Max, so much for having me. Appreciate it.

‍