Mike Gehard, Software Engineering and AI Enablement Lead at Rise8, and Nicky Pike, Field CTO for the Americas at Coder, turn the stage over to the audience. The first question sets the agenda: if AI has made code generation nearly free, where does the constraint go next? Nicky compares it to an instant flight that still lands you in the same security and immigration lines. Mike makes the case that outside-in test-driven development will deliver both speed and quality, and both push for shared, specific language about practices like TDD and spec-driven development. They trade stories about AI amplifying good and bad processes, from Stripe's fast path to production to a company whose AI rollout surfaced 57 separate secrets managers, and close with a call to run experiments, share the failures, and treat the work as engineering.

Mike Gehard:

I don't know how we follow that.

Nicky Pike:

Man, following Russ.

Mike Gehard:

I cannot read the dictionary and make you all stay awake. My name's Mike Gehard. I'm the software engineering and AI lead at Rise8. I've been running software for about 25 years, mostly startups, mostly in the private sector. So a lot of this government stuff. I'm about a year and a half into my journey, which has been interesting. Ex-Pivotal Labs, for those of you that know the lab's name, I use LLMs daily. So Russ talked about getting down there and being with the LLMs and feeling what they're doing. I do that on a daily basis. I'm also tinkering with software factories, so you might hear a little bit about my theories about Software Factory, and I'm joined here on stage by Nicky Pike, the field CTO for the Americas at Coder.

Nicky Pike:

Hello everybody. So my name's Nicky Pike. As he said, I'm the field CTO for Coder. Started all of this. I was in the Air Force as a 141 crew chief, got to spend a lot of time reading books. That's how I got my start in technology. Went from, did everything from the dot-com era. I was one of the original members that helped build and design Xbox Live and was release manager for that all the way through Xbox 360. Was also a former pivot, worked at VMware where I was the face of cloud foundry for a long time. Went in, built public cloud for CVS Health and worked at VMware trying to do the same things and now we're bringing development into a better, more consistent way for developers with Coder. So glad to be here.

Mike Gehard:

Sweet. As Kent said yesterday, asking questions is a form of leadership, so I want to open this talk to see if there are any questions. Our goal here is to send you back to work on Monday and have impact. So we could sit up here and talk probably hours based on the hour we talked at lunch today about what we're doing, but I would love to hear from the crowd if anybody has any burning questions that would kick us off that we can answer for you so you can go back Monday and have an impact.

Nicky Pike:

Don't be shy. Come on.

Mike Gehard:

Bueller. Got to yell it Drew. No microphone.

Audience Member:

Okay, cool. We talked a lot about constraints, especially today and optimizing for constraints in the system as a whole rather than locally optimizing. I think we're all on the same page that AI has really obliterated the constraint that is actually writing code. I'm curious from your perspective, what is the next one? What is the next limiting factor for getting code into production, especially high compliance domains like government?

Mike Gehard:

Okay, so let me restate that. AI has pretty much made code generation free. And I say code generation because humans meet sticks on keyboards, we're really not doing that.That has removed really, in my opinion, the biggest constraint that was in software is humans typing code. Where does the constraint go next?

Nicky Pike:

So I do think.

Mike Gehard:

Did I get that right, Drew?

Audience Member:

That's right.

Nicky Pike:

Well, so I like to describe this. When we looked at this, right now we're seeing all of the focus beyond code generation, but at this point it has. It's become free. It's become instant. I like to give the analogy of right now software development is a lot like taking a trip through the airport. The plane trip was what everybody, they planned for. That was the long part. But AI has brought us a teleporter. We can get from New York to Tokyo in terms of code generation almost instantly now. And...

Mike Gehard:

Hopefully we've got all our fingers and toes.

Nicky Pike:

That's right.

Mike Gehard:

Theoretically, we've got all our fingers and toes.

Nicky Pike:

But what we didn't fix was we made that flight instant, but we still got the same TSA lines. We still got the same baggage checks. So one of the coincidences of that is, is when flying becomes free, everybody wants to fly. So now all of the code, we're seeing this 10X, 100X of what we're seeing, GitHub is having issues because of all the code hitting it. We haven't fixed that outer loop. We haven't fixed the positions of how do we get the code that we generated through security checks? And more importantly, how do we get that outcome into production? We haven't fixed that outer loop yet. We've simply moved the bottleneck. So we can get from New York to Tokyo instantly now, but now we're spending 20 hours sitting in a TSA line with our shoes off, waiting for the ability to get into the teleporter.

Mike Gehard:

Or we show up in Tokyo and we're standing in immigration for three hours because the lines are now out the door because everybody is going instantly from New York to Tokyo. So yeah, I love that. I mean, the theory of constraints talks about you find the constraint, you fix it, and then you look for the constraint where it goes. And I think you touched on it. Either it goes upstream to specification. So what are we building? What outcome are we hoping to have with that thing? Magic happens, code gets generated, it goes out the door. Does it work? Did the machine, Russ talked about the habitat, does it work? Did the clanker actually build the thing? And then does it have the impact, I hope? Do my users use it? Do they have the outcomes they have? So now we've moved, we've taken one bottleneck and we've probably made it two.

So now the question becomes, well, what do we do with those?

Nicky Pike:

Well, and I think it changes the way that we have to think about software development. So we had this conversation earlier, what takes on the new method? When people were sitting there writing code, they would write code is what came through their mind. Yes, we had specs, but when we're talking about AI that can now generate this code almost instantly, how do we prevent it from slop? How do we prevent AI from giving us bad results? And I think Russ even mentioned this. When somebody tells me, "Well, AI gave me crap back." Well, that tells me more about what you told AI to do than it does about the AI that actually went and wrote it for you. So we got into discussion, test driven development, spec driven development. What does that mean to developers and how are they going to be able to bring that into the industry now that we have AI writing this?

And I think the answer is we're still figuring it out. There's good parts to test driven development. There's good parts to spec driven development. And then we have the nomenclature problem where when I say TDD, it might mean something different to all of you, but we have to come to how do we work with AI to give it the best set of instructions so that it can go in and write what we

Mike Gehard:

Want it to write? Yeah. I mean, Kent talked about it yesterday. The man wrote three books on software development and he said, take those three books and you just throw them in the trash and we start over. So I am a huge fan of test driven development. I spent years, let's say seven, eight years of my life pairing in person with another human, writing a test, watching it fail, watching it go green and then doing my refactoring. And I will tell you right now, I do not believe that that inside out TDD loop is the best place for us to put our focus. I'm a huge fan of outside in TDD. So let me define executable tests at the outside of the system, doing things that my user would do, the CLI commands they would use, the buttons they would press on the screen. I'm going to write those tests first because now I know that the thing works and then I can kind of mess around in the middle.

Maybe I let the AI generate some code and maybe it's slop, but it gets me to my first deliverable in front of a customer quickly and Kent talked about futures versus features. I now have a feature that I can show a customer. I can get feedback on that thing. And while folks are off doing customer interviews, maybe I tell the machine like, "Hey, I need you to get me some futures here. You just got to keep this test suite green. You can do whatever you want in the middle. You keep the test suite green and you're good." Maybe I write some architecture tests that tell it, "Well, I want to do functional core imperative shell." No comments in the code because we all know the comments go stale when humans write them, when AI writes them, good God, they may not even make sense when you write them.

But I can now get things faster, but I'm relocating the rigor that we all know has to exist to places and using the machines to get specific things. And that's why I believe that outside in TDD, you heard it here first, will be the practice that gets us both speed and quality. But again, I could be wrong. I don't know.

Nicky Pike:

Well, and you're going to hear a difference. So we've got software engineer here. I come from a platform engineering background. I'm going to bring up an old pivotal phrase. I do think that we've got to start looking at software as outcome based. We've got to get back to the outcomes. Now, a lot of people say, "Well, yeah, we do that today." Well, do you really? A lot of software engineers, we've got code drift. They start to sit down, write a function, whatever that specific thing may be. Then they think of, well, I'll add this in there and I'll add that. When you're talking with AI, you do not want AI to have to make those kind of decisions. You want to be very specific. I want this button green, nothing else. I want it to go compare these two dates, nothing else. And we've got to get back to the outcome of what we want in software and how can we get AI to give us those outcomes? Now, that may be TDD for another person. It may be spec driven. For another person, it may be a combination of the two. I think that it's going to be as variable as the frameworks that we use today, but at the end of the day, you really got to start focusing on what do you want the software to actually do, no more, no less, and provide that to AI very specifically.

Mike Gehard:

Yeah. I mean, I think it's that specificity that gets interesting. So domain driven design, Paul talked about it earlier, talks about a ubiquitous language. When I say TDD, do I mean outside in TDD, inside out TDD, Detroit style? There's I think like a UK style. I forget. Russell, correct me later. I'm sure. When I talk spec driven development, what am I talking about? Am I talking about 10,000 lines of markdown? Am I talking about formal methods? So I think this is the key to having these discussions, and I think these discussions are important, is we need to throw out the old language we use and get very clear on the mechanics of what we're talking about. So not just getting clear with the AI, but getting clear with the other human. What do you mean when you say TDD? Do you mean writing a unit test and then only writing the code to make it pass and then doing refactoring?

Because that is the definition of TDD. Kent's in the room and he corrects me if I'm wrong, but that is the definition of red, green refactor. I don't know if the clankers can do that, but if we're talking, I need acceptance tests that are fully executable, anybody that does continuous delivery will know that executable tests that tell you when you can go to production are kind of a prerequisite for continuous delivery. Those are the discussions we have to have so we're all talking about the same things. And then we can get clear with the agents. Then we can tell the agent, well, I want you to write tests first. I want you to write those in a given when then syntax. I want you to use the domain language when you write these tests. You can get really clear on what you need them to do, but until you're clear on that, now it's just going to start guessing.

Nicky Pike:

I do think it's interesting because again, I think AI gives us a lot of optionality in how we can work with it. We just heard, I know you're not a fan of Markdown. I know people out there that love Markdown. They can read that like it's a picture in their brain. Does that mean that we should use some new version? Is it YAML? Is it JSON? Is it Markdown? I don't think the answer matters as long as there's consistency there and as long as you're applying it to an outcome that you want to do. But if you start being inconsistent with AI, AI is going to read that, it's going to amplify the chaos you're giving it, and it's going to give you chaos right back. So I think it's more about focusing on the outcomes and really getting very specific about your requirements and making sure that you're consistent when you give them to them.

Mike Gehard:

Yeah. I mean, Russ had a slide. We were backstage getting ready and Russ showed the slide where it's an amplifier. So I love that point of if you have a crappy process today and you throw a bunch of AI at it, you're going to have a lot more crappy process, a lot faster. If you have a really good process, so the one I use, the example I use is Stripe. They have written a system called Minions. They brag that they put 1300 pull requests into production in a week. Now they don't tell you what those pull requests look like, so maybe their documentation changes, but a financial services company putting 1300 pull requests into production in a week, they didn't just build that system for agents. They had a system that allowed humans to move code from story to PR to production well before they had agents.

They now just leverage that existing system of moving fast to begin to incrementally, I'm guessing, turn up the knob from four to five to six to 11 to get there to add business value. And I think that's the big piece. It's not about generating code anymore, that's an output.

Nicky Pike:

Yep. Well, and I'll give you the opposite side of the story. So I had somebody on the podcast was telling a story about how a company brought AI in and immediately, I mean, it was within the first month that the C-suite said, "You got to throw this out because it leaked secrets." Well, they went in and took a look and what they saw was is that this company had 57 different secrets managers within their environments. Why? Because they allowed their developers to go and pick their own things. Well, when they put AI into that, AI went out and looked and saw these secret managers and said, "All right, I'm going to take this up to a hundred." And that's ended up how they ended up getting these leaks is because all these different secrets were in different places, there's no standardization. So talking about that amplification, if you come in with bad processes, you're not going to make your processes better with AI.

You're going to make them worse. It's guaranteed. And I guess if we're talking about things to take home, right now there is a rush for people to get AI into their systems and AI is extremely helpful, but you have to go in and take a look at your processes. It's the old platform analogy of you can't automate what you don't understand. You have to go through and do it first, then you can go through and automate it. And I think it's going to be the same thing with AI. Well, I don't think I know. You have to understand your processes. You have to correct your processes. It doesn't mean you won't iterate them on the future. And as you find mistakes, to Russ's point, AI is going to tell us what our mistakes are. We have to take those and fix this, but start off with at least a better starting point.

Understand your process before you get going.

Mike Gehard:

Yeah. And Russ talked about technical debt, cognitive debt and intent debt. He said that AI is really good at technical debt. So maybe the first step when you go back to the office on Monday is, well, where can I use AI to help expose my technical debt in a very non-judgmental way? One of the speakers earlier talked about making mistakes, just own it. We all got technical debt. We've all used that credit card way too many times. So maybe that's the first step is how do I improve my cognitive understanding of the system by using AI to go and find the places that I'm not doing a good job? That's a great one. I love using AI for triage. This is the thing that I have found it really good at. This thing is broken. Here are a bunch of log files. Can you go find the needle in the haystack for me and help me understand why this thing broke? So what am I doing there? I'm getting deeper cognitively with the machine using it to help me understand. I could go sift through those 100,000 log files. It would take me two days. So I think there's little things you can do that are not code generation. They are not slamming more code into production, but they're helping you gain the system and understand the system. The intent thing will be interesting. I think even when humans were writing code, we struggled to record the why we were building things. And I'm curious to see how we overcome that in a future where decisions are being made faster, you maybe have less humans on the team, things like that.

Nicky Pike:

Well, and you bring up a good point. I'd love to get your take on this from the software engineering side. I do believe one of the first things that we got to do is kind of check our ego a little bit. I love the fact that one, people are saying, "I don't trust AI to write code. It makes mistakes. Show me one human that's written code that hasn't made mistakes." Well, if you don't feed it good instructions, we get bad stuff out. This is why we have requirements in the human development world. Check your ego. It happens every day. People start blasting people for writing things. Well, that was clearly written with AI. I'm at the point now, I don't care. I don't care if you want to delve into something and you use M dashes. Is what you said correct? Why do I want to sit down and type out a 2000 word blog when I can have AI do the same thing for me in five seconds?

And I think it's the same thing with code, but you've got to check your ego a little bit, understand that we're not talking about anything different now than we have with humans, but it is at a faster scale. And that comes with some things that we've got to consider, that comes with some decisions that we've got to make, and we've got to set ourselves up to accommodate that machine speed.

Mike Gehard:

Yeah. I think that's the key. I have an ex-boss of mine, a guy named Chad Fowler, big in the Ruby community. I worked with him back in the day. He's actually working on a book for O'Reilly based around an idea called the Phoenix architecture. What would it look like if you never changed your code base, you just rewrote it when the specification changed? So the speed thing like, oh, we do immutable art infrastructure these days, the whole pets not cattle thing with servers. What if we did that with our software? And the first speaker talked about these journeys we take of what if, Why wouldn't we go try to make that happen? And maybe we come up short, but we learn along the way. And I was talking to Chad and I was like, dude, this is not so, but we now have the ability to think about this. So I think it's going to be really interesting as we move into it as the what could happen without cutting off the, to your point, well, it doesn't work, so I'm going to go, I'm not going to try it out. We're engineers. Engineers take really hard problems like putting people on the moon and putting satellites into space and we make them happen or flying machines when we're pasting feathers. So I think there's so much of that as like, we'll go kick the tires on it and see if you can make it happen. And if not, you've learned something.

Nicky Pike:

Couldn't agree more. I mean, this is one of the things that I find so exhilarating and fascinating about this is what is going to happen with software because of AI. I'm not talking about slop and I'm not talking about generation. I'm talking about where we can take the ideas that come from the minds of people that do not have the ability to do that. So for example, you go out and talk to anybody that's out in the field, you go and talk to the war fighter out there, you are going to hear honest, constant complaints about the software they use. This is written by people that aren't actually in the field and they're not doing this stuff. Well, now AI is given the capability for those people that are out there that know what the software needs to give them, that know what it needs to do, they can come back and take that knowledge. And even though they didn't have the technical skill, they can use AI to better that software. This is the new wire frame. We are able to take ideas from people that have no skills to write that software themselves and make working software, right? Moms are able to create home management apps for themselves that work for them. This is how we're going to see the next rush of software coming out because that provides working prototypes. And what stopped them before was, well, we don't have the money to go out and have somebody code that idea for me so I could take it out and try to market it. Well, AI does that for them now. This explosion of software that we're going to see, I mean, we're already starting to see it, but over the next couple years is going to be enormous and that is going to lead to new concepts, to new what ifs.

I mean, we were talking about, is it possible that AI becomes the end of open source software? Why do I want to bring dependencies into my software when I could write that software for free with AI at runtime? Now, there's a whole bunch of discussion that goes on there, but I mean, that's the kind of what ifs that people are looking at now.

Mike Gehard:

Yeah. I think it's that we owe it to the industry like Kent Beck did back when he codified XP Explained. We owe it to the industry to be running these experiments. We owe it to each other to be talking about the experiments we're running so we can go back and analyze the failures and be like, oh, so-and-so tried this. And not in a vibe coding, well, I tried this thing and it didn't feel right. I ran a test. I collected data. The data is worse. So I'm working on this personal software factory. There's a new benchmark called Slopcode Bench that has come out. And the coolest thing about Slopcode Bench is that most previous eval suites only had AI generate one iteration of the software. This one will actually have the AI iterate on the software for multiple checkpoints. So here's a set of requirements, go satisfy these requirements. Okay, cool. I'm going to judge it on correctness and I'm going to judge it on quality. I'm then going to run the next checkpoint. Oh, guess what? Here's another set of requirements. Does that sound familiar to anybody? I want you to implement these requirements without breaking the ones you just built and it measures, did it break anything? Did it test the new requirements and what did the quality do? And it'll step it through four or five checkpoints. And I'm actually using that benchmark to eval, the Palantir gentleman spoke about this, evaluate when I make a change in my prompt, what does it do to my software factory? Does it make it better? Does it make it worse? Here's the numbers. It increased test coverage X percent. Oh, the quality went down a little bit, but it's still okay. And now I can go, well, now I want to improve quality.

So now I'm going to go run an experiment of what sentinels and what guardrails do I need to put in to bring the quality back up so that the third iteration does not just drive this software into the ground and I run out of futures after four iterations. And I was like, now we're doing engineering, ladies and gentlemen. Like that's engineering, not like, well, this feels a little bit better, so I'm going to check this prompt in and we're going to call it good. And

Nicky Pike:

See, and I would argue that that's probably something that we should have been doing all along, long before AI come in, but this is innovation, right? With the speed, because when we were doing this with human development, it wasn't as much of a risk because we had time. When we look at this holistically, it was slow progress to get that done. We were able to go in and make some of these adjustments before they came real. Now when we're talking at the machine speed of this, we are coming up with these processes, these tools, these benchmarks that arguably we should have been doing a long time ago, but they weren't a need because we weren't doing them as fast.

Mike Gehard:

We weren't generating code as fast, so we could kind of course correct along the way. So you and I could sit up here all day, my friend. The red light is flashing. Aaron told me, don't make me red flash you. So I am going to end it here. Happy to talk about this. This is stuff that I love talking about. I think it's our duty to run the experiments. So come find me. We can chat some more and we'll go from there. Yep.

Nicky Pike:

Same. We'll be out in the crowd. Please come say hi. Thank you.

Mike Gehard:

Thanks y'all.

‍