Book notes: “The Phoenix Project” by Gene Kim

My heart lurches as all the implications sink in. I’ve seen this movie before. The plot is simple: First, you take an urgent date-driven project, where the shipment date cannot be delayed because of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date. The results are never pretty. Usually, the software product is so unstable and unusable that even the people who were screaming for it end up saying that it’s not worth shipping. And it’s always IT Operations who still has to stay up all night, rebooting servers hourly to compensate for crappy code, doing whatever heroics are required to hide from the rest of the world just how bad things really are.

But of course, now everyone knows that you don’t release work based on the availability of the first station. Instead, it should be based on the tempo of how quickly the bottleneck resource can consume the work.”

LM: What is the current bottleneck?

Wes mentioned we have a separate list for infrastructure projects and business projects. Are infrastructure projects another type of work?

“It’s like a free puppy,” I continued. “It’s not the upfront capital that kills you, it’s the operations and maintenance on the back end.”

He continues, “What use is it having all these offshore developers building features if we aren’t getting to market any faster? We keep lengthening the deployment intervals, so that we can get more features deployed in each batch.”

She looks up for a moment, reviewing her memories. She looks back at me and says, “Ever! Half the time we’re driving in the car, you have this distant look on your face. The rest of the time you’re clenching your jaw, like you’re reenacting some terrible meeting in your head. You never hear what I’m saying, because you’re so preoccupied by work.”

What can displace planned work? Unplanned work. So much of what I’ve been trying to do during my short tenure as VP of IT Operations is to prevent unplanned work from happening: coordinating changes better so they don’t fail, ensuring the orderly handling of incidents and outages to prevent interrupting key resources, doing whatever it takes so that Brent won’t be escalated to… I’ve been doing it mostly by instinct. I knew it was what had to be done, because people were working on the wrong things. I tried to take all necessary steps to keep people from doing wrong work, or rather, unplanned work.

LM: Other types of work: Business Projects, Internal Projects, Operational Change.

He’s chuckling as though he predicted this level of calamity would occur, which, come to think of it, I suppose he did, in the conference room when I first met him. Something about “clearing the calendar.” Just like clearing the change board, I realize. I kick myself for not picking up on his clue sooner.

If no one has slack time, WIP gets stuck in the system.

LM: leave open space (calendar, resources on the sprint) to allocate unplanned work

Index cards on a kanban board is one of the best mechanisms to do this (LM: create fast flow of work through Development and IT Operations), because everyone can see WIP (work in progress). Now you must continually eradicate your largest sources of unplanned work, per the Second Way.”

Book Recommendation: The Goal by Dr. Eli Goldratt.

Falling back on old training on handling frustrated people, I calmly reiterate what I already stated.

Steve is yelling so loudly by now that I’m holding the phone away from my ear. Suddenly, I’m furious. Steve is going to screw this up again. Recalling my days in the Marines, I finally say, “Permission to speak freely, sir?”

Oh, shit. That’s another part of “management off-sites” I forgot about. Touchy-feely crap.

‘Improving daily work is even more important than doing daily work.’ The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful. “Sensei Mike Rother says that it almost doesn’t matter what you improve, as long as you’re improving something. Why? Because if you are not improving, entropy guarantees that you are actually getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss.”

“A critical part of the Second Way is making wait times visible, so you know when your work spends days sitting in someone’s queue — or worse, when work has to go backward, because it doesn’t have all the parts or requires rework.

I take a deep breath, trying to decide what to tell him. A friend told me years ago, “To tell the truth is an act of love. To withhold the truth is an act of hate. Or worse, apathy.”

“When R&D capital is locked up as WIP for more than a year, not returning cash back to the business, it becomes almost impossible to pay back the business,” she continues.

As she leaves, I look at my Phoenix calculations again. We’ve spent over $ 20 million on Phoenix over three years. With all that WIP and capital locked inside the project, it will likely never clear the ten percent internal hurdle rate. In other words, Phoenix should not have been approved.

“Allspaw taught us that Dev and Ops working together, along with QA and the business, are a super-tribe that can achieve amazing things. They also knew that until code is in production, no value is actually being generated, because it’s merely WIP stuck in the system. He kept reducing the batch size, enabling fast feature flow. In part, he did this by ensuring environments were always available when they were needed. He automated the build and deployment process, recognizing that infrastructure could be treated as code, just like the application that Development ships. That enabled him to create a one-step environment creation and deploy procedure, just like we figured out a way to do one-step painting and curing.