Project Metrics: The Double-Edged Sword
By Pam Stanton
We all know that metrics are critical to running a business. We need those leading and lagging indicators of performance, so we can change course where needed and keep doing what works.
Metrics, however, are a slippery beast. In addition to measuring the results of actions, they also motivate action. And this is where things can go either fabulously right, or terribly wrong. What’s worse is that we may not realize what outcome we are promoting until it is too late.
Here’s an example from early in my career. This experience seared onto my brain the awesome power of metrics to influence behavior. And with that power comes great responsibility to ensure that the outcome we motivate is the one that we want.
I was working at a large financial services firm as the relationship manager to the outsourced IT vendor. In this arrangement, the vendor would prepare financial estimates for each system change request from us, the customer. These financial estimates were then used to secure a budget to complete the work.
We had a huge problem. The financial estimates varied wildly from the actual project costs, with swings of plus-or-minus 75%. This caused heartburn for the business owners, because they couldn’t predict whether their funding requests would be wildly overstated or understated. I could understand the problem of being over budget, but I didn’t get why they cared so much if they came in under budget. That’s saving money right? Wrong! It was then I learned that coming in significantly under budget is just as bad, if not worse, than coming in over budget—because you have tied up company dollars that could have been allocated elsewhere.
My division’s Vice President had a solution—or rather, a demand. We would impose a new metric immediately that required the vendor to deliver each project at no more than plus-or-minus 10% of the original estimate. Failure to meet this metric would result in a financial penalty to the vendor.
Within 3 months, we had a perfectly green scorecard. It was amazing! All we’d needed all along to keep that vendor sharp was a metric! Eureka!
Or not? Even though I was still early in my career, I smelled something funny. I wanted to understand this power of the metric, and how it had changed everything about the vendor’s performance. Were they just now being more careful? Did the old “What gets measured gets done” adage apply? I had to know.
I kept poking around about this measurement. Finally, I got the answer I sought from one of my vendor counterparts. His terse revelation went something like this.
“Well, we’ve always gotten really vague requirements from the business side, and we still do. We always tried to do our best on the estimates, given limited information and broad assumptions. Now that we have to meet this metric, here’s what we do. We pad all the estimates to make sure we won’t be over budget. And then if it turns out that we’re running more than 10% under budget, we just pad more hours onto the project until we’re within the 10% tolerance.”
Ouch! We had gotten what we’d asked for. All projects were delivered within a 10% budget tolerance. Turns out that all projects had also become more expensive. Oops. We weren’t measuring that…
This lesson has stuck with me throughout my career, and I call on it whenever creating performance measures. It’s critical to analyze what behaviors will be driven by the measure, and how they may actually be counterproductive. It’s still true that “What gets measured, gets done.” The trick is to make sure that what gets done is what we really want to get done.
Pam Stanton is the owner of Heart, Brains, & Courage, LLC, the parent entity for “Pam Stanton, The Project Whisperer” and “Perspectives Gallery.” She has 25 years experience leading business initiatives of all types, focusing in corporate I/T.