Why Scrum Doesn’t Work in the Enterprise

Why Scrum Doesn’t Work in the Enterprise
By Curtis Guilbot

Stop me if you’ve heard this one before: A couple of Fortune 50 clients I worked with had recently spent a few million on training their development staff in “enterprise scrum” with a well-known author/trainer in the field. There were posters everywhere, town hall meetings (which were expensive all-day affairs, since this was a global company), a gung-ho spirit among many department managers. Guess what happened?

“Nothing,” you say? Worse. Something happened: the wrong thing. All that idealism of managers came barreling down from on high and crashed smack into the Reality Wall of security protocols, release gates, change management nebbishes, process labyrinths, disconnected architectures, documentation tomes, 16 levels of approvals, and oh, yeah, cynicism and resistance from employees who had seen it all before. Result? A thick atmosphere among developers, PMs, and most managers that “agile just doesn’t work here.”
Head fake

They’re right- scrum doesn’t work in their big organization. But that does not mean scrum can’t work there. I’ve made a lot of money in my career installing effective scrum methods on multi-million, even multi-billion dollar projects for these same hidebound companies, including the one in my example above. I typically see productivity gains of 400% in three to four months, by my clients’ own estimates.

So why are so many big companies stuck when it comes to scrum?

Gates of Hell

A Fortune 500 company whose e-commerce website generates $1 million per hour in sales must first do no harm. Black hat hackers abound, and juicy corporate sites are plum targets. Distributed global teams, which help keep labor costs down, require coordinated efforts among time zones and asymmetrical communication through some sort of system “gate.” So, gates exist, as they must, and often slow release speeds to a crawl. But they don’t have to.

Gates and (relative) speed can co-exist. Global corporations that depend on interconnected complex legacy systems can never move at the speed of startups- that’s snake oil peddled by training gurus. But they can move very fast, indeed.

First, decide which gates are truly useful. For example, most projects require a “sprint zero,” a pre-meeting-meeting phase that creates a lot of churn and no business value. Is this necessary? Could your teams simply go to the Customers, start drafting a product wish list, then begin estimations and iterative development right away? If so, you can likely shave 2-3 months off of every single project, even if you did nothing else.

On a more technical level, some things that seem like necessary precautions from a distance just don’t make sense close-up. Take database changes. If each database update is required to be done by an authorized DBA with traceable access, do you ALSO need 17 levels of approval of 42 types of documents filled out on 5 different systems before each release? Or is it enough to simply hold the DBA accountable if the system is broken and requires a fire drill to fix? That is a more “market-driven” approach to quality, as DBAs would quickly learn that it’s in their self-interest to review the code first. Further, if a break does happen, is it usually quickly traceable to the most recent local release on that system or a proximate system? If so, then it’s probably not necessary to make all of your global launch managers petition for approval and document every release in excruciating detail, especially since the nebbishes who are usually hired to vet that documentation are not programmers, have no domain knowledge, and are generally clueless about the release or it’s impacts. All they can do is verify that the 127 different steps were taken and 53 checkboxes were ticked. What value add is that?

Kabuki Theater

A popular gate in the enterprise is the requirements spec, but in fact, it’s more like a legal contract. These often come in many flavors for each: a high-level business case, a mid-level verbose document, usually assembled by a business analyst, and various technical specifications, architecture documents, mapping documents, database models, etc.

I ran operations for a private mobile application firm, and delivery specs were, quite literally, part of the contract that we made with our customers, so we took great care in crafting them. Much more care, in fact, than most Fortune 500 companies do.

After receiving this spec, the IT organization is usually asked to go into a cave for weeks, and emerge with a prediction of duration and cost to deliver. Despite decades of experience and research that tell us this is a grossly inaccurate way to estimate projects, the Kabuki Theater of project delivery will proceed from this premise: that the business has provided all of the requirements, and that IT has accurately estimated the project, based on these requirements. As the project deviates from planned schedule (which it will), communication between the Business and IT org quickly devolves into a courtroom battle:

Business Customer: “You didn’t deliver according to the spec and date!”
IT: “Well, your spec left out a bunch of important requirements details that we didn’t discover until late in the project!”

The result is acrimony and mistrust of the IT organization, and of course reciprocal mistrust of the Business Customer’s motives. The problem here is that requirements documents are treated more like binding legal contracts than like “wish lists,” which Business and IT must work together to refine and communicate to senior management.

Void ab initio

In fact, it’s worse than a legal document, because there is not an agreed adjudication method. In most courts, these requirements contracts would be considered void ab initio, or “null and void from the outset,” because every contract requires what is called a “meeting of the minds” before it’s considered valid. If I thought I was delivering X and you though I was delivering X+Y, then we never really had a contract, because we had no meeting of the minds. Most haggles over delivery vs. requirements spec that I’ve seen were void ab initio for this reason.

So how does a CIO make their cumbersome enterprise IT organization more nimble?

Reality wall

The proper starting point is not to assume that the reason scrum fails is lack of effective training. The right starting point is to ask “Why? Why doesn’t scrum work in our company?” A standard scrum trainer or “scrum certified whatever” is next to useless in answering these questions for you. Find a consultant, not simply a trainer, who is used to working with senior leaders, diagnosing intractable problems, and innovating with clients.

Trainers are in the “fill a room” business. Certifiers are in the “certification” business. Neither is particularly qualified nor interested in helping you (the enterprise leader) improve your organization. Doing that takes understanding the specific goals of your leadership, the inherent barriers in company process, and the unique mix of people, deadlines, budget, technology, and other constraints that you face. Trainers and certifiers don’t do this because (a) most are not qualified, and (b) it’s not their business model.

Much scrum training, for example, assumes that the entire organization will conform to the “best case scenario” when employees leave the classroom and attempt to implement their new Magical Scrum Method in the field. Are they in for a shock! Turns out the other 37,000 people in the organization didn’t get the memo.

Delivery Smart or Delivery Stupid?

A recent CIO Magazine survey of thousands of top technology execs said that their biggest challenge was “proving that the IT organization adds value.” Wow. In a world so dominated by technology, you would think IT’s value-add would be a foregone conclusion; instead, it’s a questionable proposition, at best. This explains why more and more CIOs now report to the Chief Financial Officer, instead of directly to the CEO, or even to the Chief Operating Officer.

So what can you, as an enterprise IT leader, do to implement scrum in your skeptical organization?

You have smart people. In fact, some of the smartest people in the world, by definition. But they are delivery stupid. That is, they do not know how to defeat the corporate handcuffs placed on them by the change management process. That’s not their fault; it’s yours. Give them the tools they need to unlock those shackles and get busy.

Execution Eats Training for Breakfast

Paraphrasing Peter Drucker, ground-level execution will override any and all utopian visions imparted by trainers whose business model is to charge you by the head for a few days of training, pat you on the head, and wish you “good luck with that implementation thing!” And then charge you again in 3 years for RE-certification in their impotent methodology.

To understand how to implement better, you must understand intrinsically how implementations happen now. If you’ve been in a managerial role for more than 18 months, your understanding is out of date. Enterprises are generally slow to deliver software, but they seem to change processes, regulation, and systems quicker than a duck on a junebug, as we say in the south. In many companies, each attempted release to production is a game of discovery: this gatekeeper interprets the rules this way, but that gatekeeper sees them differently. This is a major reason that big companies cannot ever get past CMMI Level 2 (reactive) to having defined, repeatable processes in practice, despite what the SOP says.

Now, Managers, Directors, and VPs are not going to jump into the middle of a project and start personally managing a launch. They have no time and no interest in doing so. But neither does your Trainer or Certifier. Instead, find someone who can jump into the trenches with your teams, identify the unique challenges, and define a process that works in your organization. Then have that same person do the same with another team, and another, collecting lessons learned from each, and propagating effective implementation across your IT staff in the trenches, rather than from on-high.

Enterprise IT delivery is not about IT (!)

To effectively delivery enterprise projects, your project manger (aka “scrum master” or “team leader” or “Swami”) must do three things:

• Establish authority, accountability, and trust with the delivery team;
• Establish rapport and trust with the Business customers and leadership;
• Define an iterative process that works within your corporate constraints.

Precisely none of these things is dependent upon a particular methodology. In fact, the immovable objects of corporate IT will squash 98% of most well-meaning agilistas or waterfall evangelists.

Successful software delivery is not about methodology; it’s about trust. The key to that trust is communication, not requirements contracts or a more novel method. Building trust requires deep communication expertise- interpersonal, small group, large group, public speaking, presentation, written and verbal skills. The AT&T Bell Labs Research Center, one of the most technical environments in the world, found in an internal survey of “superstar employees,” that communication was the single most important factor in a person’s effectiveness and rise through the ranks (read Robert E. Kelly’s excellent book How to be a Star at Work for more on this).

More liberal arts, less STEM

Most technology workers, from developers to project managers, to leaders, have little or no training in communication. They have even less in sales. They think the “retrospective” idea was originated in the 1990s by the founders of the Scrum Alliance. It actually dates back to St. Ignatious of Loyola, founder of the Jesuit Order, and was used extensively by the RAND Corporation think tank in the 1950s under the name wide band Delphi. They don’t understand the powerful psychological value of framing an issue in a particular way. Nor do they realize that the founder of the Waterfall methodology E.A. Edwards was actually a proponent of “adaptive” (iterative) development. To effectively deliver software at speed in the enterprise, you need more than a certification and a technology degree; you need to be a confident, insightful, and effective communicator with enough appreciation of human psychology, business, and history to not repeat the mistakes of the past.

Is your IT organization more valuable than shrubbery?

No institution in the Western world is under more fire than the state schools system (rightly so). This one-way communication model (teacher-to-students) was designed to assimilate rural farm workers to urban life and prepare them for assembly-line jobs of repetitious manual labor. But we don’t build software the way Ford built Model Ts. And yet, this same pedagogy is employed in most IT training programs.

Leaders who choose “solution-in-a-box” training programs often make price the primary decision criteria. They shop for consultants the way that facilities managers shop for lawn care companies to trim the hedges. In fact, many major corporations spend more on upkeep of plants than they do investing in their smartest people! That’s ridiculous. Improving IT delivery capability is a long-term strategic investment, not merely operational expense.

If your IT department is at least as important as, say, the company delivery trucks, then treat performance improvement like the capital investment it is. Scrum and other agile methods can work in your organization, but you’ll need an informed approach, coaching in the trenches, and experienced help to overcome corporate hurdles.

With average improvements like the ones I’ve seen (300% – 1,200% within six months), the payoff is well worth the investment.

Curtis Guilbot has helped Fortune 50 companies implement new processes and technology with ROI of $2 billion dollars. He’s helped mid-sized services firms generate $2M per year in new revenue with products that did not exist before. He’s helped entrepreneurs and start-ups perform “strategic pivots” for higher top-line revenue, and greater personal alignment. And he’s coached senior executives to become more effective through building their management and communication skills, confidence, and operational tool kits. You can read more from Curtis on his blog. You can contact Curtis through his website.