Scrumfall: When Agile becomes Waterfall by another name

By Jeremy Duvall, for the CIO Online
This story was originally published on CIO.com on April 25th, 2022.

Businesses reasserting outdated development principles are undoing not only the promised product benefits of Agile, but also its human-centric principles. It’s time to recommit.

It was once believed that Agile was the great hope of humane project management. But the “Church of Agile” is now being corrupted from within by institutional forces that have refused or been unable to adapt to the very “radical humanity” self-baked into its supposed success: collaborative, self-organizing, cross-functional teams. Now a hollowed-out husk of Scrum conceals the same taskmasters of much-maligned Waterfall methodology, which has created a Trojan Horse of betrayal.

With Waterfall methods masquerading as Agile’s beloved Scrum — call it Scrumfall — businesses are reasserting top-down micromanagement, fortifying silos, fettering flexibility, and commoditizing software engineers as mere cogs cranking out code. This sham of true Scrum is undoing not only the promised product benefits of Agile, but also its human-centric principles emphasizing trust, collaboration, respect, connection, and sustainable work environments.

Agile wasn’t supposed to be this way.

The broken promise of Agile

The Agile Manifesto is short, sweet…and revolutionary:

• “Individuals and interactions over processes and tools”
• “Working software over comprehensive documentation”
• “Customer collaboration over contract negotiation”
• “Responding to change over following a plan”

“Uncle” Bob Martin — one of the manifesto’s primary instigators and signatories, and author of Clean Code: A Handbook of Agile Software Craftsmanship — is reported to have characterized the manifesto as “a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we would want to work.”

Agile is supposed to be centered on people, not processes — on people collaborating closely to solve problems together in a culture of autonomy and mutual respect, a sustainable culture that values the health, growth, and satisfaction of every individual.

There is a faith embedded in the manifesto that this approach to software engineering is both necessary and superior to older models, such as Waterfall. Necessary because of the inherent complexity and indeterminacy of software engineering. Superior because it leverages the full collaborative might of everyone’s intelligence. But this is secondary to Agile’s most fundamental idea: We value people.

It’s a rare employer today who doesn’t pay lip service to that idea. “We value our people.” But many businesses instead prioritize controlling their commodity human resources. This now being unacceptable to say out loud — in software engineering circles as in much of modern America — many companies have dressed it up in Scrum’s clothing, claiming Agile ideology while reasserting Waterfall’s hierarchical micromanagement.

A sham of true Scrum

The misapplication of story points is the vector by which this sham of Scrum is delivered.

Scrum story points were originally intended to represent the relative complexity of stories. They are an approximation of an abstraction of relative expected effort, one made in advance of actually engaging the engineering. They are not an estimate of time, and they are not a legitimate basis for setting a deadline.

And yet this is precisely how many businesses use story points, tying them to timelines to pull shiny estimates out of their … well, let’s say hats. They then expect their human assets to adapt to the plan rather than the plan adapting to the people and the emerging lessons of implementation.

Then they push their people to crank out code, staying on schedule to deliver stories on time. If it turns out that a story estimated at 4 hours will actually take 16, well, time to pound some Red Bulls, then hustle and grind to balance the fantasy math that zeroes out the value of humanity.

Does that sound like a methodology that puts “individuals and interactions” over “processes and tools”? One that puts “customer collaboration” over “contract negotiation”? Is it “responding to change” rather than “following a plan”?

Obviously not. These are stories as mini-Waterfalls, each one treating the engineer as a cog in their employer’s machine. Stories as fully specified orders for chunks of code, with no understanding of the craft, creativity, and critical thinking required to solve such complex problems.

Waterfall by any other name is still obsolete

These mini-Waterfalls masquerading as Scrum reflect a top-down, linear approach that doesn’t give software engineers the autonomy, access, and authority to drive the implementation of what the product team requests.

There’s no room for actual conversations and genuine collaboration. No room for better ideas to emerge, or for anyone to change their mind.

Scrumfall relies, in other words, on the product team (made up of human beings) providing a complete and perfect specification before development begins. And it relies on the development team (also humans) planning out a complete and perfect implementation before a single line of code is written.

Not even Dr. Winston W. Royce, who wrote the seminal paper on what we now call Waterfall, believed this was possible. “I believe in this concept,” he wrote, “but the implementation described above is risky and invites failure.”

When the inevitable imperfections of specification and implementation later emerge, Royce wrote, “The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated.”

No wonder we needed a different approach.

Can Scrumfall ever work? Sure, in the most trivial of cases. Need a standard login page? We can specify, estimate, and execute on that in a timely manner without a lot of ongoing collaboration.

But for all the truly interesting, valuable, even revolutionary challenges in software engineering — the work that is complex and non-deterministic — we need human beings, collaborating in meaningful ways to craft better solutions together.

The butterfly effect breeds bugs

Collaboration introduces inevitable uncertainty into the process. (And the invading Waterfall taskmasters hidden in Scrum’s Trojan Horse absolutely hate uncertainty.) However, uncertainty is not a bug but a feature of Agile, which recognizes that better solutions can only emerge if the process embraces the unknown and protects the prerogative of people to change their minds.

The entire field of software engineering is so young, only slightly more than 50 years old, and changing so fast. Compare that to civil engineering’s millennia of advancements, refinements, and standardizations. So much of what we do is new, an exploration of the unknown.

Software is also orders of magnitude more complex. Though raw counts of lines of code are only a crude measure of complexity, consider that even a simple iPhone app typically has 40,000 lines of code, with more substantial software typically topping millions of lines of code. Google’s codebase is estimated to contain 2 billion lines of code, comparable to the human genome’s 3.3 billion.

When you consider all the interactions among these lines of code, you’re entering chaos theory territory, making software highly sensitive to butterfly effects: to wildly unintended consequences.

In short, we’re dealing with an inherently non-deterministic process. Separating, isolating, and confining people is never going to solve away the uncertainty. Enforcing adherence to arbitrary deadlines and rigid plans will not conjure deterministic outcomes.

Instead, these chains of mini Waterfalls create chaos, crappy code, and, ironically, cost overruns and missed deadlines. Long-term success is sacrificed to Scrumfall’s short-term imperative to meet arbitrary measurements. It’s also a sure path to burning out engineers, who will soon leave for other opportunities with more satisfaction and sustainability, less hustle and grind.

The fall of Scrumfall unlocks the true power of Agile

Agile remains a very good idea that places the value of people over that of processes. Its radical restructuring of the software development life cycle will continue to transform our field and produce superior solutions.

But to unlock the true power of Agile, we have to fully accept the non-deterministic complexity of the meaningful problems we solve. There is so much we don’t know at the start of a new project, and Agile says that’s perfectly OK. Smart people collaborating in a human-centric culture can find their own way to a good solution. You just have to trust them with the flexibility to solve things as they think best.

There will be wrong turns along the way. People will change their minds or see a better solution. Some stories will take longer to complete than you expect. (And some may take less.) You may have to alter your timeline, rethink your MVP, or reprioritize your features.

Agile holds that none of this is broken, none of it is a mistake, and none of it is wasted. It’s what smart people do when they’re solving complex problems. Rather than attempting to restrain all this uncertainty, Agile welcomes it as the very essence of creation.

There’s a balance to be kept, of course. Agile can’t be developer anarchy that never leads to a completed product. Properly implemented Scrum (or Kanban) keep the team aligned with what’s needed and what will reasonably lead to the desired outcome within finite time and budget.

But we have to stop degrading the power of Agile by obsessively measuring vanity units of work rather than the value of what’s delivered.

To realize the true potential of Agile, we have to recommit to its core principles and live them in everything we do by banishing the persistent taskmasters of Waterfall hiding inside Scrumfall.