Top-Down Agile Beats Bottom-Up

Leaders who publicly demonstrate agile methodologies and promote them top-down drive their organizations to sustain agile practices and succeed. But bottom-up agile transformations lack resiliency and generate cultural strife.

Taiichi Ohno, agile coach for Toyota and its executives

Agile methodologies are now widely recommended for managing software development, but most large companies require transformation from entrenched “waterfall development,” an intuitively appealing strategy that has created massive project disasters (see Why Software Fails). Traditionally, most large agile transformations have been pursued bottom-up. One approach starts with a single team, proves agile works, and then expands further and higher in the organization. Hopefully that agile team’s success inspires others to become agile. Another approach religiously converts all engineering teams to adopt a specific agile methodology, but leaves management teams and hierarchies, dependencies, promotion policies, job titles, roles, recruiting and budgeting in their previous form. The developers adopt agile, but the managers don’t.

Bottom-up agile coaching is revolutionary coaching. If bottom-up coaches give managers advice at all, they counsel to “let teams self-organize” and avoid command-and-control management, not that managers should organize their peers as agile teams. Bottom-up agile coaches sometimes idolize individual contributors, and discount manager value with amusing but threatening statements like “Kill all the managers.” Yet managers make important strategic decisions, wield diplomacy and budgetary authority to remove significant impediments, and institute coordinated organizational change.

Agile methodologies are difficult to sustain. Agility requires discipline, which benefits from coaching, and coaches cost money, whether they are internal enthusiasts or external contractors. Agile introduces more overhead in iterated build, test and deployment. Agile requires unbiased observation and rapid experimentation, which can unsettle traditional managers and executives. Conservative managers can irrationally fear agile’s occasional low-cost failures, forgetting that waterfall’s failures were massive and deadly. Agile’s transparency can reveal contradictions between manager behavior and corporate goals, creating embarrassment at best and threatening careers at worst. If we fail to publicize evidence that proves agile has helped the organization find markets, improve development and adapt to change, then agile retreat can be the first option executives consider in a crisis—agile process is a pain in the ass, why should we keep it?

Bottom-up agile initially seems cheap, but can cost a fortune. While the expertise required to teach engineering teams how to adopt agile practices is routine, and thus less expensive per coach, companies may ultimately employ an “army of coaches” to complete the conversion, raising the price of bottom-up agility far higher. But bottom-up agile has high recidivism, resulting in waves of “coach army” sweeps to recover lost agility. Examples of companies who have experienced four or more bottom-up agile conversion waves are easy to find.

Bottom-up agile will eventually collide with top-down waterfall assumptions and career demands. Jeff Sutherland reports recently that 80% of “Scrum” teams he visited in Silicon Valley could not ship a product at the end of every sprint, despite that shippable product increments are a bedrock agile practice. I blame executives: They should demand that their teams ship products to users every Sprint. But for many executives, avoiding rapid delivery and feedback loops can preserve the unexamined big-budget funding for a flawed project … and the executive’s career.

Non-software companies suffer the same problems with bottom-up agile. In 1984, long before GM’s bankruptcy, Toyota trained General Motors employees to use the Toyota Production System (TPS) in the NUMMI Fremont manufacturing plant. Employing exactly the same union workers, this plant went from being GM’s worst performing, with drug-dealing and sabotage on the plant floor, to its best, with GM’s highest quality cars rolling rapidly off the line. You might think executives would ecstatically replicate that TPS management approach to other plants, but no. If you have a free hour, listen to this hilarious, compelling and sad episode of This American Life, “NUMMI” [lang2010]. Bottom-up failures in agile transformation will no longer surprise you. If the leaders aren’t themselves agile, you have a problem.

Top-down Agile Conversion

Who can fix this fastest? The highest-ranking officer willing to adopt agile for themselves and their direct reports. Managers at every level have strategic activities that gain from collaboration, belong in a backlog and see greater success when managed with agile methodologies.

Scaled agile is a form of multi-agent optimization, and optimization analogies show top-down wins. In a former life, I examined global optimization in distributed computing, which has many analogies to organizations [gree1995], concluding that top-down optimization reaches a better result faster. Others have found similar results [cres2008].

What does top-down agile conversion look like? Top-down conversion first identifies the organization’s larger strategic goals, decomposes them into a tentative backlog, establishes a collaborative strategy team (composed of executives) to achieve those goals, constructs adaptive experiments to test the hypothesized outcomes, responds to invalidated hypotheses by changing the goals or the process, and solves systemic problems. It looks like Scrum for strategic work, and that’s exactly what it is.

Executives who collaborate under agile become allergic to waterfall dysfunctions in their staff. They stop tolerating activities misaligned with corporate goals and start measuring progress on the company mission. They insist that teams and product managers validate hypotheses before making wild claims. They limit work in process to avoid investing too much valuable time under uncertainty. They demand that their employees take collective responsibility for fixing problems, rather than throwing peers under a bus. They welcome systemic problem solving, including employees examinating how executives can better help the organization succeed.

In short, in top-down agile, executives become the exemplars of self-organization. If they need an agile department to support their efforts (and most will), they will demand agility from the department.

Bill Gross, Minimum Viable Product Approaches You Can Learn From

Our most recognized agile successes—Idealab, Salesforce, Spotify, Toyota, etc.—were led by CEOs or CTOs who understood and embraced agile concepts not only for their teams, but for themselves. Idealab, a firm that pioneered Lean Startup concepts, started over 160 companies with 100 successes (along with 60 failures) [gros2014]. Its entrepreneur leader, Bill Gross, is estimated to be worth more than $300 million. Salesforce dominates the CRM industry, with a market cap of $42B, and was an early adopter of Scrum under the direction of CEO Marc Benihoff. Spotify rapidly grew to a private valuation of $1B, giving its CEO Daniel Ek, a natural agilist, a paper valuation of $300M. Agile coach Henrik Kniberg advises the CTO on down. Toyota became the most profitable car company through the leadership of its CEO Eiji Toyoda, coached by Toyota Production System inventor Taiichi Ohno.

In these top-down cases, the corporate leaders didn’t just promote agile, they themselves were agile. Through their fascination with agile practices, their adoption of agile for themselves, and their insistence that employees follow suit, these leaders’ companies became agile and stayed that way.

References

[char2005] Robert Charette, “Why Software Fails,” IEEE Spectrum (September 2005), http://spectrum.ieee.org/computing/software/why-software-fails.

[cres2008] Valentino Crespi, Aram Galstyan, Kristina Lerman, “Top–Down vs Bottom–up Methodologies in Multi–Agent System Design,” Autonomous Robots 24:303–313, http://www.isi.edu/~lerman/papers/methodRevisedFinal.pdf.

[gree1995] Daniel R Greening, Simulated Annealing with Errors, PhD Dissertation, UCLA Computer Science (1995) https://www.researchgate.net/publication/270648114_Simulated_Annealing_with_Errors.

[gros2014] Bill Gross, “MVPs You Can Learn From,” Lean Startup Conference 2014, https://www.youtube.com/watch?v=sRkdV5Vc_4I.

[lang2010] Frank Langfitt, “403: NUMMI,” This American Life, http://www.thisamericanlife.org/radio-archives/episode/403/nummi.