While agile has zealots, it is not a religion. Agile is a scientific method that converts economic chaos to profit.
Critics tell me “agile is just a religion” at least weekly. My friend Stephen Tryon likes to say that religions observe, construct a hypothesis, then defend the hypothesis at all costs. Religions do not change with new scientific discoveries, their rigor can be short-lived, and their rituals benefit us indirectly at best. The religion premise produces absurd results. Some companies have “converted” from Scrum to Lean Startup, a laughable idea since they affect different groups and used together they magnify profit. (How about “converting” to both?) Some companies claim to use SAFe (Scaled Agile Framework), but not its underlying team-methodology, Scrum. The result cannot adapt rapidly to market and production change, and thus fails to fulfill the agile goal. People reinvent hybrid agile-waterfall chimeras or allow teams to choose waterfall or agile, saying “we’re agnostic,” failing to realize the company’s time-to-market and profitability depend on their choice (agile may not be the right choice for an activity, but thoughtful analysis will reveal a right choice). A colleague argues people love Scrum because “engineers like to build stuff” and “engineers don’t like to be yelled at” [deha2013].
In fairness, agile advocates can behave religiously. The popular Agile Manifesto for Software Development, used by many to define “agile”, leaves much to interpretation. The Manifesto does not explicitly discuss measurement, experimentation, or limiting work-in-process, all essential characteristics of all agile methodologies (and science, by the way). The Manifesto is not testable. The Manifesto focuses on software, yet when people outside software ask us for an agile definition, we still offer the Manifesto (and their interest immediately declines). We advocate agile methodologies as a panacea for production and market problems. We don’t proactively discuss agile’s costs, tradeoffs and sweet spots; worse, most of us don’t know them. We defend the hypothesis at all cost. Religion.
Creativity Science
Though critics and friends alike spiritualize agile, agile methodologies are scientific frameworks that optimize creativity for chaotic economies. We can study these things. To be recognized as a scientific field, rather than a religion, agilists must describe their activities as science. The scientific method teaches us to observe, hypothesize, test the hypothesis and repeat. Guess what? In well-managed Scrum, Retrospective meeting participants observe and hypothesize the effects of process specifics (such as Definition of Done), Sprints test the hypothesis, and then we repeat! Every agile methodology has similar practices.
Agile methodologies adapt incrementally to improve chaotic economies. In analyzing this field, I am taking a “big tent” perspective on agility: if a methodology “feels agile,” I try to incorporate it. If conflicts arise, they help mark the boundaries of agile. Scrum and Lean/Kanban optimize creative production economies in general; XP (Extreme Programming) optimizes software economies in specific; Lean Startup optimizes customer marketing economies; GTD (Getting Things Done) and Pomodoro optimize personal production; Toyota Production System and XM (Extreme Manufacturing) optimize complex manufacturing; Hoshin Kanri optimizes executive strategy; the Rockefeller Habits optimize executive leadership.
Chaos theorists study emergent properties. Applying simple rules to entities in a chaotic system (such as individuals in a Scrum team) can create a complex adaptive system (such as higher productivity teams). We can’t predict the exact form of the resulting system, but we can seek improvements in key metrics, such as production rate. Imposing strict rules can create an ordered system, but ordered systems rarely produce unexpected outcomes. A system can lose creativity when it strictly controls individuals [Six Sigma is Draining Employees’ Creativity]. Since agile methodologies always address creative processes, we must not overly control our systems.
Emergent Profit
What simple rules generate emergent profit from chaotic economies? (Here, I use “profit” to represent value returned divided by cost; the units need not be currency. See Measure Economic Progress for more.)
We can take two ideas and turn them into rules to help generate agility: 1) Unless we can measure “profit”, we can’t identify whether the emergent outcomes are better or worse (many businesses suffer from unmeasured ad hoc decisions). 2) Given that emergent outcomes cannot be inferred from the simple rules we impose, we must experiment. Their corresponding patterns look like this:
- Economic actors sometimes focus so intently on activity, they can’t tell whether they are improving …
… therefore measure economic progress, as often as feasible. - Creative actors produce new things with new tools in chaotic economies. Since it was never produced this way before, no one can predict long-term outcomes …
… therefore proactively experiment to improve.
In chaotic systems, outcome predictions worsen exponentially over time. We can predict how a simple rule change will affect the outcome in the short-term, but the long-term it is impossible. If we can keep the rule changes simple and the time short, we can perhaps improve the short-term outcome slightly. If we keep making incremental changes, rapidly observe and adapt, we improve the long-term economic outcome, otherwise we won’t. This observation forms the third pattern in our agile pattern language.
- Creative activity exploits current resources to serve current markets, but chaos will soon erode any advantage …
… therefore limit work-in-process to rapidly test and adapt the work.
These three agile base patterns, present in every agile methodology, transform chaotic economies into complex adaptive profit systems.
Religion or Science?
How can we tell whether agile is a religion or science? If we believe that our economy is chaotic, these three agile base patterns produce a complex adaptive system. Many economists believe economies exhibit chaos [good1990]. The likelihood of this system improving over time depends on two questions:
- Do the experiments we construct in pattern #2 explore the space productively?
- Does the work-in-process limit in pattern #3 generate a complex adaptive system?
It turns out these two questions also relate very strongly to my PhD dissertation, which explored a complex adaptive system called simulated annealing. I found that if simulated annealing experiments are fractal (they explore a wide-ranging space in early “sprints”, then narrow the range with time) and if random errors have the same properties (they get smaller over time)—the system will converge to a good result [gree1995].
This gives us guidance about agile Retrospectives. Early retrospectives for a team should explore ideas much more freely—pair programming or not, co-working or not, multi-lingual programming or not, longer or shorter sprints, etc.—and then reduce the “radicalness” of experiments over time (analogous to fractal exploration). Early sprints can survive a looser Definition of Done (big difference between sprint completion and “truly shippable”), but later sprints should ship to customers (analogous to fractal errors). [I must admit, I didn’t realize this surprising parallel until I worked on this article.]
If we don’t assume our economy is chaotic, we have defined agile sufficiently that we can construct experiments to compare it to our previous management approach. Since agile uses economic metrics to guide its progress, but “waterfall methods” typically do not, we can often conclude from looking at an organization, “Anything that measures and adapts is better than this!” Otherwise, we can make an experimental comparison.
If a non-agile approach ever wins, I’ll be surprised. Systematic Corporation, in Aarhus, Denmark, made numerous comparisons in its highly controlled CMMI Level 5 software development environment, and concluded that Scrum-based management reduced defect repair cost by 40% [jako2009] and increased delivery rate by 100% [jako2015], compared to waterfall under CMMI 5. Similar results are likely from the other agile methodologies.
I hope I have convinced you agile is science, not religion. Many enthusiastic colleagues do feel spiritually attuned to agile. Agile’s transparency, experimental approach and collective responsibility seem to produce greater trust and happiness. But at the core, agile methodologies are testable management approaches to maximize the economic benefits of creativity. They are science.
References
[deha2013] Brian de Haaff, “We are in a Holy Scrum War: Scrum zealots are like vegans and barefoot runners,” https://medium.com/i-m-h-o/we-are-in-a-holy-scrum-war-d3511064dd71 (September 13, 2013).
[good1990] Richard M. Goodwin, Chaotic Economic Dynamics, Oxford University Press (1990).
[gree1995] Daniel R Greening, Simulated Annealing with Errors, PhD Dissertation (1995), UCLA Computer Science Department.
[jako2009] Carsten Ruseng Jakobsen and Jeff Sutherland, “Scrum and CMMI – Going from Good to Great: Are you ready-ready to be done-done?,” Agile Conference 2009, IEEE.
[jako2015] Carsten Ruseng Jakobsen, personal communication (May 4, 2015).
Reviewers
Many thanks to Hala Saleh, Charles Eliot, Erik Gibson, Stephen Tryon, Laureatte Loy and Bianca Sias for early draft reviews.