How in the world do Agile software development methods like XP succeed where decades of Waterfall approaches have failed? How do they avoid classic late-project failures, and how do they deliver high ROI, release after release?

Agile methods were designed by industry veterans who, after suffering project failures time after time, had finally evolved best practices that tend to prevent that failure. We've summarized some of the key characteristics below.

Agile approaches strive first and foremost to break down the barriers between Us and Them -- between programmers and each other, between programmers and customers. On a healthy agile team, everyone shares the challenges, the feedback, and the problem solving. There is a basic human bond of mutual respect and trust that is essential to creating this kind of strong community. Without this community, we increasingly find, you don't have nearly enough foundation for the other practices. People don't care enough about each other's perspectives to share their problems and concerns, or their best, most creative ideas.

Learning to obtain, measure, and react well to feedback is the essence of agility, and it is an art. There is a saying that "perfect is the enemy of good enough," meaning that if you attempt to plan for every contingency before beginning real work, you are usually making two mistakes: wasting time, and anticipating the wrong things. By simply trying something (sometimes, anything!) in order to see how close or far off it is from a good solution, you can at least react to something real, instead of speculation. Once we have a real automated story test, or a real unit test, or some real code, or a real GUI, we have something to discuss, to like, to hate, to improve, to accept, to throw out, or to reject outright. Skillful agilists learn to prefer to gather a bit of real data from real feedback as soon as possible in any project, or indeed any situation.

Most projects involve way too many programmers, dollars, and function points. Studies such as the famous Standish CHAOS Report show conclusively that the larger and longer the project, and the more function points it addresses, the likelier it is to fail. The decline in failure rates from 31% in the 1994 CHAOS report to 25% in the 2000 report are largely credited to smaller, shorter projects and smaller teams. Another large case study "showed that defects rise non-linearly as project size grows." Yet another study showed that the percentage of useful code increased as project size decreased. In short, there is plenty of evidence to support dividing inherently large projects into smaller projects, and then dividing those into iterations.

Several studies (as quoted in Craig Larman's book) have failed to correlate project success with waterfall-style predictive planning, in which scope is nailed down up-front. No matter how hard we try to predict detailed project scope at the outset, it is not possible to do. Instead, plan and build a discrete system increment in each closed-end iteration, letting its scope slip but not the date. At the end of the iteration, compare metrics on predicted progress vs. actual progress, and use any discrepancy to plan the next iteration. Use the data from the first few iterations to arrive at a concrete estimate for final system scope. This is the recipe perfected by XP.

Two different studies have shown large rates of requirements change during projects. Another study found that iterative, incremental delivery "appears to contribute to both a lower defect rate and higher productivity."

The best practice is to iterate and deliver incrementally, treating each iteration as a closed-end "mini-project," including complete requirements, design, coding, integration, testing, and internal delivery. On the iteration deadline, deliver the (fully-tested, fully-integrated) system thus far to internal stakeholders. Solicit their feedback on that work, and fold that feedback into the plan for the next iteration.

Craig Larman's book quotes studies showing that "broadly, defect reduction comes from avoiding defects before they occur and from feedback" such as tests and evaluations. One study showed that as the time lag between coding and testing decreased, defect rates likewise decreased.

Test-Driven Development, together with refactoring, effectively eliminate the sad tradition of open-ended debugging, replacing it with "pre-bugging": finding and eliminating bugs at inception. The practice of continuous integration is also credited by Larman and others with lowering defect rates and increasing productivity.

Several studies correlate project failure with failure to tackle issues such as integration soon enough. The best practice is to address high-risk and high-business-value issues as early as possible. For example, program and test the core architecture, and integrate its major components, in early iterations. Some call this approach to architectural approach the "thin, vertical slice." The earlier you discover unworkable elements in a project plan, the likelier you are to have time to address them.

In XP (for example), you rank all features by business value. In each iteration, you plan to build the highest-priority remaining features. The finished system consists only of high-ROI features. And should the project be cancelled early, the system is robust and fully integrated, and may contain enough business-critical functionality to pay back your development investment so far.

Responding quickly to unanticipated requirements changes requires keeping the system simple and extensible. In turn, this requires understanding and applying the principles and patterns of object-oriented design. Don't let unhealthy dependencies develop between components. Continually refactor the design, so that it can accommodate arbitrary change at any point in its lifecycle.

-- Patrick Wilson-Welsh, Adaption Software     10/28/2004

[More of Adaption's Articles.]