Waltzing with Bears, Lean Software Development

This article appears in slightly different form in the July/August 2003 issue of IEEE Micro © 2003 IEEE.

Managing Software Projects

This time I look at two books on software project management. One addresses a gap in current standard methologies. The other provides a toolkit for a new kind of software development -- one that has grown in popularity in recent years.

Waltzing With Bears -- Managing Risk on Software Projects by Tom DeMarco & Timothy Lister (Dorset House, New York NY, 2003, 208pp, ISBN 0-932633-60-9, www.dorsethouse.com, $33.45)

Dorset House Publishing seems to have found a great formula, and I hope they stick to it. In recent years they have produced a steady stream of books with the following charactersitics:

• Short - generally well under 300 small format pages.
  
• Packed with practical advice on software project  management.
  
• Written by people who have been managing software development successfully for a long time.
  
• Iconoclastic - often showing common practices and beliefs to be short sighted, counterproductive, and generally absurd.

This formula works because of the "software crisis" that the industry has been bemoaning for at least 35 years. Software project management, as practiced by the mainstream of the industry, has not advanced significantly during that period. Relational databases, object-oriented design, and powerful development tools have made individual programmers vastly more productive, but project management has not gone far beyond where it was in the 1960s.   

As I noted in my review of DeMarco's novel The Deadline (Micro Review, May/Jun 2000), I was inspired by his Structured Analysis and System Specification when I first read it in 1979, and time has improved his understanding. DeMarco and Lister are long time partners in a consulting firm that specializes in software management issues. They have lectured and written extensively about corporate culture, management, productivity, estimation, and risk.

Managing risk receives more lip service and less real effort than any other issue in software project management. In fact, the authors have seen so much fake risk management that they close the book with a test you can use to determine whether your company is really managing risk, not just talking about it. The test is about a page long, checking for 9 key aspects of risk management. Unfortunately, most companies would score poorly on this test.

The book's title comes from a song in the Dr Seuss Cat in the Hat Songbook (Random House, 1967). Uncle Walter (or Uncle Terwilliger in some versions) finds that some risk is essential to his life: 

He goes wa-wa-wa-wa, wa-waltzing with bears, 

Raggy bears, shaggy bears, baggy bears too.

There's nothing on earth Uncle Walter won't do, 

So he can go waltzing, wa-wa-wa-waltzing, 

So he can go waltzing, waltzing with bears!

De Marco and Lister come to a similar conclusion. If a project has no risk, it's not worth doing. Companies that stick to safe projects often go out of business or have a lot of catching up to do. The authors cite the case of Merrill-Lynch in the area of online stock trading. Pioneers like E*Trade and early adopters like Fidelity and Schwab, grew dramatically in the 1990s. Merrill Lynch avoided the risks of developing an online trading facility, but struggled to stay even.

All worthy projects entail risk. The key is to manage the risk. The authors cite the case of the Denver International Airport, in which the City of Denver, apparently for political reasons, ignored and made no attempt to mitigate the risks that ultimately cost them a half billion dollars. This case is dramatic and public, but many cases like it occur every year in the more private settings of business projects.  Many companies climb the capability maturity model (CMM) ladder, and they see real benefits from doing so. Nonetheless, companies at high CMM levels often design processes that look good on paper but make no real effort to estimate the effects of risk or the expected value of completing each component of the project.  

DeMarco and Lister outline the risk part of their program this way:

1. Construct and maintain a census of project risks. Explicitly escalate to the next level any risks you don't plan to, or can't, manage.
   
2. Create an ongoing risk discovery process and develop a corporate culture that does not punish discussion of risks.
   
3. Estimate the probability that each risk will occur and the cost you will incur if it does.
   
4. Use simple math or a tool (the authors provide a free tool on their website) to create diagrams that show the uncertainty in project estimates.
   
5. Set goals optimistically, but estimate outcomes realistically, using the uncertainty diagrams.
   
6. Include mitigation actions (to be performed unconditionally) for each risk. Include these in the project's work breakdown structure.
   
7. Define a transition indicator for each risk, and define a contingency plan to go into effect if the indicator appears. Include the contingent actions in the project's work breakdown structure. 
   

There's more to it, of course, but you'll have to read the book.

The other side of the risk coin is value. The authors have no sympathy for companys that skimp on estimating value. They ridicule justifications like "we've got to have it to survive."  Their rule is to estimate value with the same precision you use to estimate cost. This should lead to uncertainty diagrams similar to those for risk. Alternatively, if the best value estimate is "we've got to have it," then an appropriate cost estimate should be "it's going to be expensive."

The authors invest a good deal of energy into fighting with general practice. Statements like "risk management is project management for adults" are meant to needle people into changing the status quo. The following statement attacks a few sacred cows:

People who don't have the requisite talent [to be a good manager] fall back on a host of mechanical approaches, such as Management by Objectives, Parkinsonian Scheduling, and "a culture of fear" to scare their subordinates into performing. . . .  These practices are incompatible with any risk management scheme.

One of the biggest parts of risk management is mitigation. This is work that you do unconditionally to lay the groundwork for contingency plans that may or may not become necessary. For the Denver International Airport, for example, this might have meant building tunnels tall enough so that humans could drive carts through them if the contractor for the automated baggage handling software did not finish on time.

Mitigation is an expense that translates into waste if the contingency plan is not necessary. The authors suggest a mitigation strategy that reduces potential waste: incremental implementation. Their plan for incremental implementation requires you to identify which parts of the project deliver the greatest value to customers and which can eliminate potential risks before mitigation actions are necessary. These are the parts you implement first. This approach provides many benefits, including pushing low valued "pet projects" to the end of the schedule, where they can often be eliminated harmlessly.

One of my favorite parts of the book -- though I doubt that it will influence many readers -- is the story of William Kingdon Clifford, who in 1876 scandalized London's Metaphysical Society with his paper, The Ethics of Belief. The authors kindly include the entire text of this paper in an appendix. Clifford held that it is unethical to hold and act upon beliefs that you have no real basis for. The authors take off from Clifford's work to define risk management as the business of believing only what you have a right to believe. 

There is a great deal of useful advice in this little book. The authors also provide practical techniques for implementing their ideas. If you have anything to do with software, you really need the information in this book.

Lean Software Development -- An Agile Toolkit by Mary & Tom Poppendieck (Addison-Wesley, 2003, Boston MA, 232pp, ISBN 0-321-15078-3, www.awprofessional.com, $39.99)

Agile software development is an umbrella term for methodologies like extreme programming (Micro Review Nov/Dec, 1999). The Manifesto for Agile Software Development (http://agilemanifesto.org/) states:

We are uncovering better ways of developing software by doing it and helping others do it. 

Through this work we have come to value: 

• Individuals and interactions over processes and tools 
  
• Working software over comprehensive documentation 
  
• Customer collaboration over contract negotiation 
  
• Responding to change over following a plan 
  

That is, while there is value in the items on the right, we value the items on the left more.

Lean manufacturing refers to the kinds of techniques pioneered by Toyota and others, as described in The Machine That Changed the World by Womack and Jones (Harper Collins, 1991). This approach regards every unnecessary element as waste. An unnecessary element can be acquiring inventory before you need to use it, deferring to management on a decision that the workers can make for themselves, or any of a wide array of similar inefficiencies. In this book the authors have pulled these lines together into what they call a set of thinking tools for bringing agile practices into your unique environment.

Like the authors of Waltzing With Bears, these authors are fighting the status quo. For example, they claim that the Project Management Institute (PMI) certification programs teach project managers to use wasteful formalistic substitutes for learning and innovation, the real keys to the success of development projects.

The authors state the following lean principles:

• Eliminate waste -- anything that does not add to the value the product brings to customers. But don't throw away all documentation.
  
• Amplify learning -- recognize that development is not manufacturing. Design, implement, get feedback, and repeat. But don't repeat indefinitely.
  
• Decide as late as possible -- keep gathering information to support your decision for as long as you can get away with (but no later -- don't procrastinate). This requires you to structure your project in a way that supports rapid change.
  
• Deliver as fast as possible -- use short cycles that give customers what they need when they need it and give you the feedback you need to refine your design. Implementation speed allows you to defer design decisions until you know enough to make them intelligently. But don't rush so much that you do sloppy work.
  
• Empower the team -- rely on the team to make decisions on the spot without referring them to a higher authority. But if you're the leader, continue to lead.
  
• Build integrity in -- provide highly usable software that gives customers exactly what they need now. Provide the infrastructure to support graceful evolution. But don't rely on a big up front design process.
  
• See the whole -- don't allow specialists to overbalance the design by focusing inordinately on their own specialties. But pay attention to details.

The book elaborates these principles into a set of 22 tools -- not a cookbook, but a schema for designing recipes. For example, the principle of building integrity in gives rise to tools called Perceived Integrity, Conceptual Integrity, Refactoring, and Testing. The authors explain the tools in the context of fascinating real world examples.

A complete commitment to agile development may not be ideal for your organization, but you might find the information in this book useful and relevant anyway. The authors bring a lot of practical experience to it, so it's worth reading, no matter what development process you follow.