Wednesday, August 25, 2004

Attacking Complexity

This article appears in slightly different form in the July/August 2004 issue of IEEE Micro © 2004 IEEE.

For as long as I have been in the computer industry, increasing complexity has been a constant problem. Every advance in computer speed and storage capacity has been met by increased requirements and expectations for the software that runs on those computers. Moore's Law continues to bail us out, but it does not encourage efficiency. 

Efficiency is not a problem if we simply trade inexpensive hardware for expensive development time. Bloated and inefficient code and cumbersome development methodologies cause few problems for products that provide basic functionality with minimal demands on computing resources. Inefficiency becomes a problem at the cutting edge, when we try to squeeze out that last bit of added functionality to gain commercial advantage over our competitors.

This time I look at three books. One book shows how to apply the techniques of agile software development to projects of large size and scope. Another explains the ins and outs of an operating system that might be a little simpler than the one you're currently using. The third provides principles and techniques for simplifying Java application programs.

Agile Software Development in the Large -- Diving into the Deep by Jutta Eckstein (Dorset, NY, 2004, 246pp, ISBN 0-932633-57-9,, $39.95)

Since reviewing Extreme Programming Explained (Micro Review, Nov-Dec 1999), I have often written about agile software development techniques. Extreme Programming is the most famous of these techniques. Some others are Scrum, Crystal Methodologies, Feature Driven Development, and Adaptive Software Development. The proponents of these and other techniques have banded together into the Agile Alliance ( The alliance has agreed upon a set of choices and a set of principles, known collectively as the Agile Manifesto.

Their basic choices are to value:
  • Individuals and interactions over processes and tools.
  • Working software over comprehensive documentation. 
  • Customer collaboration over contract negotiation. 
  • Responding to change over following a plan.
You can find the agile principles that flow from these choices expressed in twelve short paragraphs on the Agile Alliance website.

Many firms have tried to adopt or adapt agile techniques. Many of these attempts simply entail calling their existing processes agile or adopting a few superficial aspects of Extreme Programming. Other firms have seized on the second value choice as an excuse to eliminate documentation or to stop commenting code.

Such superficial or misguided attempts to use agile techniques have little practical benefit. Other attempts, however, have been successful, but almost always in projects of relatively small scale. The techniques described in most books on agile development do not apply easily to large projects. For example, using Extreme Programming on a large project might entail placing 100 people in a single room. This is likely to be impractical for most projects.

Jutta Eckstein is a member of the board of Agile Alliance and a software development consultant. In this book she shows  that there are practical ways to adapt agile development techniques to projects of large scale. As you might expect, these adaptations rely heavily on good project communication and on finding ways to assign tasks to small subteams.

Eckstein proceeds methodically through the aspects of agile processes that work differently in large projects. She brings the benefit of her real world experience to these questions. If you work on large projects, you'll find it well worth your time to read this short book.

How Linux Works -- What Every Superuser Should Know by Brian Ward (No Starch, San Francisco, CA, 2004, 366pp, ISBN 1-59327-035-6,, $37.95)

Unix in its many forms has been around for about thirty-five years. When it was half that age , I reviewed Maurice Bach's The Design of the Unix Operating System (Micro Review, Sept/Oct 1987). In the same column I reviewed Douglas Comer's book Operating Systems Design, The XINU Approach. XINU is a recursive acronym that stands for "Xinu is not Unix." It is also Unix spelled backwards. 

Xinu is one of many attempts to simplify and reinvent Unix, wholly or in part. Linux is the most successful of these attempts. It began in late 1991 as the work of Linus Torvald, and has grown into a poster child for open source development. Linux is the base operating system for many production web servers.

Brian Ward has been working with Linux since 1993. In this book he tries to give you an understanding of the inner workings of Linux. Rather than providing procedures for common Linux tasks, Ward provides conceptual information. He hopes that after you read his book, you can read and understand the documentation of any system program.

One interesting part of the book is Ward's explanation of the boot process. If you've ever watched a Unix system start up, you know that screen after screen of information goes scrolling by. In about ten pages, Ward explains what all of that information is meant to tell you.

Because Linux is frequently the base operating system for web servers, Linux administrators must understand networking and firewalls. Ward explains how these features work.

Operating systems must often support software developed by third parties. For example, in Windows, the .NET framework provides a way to encapsulate applications and avoid the DLL hell of earlier Windows systems. Shared libraries provide support for third party software in Linux. Ward shows how to manage -- and avoid the pitfalls of -- shared libraries.

One of the most powerful features of any Unix system is the programmable command shell. Many books explain how to use different shells. Ward gives a clear account of the basics of using the Bourne shell.

Ward addresses all of the main tasks that Linux administrators must face, but this book is not for current Linux administrators. They already know all of this and more. But if you have little experience with Unix and you want to set up a Linux system -- possibly on an old computer that can no longer run the latest Windows version -- this book is essential reading. You'll probably find the chapter on hardware especially useful.

Better, Faster, Lighter Java by Bruce A Tate & Justin Gehtland (O'Reilly, Sebastopol, CA, 2004, 262pp, ISBN 0-596-00676-4,, $34.95)

Tate and Gehtland are well known authors of books on software development. In this book they attack the current state of Java development, which they characterize as follows:
Development is getting so cumbersome and complex that it's threatening to collapse under its own weight. Typical applications use too many design patterns, too much XML, and too many Enterprise JavaBeans.
This is not,as you might have expected from the title, a book about improving your personal coding habits. Its main approach to producing better, faster, lighter Java applications may require you to change things that you do not control. Specifically, the authors ask you to consider abandoning heavyweight frameworks like WebLogic, Jboss, and WebSphere in favor of lightweight open source architectures like Hibernate and Spring. If such decisions are outside your control, you probably don't need to read this book. But even if you can't follow the book's advice, you may still find it enlightening.

The authors begin by painting a bleak picture of creeping bloat, though their principal target seems to be container managed Enterprise JavaBeans. They then enunciate basic principles that few are likely to disagree with:
  • Keep It Simple.
  • Do One Thing, and Do It Well.
  • Strive for Transparency.
  • Allow for Extension.
  • You Are What You Eat.
These are understandable enough, except for the last. In that one, the authors appear to mean that you shouldn't swallow every bit of conventional wisdom and vendor hype that comes your way. Instead they want you to be a heretic -- challenge a few things that "everybody knows."

From this point, the plan of the book is clear: discuss the selection of underlying technology in the light of each of the basic principles. Then introduce Hibernate and Spring. Finally, develop actual applications.

The authors have a lot to say, and they are fighting a good fight. It won't take you a long time to read this book, and you might learn something important.