Twitter

This article appears in slightly different form in the July/August 2009 issue of
IEEE Micro © 2009 IEEE.

The Twitter social messaging service is simple, but hard to understand, at least at first. I poked around its edges for a while, until I found the Twitter Book.

The Twitter Book

The Twitter Book by Tim O'Reilly & Sarah Milstein (O'Reilly, Sebastopol CA, 2009, 240pp, ISBN 978-0-596-80281-3, www.oreily.com, $19.99)

Tim O'Reilly is the founder and CEO of O'Reilly Media. I have never before reviewed a book that he wrote, but I have been reviewing books that he published for about 20 years. Starting with a few how-to books for Unix users in the 1980s, O'Reilly Media has become the leading publisher of books about hot software topics. In recent years, Tim O'Reilly has become a leader in the open source and web 2.0 communities.

Sarah Milstein started out as a freelance writer and editor. She was a regular contributor to The New York Times. Later she joined O'Reilly media, where she led the development of the Missing Manual series. She now travels around the country, lecturing and teaching about Twitter. With fellow Times contributor Marci Alboher, Sarah is cofounder of 20slides.com, which will soon launch a series of career-related online workshops.

The Twitter Book has been my gateway to Twitter. It contains solid basic information about all aspects of Twitter and provides context for a large number of pointers. By following those pointers, I learned everything I know about Twitter. If you invest a few hours reading this clear and comprehensive book, you will be well on your way to understanding how Twitter fits into your life.

Basics of Twitter

To begin using Twitter, go to twitter.com and sign up. Supply your full name, and pick a username that is short and to the point. My username is xrmxrm, which includes my initials and echoes my email addresses and Facebook ID. No username that includes a significant indication of my full name would be short enough to be an effective Twitter username.

Next, set up your profile. Twitter allows you a picture, a 160-character biography and a location. My biography is:

Technical communicator specializing in documentation for programmers -- APIs, web services, programmer guides. Columnist for IEEE Micro.

My location is Berkeley, CA.

Your name, your location, your biography, and your contributions are all that a person considering following your contributions has to go on.

Before using Twitter, you should understand updates, following, @references, retweeting, hashcodes, and searching.

Updates, also called tweets, are your contributions. You can enter a message of up to 140 characters, and everyone who has signed up to follow you will see that message, with your picture, among the tweets that come to them. Anyone can follow your updates, unless you protect updates, which few people do, because there is little point to doing so. You can send updates from a browser or a third party Twitter client, on your computer or your cell phone.

Following is an easily established asymmetric relationship between you and another twitter user. You simply go to that user's page (twitter.com/their_username) and click the Follow button. You can follow other users without their permission and without their following you. Twitter will, if you choose, notify you by email when other users choose to follow you. If you like, you can view their pages and decide whether to follow them. Many people automatically follow people who are following them, but there is no need to do so. Some people follow you as a form of spam. Obviously, you should not follow them. Twitter provides an easy way for users to report spam.

The most interesting uses of Twitter are conversational, so you need a way to refer to other people's tweets. Twitter interprets content of the form @username as a link, called an @reference, to that user's page. By a quirk of Twitter, if you begin a tweet with an @reference, most Twitter users who follow you but do not follow the user referred to in the @reference will not see the tweet. As a result, if you want to pass a tweet to your followers, a process called retweeting, you should place the @reference in the body of the tweet. The standard formats for retweets are to begin with RT @username or to end with via @username.

Twitter has a powerful search mechanism. In addition, it automatically brings to your attention the topics that are most popular at the moment. Search results can contain the tweets of users you are not following, so searching enables you to follow interesting trends and find interesting users to follow.

Often, a group of users interested in a topic choose a search term to include in tweets about that topic. Such search terms usually begin with hash mark (#). These are called hashcodes. As I write this, #iranelection and #michaeljackson are widely used hashcodes. Nobody assigns hashcodes. Users simply start using them. Different versions soon converge, and conflicts work themselves out.

Twitter Helpers

Twitter provides a certain amount of pushed summary information. In addition, a number of third parties follow Twitter, massage the flow of tweets, and make the information easily available. They can notify you about matters of interest to you, so you don't need to read millions of tweets each day. For example, TwitScoop shows hot topics as a tag cloud, in which terms appear in larger or smaller font sizes, depending on their recent frequency.

Another difficult problem for most new Twitter users is deciding which users to follow. Sites like MrTweet provide personalized lists of Twitter users for you to consider following.

If you want others to find your tweets worth reading and retweeting, you should link to interesting material. URLs can be long, and Twitter's limit is 140 characters for your entire tweet, so you need to find a good URL shortener. Such services (for example, TinyURL, Bit.ly, is.gd, and Twi.bz) replace a long URL with a unique, permanent short one that points to the same place. Different shorteners provide different advantages. For example, some give you some control over the resulting URL, while others make it easy for you to track clicks.

Pictures are a compelling component of many tweets. You can post a picture to a site like flickr.com and use a shortened URL to point to it. Others can leave comments at the site or retweet the link with their comments. The twitpic.com site integrates especially well with Twitter.

If you interact with Twitter solely through twitter.com, the primitive user interface soon becomes limiting. You must refresh the page to see new tweets. Simple tasks like replying and retweeting entail tedious manual steps that cry out for automation. Because Twitter provides an application programming interface (API), many third parties have created programs to put a different front end on Twitter. Most such programs run on computers, but others work on smart phones. I have tried Twhirl, SeesmicDesktop, TweetDeck, and DestroyTwitter on my laptop. Each provides improvements over the twitter.com interface, but I keep coming back to twitter.com because I feel more comfortable and less restricted there.

One drawback of the clients I've tried is that they provide little in the way of accessibility. I like to change window sizes and enlarge the text and graphics on my screen. This is easy to do in most browsers. The clients I tried, however, all use Adobe Air. If Air provides that sort of control, these clients haven't figured out how to make use of it. Or if they have, they haven't made it clear to me how I can use it. For that matter, the same criticism pertains to Adobe FrameMaker, which uses Air as the delivery environment for its online help.

What Twitter Is and Isn't

Twitter enables people to share facts, opinions, news, and commentary about everyday life or extraordinary events.

Recently Scott Williams posted an article at http://bigisthenewsmall.com/?p=2170 called Seven Things Twitter Is Not. He makes the point that Twitter is like a river. You can stand on the bank and watch the tweets go by. They are not piling up in your mailbox. Nobody expects you to answer them. If you do toss a tweet into the river, it flows away quickly. Someone may retweet it or reply to it, but the reply floats away too. It may include some context, but 140 characters soon become inadequate to contain an ongoing chat.

Another interesting metaphor is that of a great collective consciousness. As you watch the river, a subject may suddenly attract your attention. For example, on November 26, 2008, reports about the terrorist attacks in Mumbai spiked suddenly on Twitter, drawing worldwide attention before the news media could react.

One frequently asked question about Twitter is, "How can I use it to make money?" Twitter does not seem to have answered that question for itself, but many businesses have found ways to integrate Twitter into their ongoing relationships with actual and prospective customers. The Twitter Book devotes a substantial section to the do's and don'ts of using Twitter for business. No business should jump into Twitter until its management understands the principles that O'Reilly and Milstein lay out there.

You may not understand Twitter at first, but if you try it patiently for a while and get a sense of how others are using it, it will grow on you. Go to twitter.com and give it a try.

Software Architects

This article appears in slightly different form in the May/June 2009 issue of
IEEE Micro © 2009 IEEE.

This time I look at a collection that, according to its editor, is completely different from any other book you've read. That statement is true of any book, I suppose, but this book is unusual.

97 Things Every Software Architect Should Know: Collective Wisdom from the Experts edited by Richard Monson-Haefel (O'Reilly, Sebastopol CA, 2009, 218pp, www.oreilly.com, ISBN 978-0-596-52269-8, $34.99)

Richard Monson-Haefel is a co-author of the O'Reilly books Enterprise Javabeans and Java Message Service. The project grew out of a presentation called 10 Things Every Software Architect Should Know, which he was planning for a No Fluff, Just Stuff symposium. He solicited inputs on a mailing list, and nearly 50 people contributed the "things" that became this book. Each fills a pair of facing pages and includes a contributor photo and brief biography. The regularity of the structure ends there.

The free format allows contributors to formulate their wisdom in whatever ways work best for them. They address a variety of subjects in a variety of ways, but looking at the book as a whole, I see themes emerge. They have very little to say about project retrospectives or managing risk, so I hope they will all read two excellent books from Dorset House. Norman Kerth's Project Retrospectives (Micro Review, May/June 2001) and DeMarco & Lister's Waltzing With Bears (Micro Review Jul/Aug 2003) are essential reading for any software architect.

Communication

Communication receives a lot of attention in this book. Udi Dahan of Microsoft, for example, advises you to stand when you speak to a group. It makes people take you more seriously.

Other contributors stress the importance of knowing both the language of the business decision makers and the language of the software developers. Communication failures between the two groups cause many project problems, and these contributors assign to architects the responsibility for making that conversation work.

Norman Carnovale of Lockheed Martin Professional Services emphasizes the importance negotiating with the business decision makers to ensure that mid-project changes don't lead to unrealistic schedules and ultimately to failure. David Muirhead of the Blue River Systems Group recognizes that the business must drive a project, so he advocates developing the kinds of information that business leaders can use to make good decisions. Yi Zhou lays out the elements that make up a strong business case for a proposed expenditure.

Michael Nygard reminds us that we should remember the difference between a negotiation and an engineering discussion. When business decision makers ask "do you really need x," the answer should be "yes" if that's what the analysis led to. In an engineering discussion of the point, you might say "you could do without it if . . ." but business decision makers won't hear the "if" or anything that follows it.

Another communication theme is that architects must manage their own attitudes so that they can engage in open conversations with developers and customers. They must not let overconfidence or arrogance prevent them from hearing the good ideas of others or listening to the specific needs of customers.

Timothy High of Sakonnet Technologies gives advice that is easy to follow but often ignored. He advocates recording the rationale for decisions. He questions the value of most design documentation because it is hard to keep up to date. The rationale for decisions, on the other hand, doesn't require changes and is a valuable resource for the life of the project.

Requirements

Another theme of the book is requirements. Understanding, examining, and negotiating requirements and turning them into specifications is one of an architect's key jobs. Everyone understands that this process is a negotiation that must balance the needs of customers, funders, and developers. Eben Hewitt puts forth the beautiful metaphor of a consommé -- an extremely clarified broth achieved by repeatedly straining out the solids that cloud it. You must test the language of the requirements again and again until it is clear. This reminds me of the Mary Had a Little Lamb heuristic in Gause & Weinberg's Exploring Requirements (Micro Review, Sept/Oct 2001), another book that all software architects should read.

Many of the contributions boil down to pushing back against stated requirements in various ways. Einar Landre of StatoilHydro advises seeking the value in the requirement. He tells the story of the F-16 Falcon aircraft for which an initial requirement was speeds of Mach 2 to 2.5. By asking why, the contractor was able to change the requirement to an agile aircraft that can accelerate quickly to escape from combat. Similarly, Eben Hewitt tells you not to rush to solve the problem. First interrogate it to see if you can change it into a more tractable one.

Keith Braithwaite of Zuhlke exhorts us to quantify requirements. Fast, responsive, and extensible, for example, are terms that mean different things to different people. Developments based on such terms can lead to serious disputes at acceptance time.

Assumptions, as we all know, can be dangerous. Timothy High advises us to challenge assumptions, especially our own. Urban myths -- or even facts that used to be true -- can lead us away from excellent solutions to design problems.

Not all requirements can or should be satisfied. Good design involves tradeoffs. Mark Richards of Collaborative Consulting recounts the story of the Vasa, an early 17th century Swedish ship that satisfied all of its requirements and sank the first time it fired its guns. Bill de hÓra of NewBay Software says that requirements often come in threes (like fast, cheap, and accurate) and that you should be prepared to pick two and jettison the third. When a requirement seems absolute, be skeptical and ask why. Dave Quick of Thoughtful Arts puts it another way: Scope is the enemy of success. The greater the scope, the worse the chances of success.

Another way to test requirements, according to Stephen Jones, is to stretch key dimensions to see what breaks. This kind of stress testing can identify future problems long before they occur. It enables the necessary changes before the design becomes set in stone.

A number of contributors show concern for end users. Your customer may not be the end user, as Eben Hewitt points out, but your customer won't succeed if the system serves end users poorly. Keith Braithwaite, following Heidegger, uses the german term zuhanden to suggest that systems should be invisible to users. However, Steve Talbott in Devices of the Soul (Micro Review, July/Aug 2007) takes the opposite point of view: Because it is so important to remain conscious of the assumptions and unseen factors that affect us, it's a good idea not to forget about our tools. An invisible tool still embodies the ideals and assumptions of its creator.

Pitfalls

Many contributors offer advice about pitfalls and how to avoid them. Avoiding unnecessary complexity or generalization is a common theme. Neal Ford of ThoughtWorks defines accidental complexity as the complications that arise as byproducts of addressing the essential complexity of the problem. Often accidental complexity comes from adopting frameworks that are too general or fail to match the problem.

Kevlin Henney puts it another way: simplicity before generality, use before reuse. Eric Hawthorne says to choose frameworks that stick to their own domain and don't impose unnecessary requirements. Keith Braithwaite points out that the real world has multiple, inconsistent, overlapping frameworks. Sometimes it works to use each in its own domain. Or as Randy Stafford of Oracle puts it, there is no one-size-fits-all solution. Exercise contextual sense to understand what works in a given domain.

The editor of the book, Richard Monson-Haefel, contributes his own pitfall. Avoid trying to future-proof the system, because you can't predict the future. Trying to guess the technologies of the future can lead to analysis paralysis.

Design and Development Approaches

As you would expect in a book about architecture, design and development approaches receive a lot of attention. Most contributors implicitly or explicitly advocate iterative and incremental processes. Bill de hÓra summarizes this view by saying that great software is grown, not built. Similarly, David Bartlett advocates Martin Fowler's continuous integration (CI) approach, which includes automated builds and testing at frequent intervals.

Dan Chak of CourseAdvisor, however, focuses on designing the underlying database. He feels that the data structures and relationships that underlie applications do not evolve rapidly and require a complex, comprehensive technical design up front. The resulting solid and self-protective data fortress will save application data from the inevitable bugs in the application level.

George Malamidis of TrafficBroker adopts the language of business. He says that every architectural decision is an investment. You should consider the return on investment (ROI). This is true whether you use an agile process or the waterfall model.

George Holpe of Google says welcome to the real world. He implies that most software engineers are clinging to a bygone world of predictably ordered processes controlled by the program rather than by external events. In that world, systems are tightly, rather than loosely, coupled. He quickly tears down that straw man, using the word "scary" a couple of times to characterize the new environment. I get his point, and he's certainly right about the need for distributed, loosely coupled, event-driven systems, but I'd like to point out that we were building event-driven systems in the 1960s, so the idea isn't all that new.

Several other contributors also use uncertainty to their advantage. Kelvin Hanley suggests than when you come to a fork in the road, design so that choosing between the two branches has fewer consequences. Erik Doernenberg of ThoughtWorks suggests trying both branches and deferring the choice until it's obvious which is better.

Performance and Maintenance

Most of the life of a successful software project happens after its initial installation. Many of the contributors address issues about performance and maintenance.

Randy Stafford points out that application architecture determines performance. Rebecca Parsons of ThoughtWorks advocates beginning performance testing early in the development process. This can help you identify problems early. It also gives you an ongoing record that shows which changes helped or hurt performance. Craig Russell of Sun Microsystems takes a broader view and advocates looking at the performance of the developers implementing the design.

Tools and Techniques

I don't have room to go into all of the suggestions about tools and techniques. One that tickled me is Paul Homer's suggestion that you can get a good idea of how a system works by looking at its underlying data structures. In 1967 I attended Butler Lampson's course on operating systems at UC Berkeley. Lampson made exactly that point.

Architects

Many contributions address the necessary qualities or skills of software architects. Many see themselves as leaders of the development project and focus their comments on how to interact with developers. Many say to lead by example or to keep a hand in the development work. The objectives are to maintain the respect of the developers and to experience the design from the point of view of a developer. It's easier to recognize design mistakes if you're the one who has to implement them.

Many stress the importance of maintaining respectful and open relationships with developers. Evan Cofsky of Virgin Charter invokes a taxonomy in Neal Stephenson's Cryptonomicon. Architects are kings and must provide opportunities for elves, dwarves, and wizards.

Many also look at the social or ethical aspects of an architect's work. Barry Hawkins advises valuing stewardship over showmanship. As he points out, you are playing with other people's money. Michael Nygard points out that many decisions restrict or enable end user behavior. A few days of programming time may save end users a few seconds each time they use a feature. With many users and many repetitions, that could be a huge savings for them, but at a cost that you won't recoup. It certainly raises an ethical question.

Yi Zhou tells architects to take responsibility for their decisions. That means putting it in writing and communicating it to everybody it affects. You must also follow through to ensure that developers implement your decision, and you must review the decision periodically to see how it's working out. Taking responsibility also means not making decisions that you're not qualified to make. Delegate those decisions to experts.

I hope this review gives you an idea of the flavor of the book. If you design software, you're sure to find enough nuggets of information in it to justify its price many times over.

Software Testing, Freelancing

This article appears in slightly different form in the March/April 2009 issue of IEEE Micro © 2009 IEEE.

This time I look at two subjects that confuse a lot of people and give rise to a great deal of wishful thinking. Misconceptions about software testing can cause great damage to large and small enterprises. Misconceptions about freelancing can be catastrophic for individual entrepreneurs.

Software testing

Perfect Software and Other Illusions About Testing by Gerald Weinberg (Dorset House, New York NY, 2008, 198pp, www.dorsethouse.com, ISBN 978-0-932633-69-9, $29.95)

Testing is a sampling process designed to obtain information to use in decision making. Some testing activities entail banging on a keyboard, but many others do not. Suggesting a design review, for example, is a form of testing and a developer's reason for not wishing to submit to a review can be a useful test result. Many people see testing purely from the banging on keys point of view. That and other misconceptions lead to painful consequences for them. Gerald Weinberg, a self-described empathic person, has written this book to ease their pain.

Weinberg has been in this business a long time. He was an old timer in 1973, when I first read his enormously influential Psychology of Computer Programming, and he has been an effective and productive writer and consultant in the field of developing and testing software ever since. I have reviewed several of his books in my columns over the years. In addition, many books by other authors, especially those published by Dorset House, show the profound influence of Weinberg's style and approach.

The insights of the late Virginia Satir, a family therapist, inform this book, as they inform much of Weinberg's work about the social aspects of managing software development. Weinberg applies Satir's decomposition of communication into intake, meaning, significance, and response to structure his lessons about how to use testing. He also uses her insights about defensiveness and survival rules to understand the natural immunity to information that people often experience in threatening situations. Nonetheless, there is no new-age mumbo jumbo in this book. Weinberg translates all of his psychological insights into practical terms.

The book's title fallacy concerns the widely held belief among the general public that companies could produce bug-free software if only they tested everything before selling it. This belief falls apart upon cursory examination. Testing "everything" is not possible in a finite amount of time. Furthermore, while testing can identify problems, it cannot correct them. Testing can only provide information. Those in charge of a project can use that information to help them assess the risks associated with various options -- such as shipping the product or sending it back to the developers for more work. This decision is not always easy, because fixing one problem often leads to introducing or uncovering another, sometimes more serious problem.

Because testing can only provide information, you should test only if you need information to help you make a decision. Furthermore, written test reports rarely tell everything that the testers know, so if you want your money's worth from testing, you need to question and interpret the results. It also follows that testers do not produce decisions. Bad managers ask testers if the product is ready to ship. Good managers ask testers what they found, then use that and other information to help them make their own decisions.

Weinberg has seen the insides of far more projects than most of us have and has worked with some relatively dysfunctional organizations. His stories of scapegoating, intimidation, and other misuse of testers are hair raising. If you work under such conditions, you need to read this book right away.

Weinberg seems to have developed a special style just for this book. His chapters average about nine pages. Each ends with a short summary and a long list of common mistakes. The mistakes section reiterates the earlier chapter material in negative form. For example, his chapter on information intake explains why you should choose which information to look for among the large amount of raw data available to you. The first common mistake at the end of the chapter is "Not thinking about what information you're after. Don't just take what comes. You have choices."

As in many of his other books, Weinberg presents some lessons in the form of vignettes. For example, he builds his chapter about major testing fallacies around a few hours in the life of Rose, the new testing manager. The dialog is contrived, and the realistic touches (Rose spears a pickle with her fork and waits for Barry to finish his chili) are lame. Nonetheless, Weinberg uses this form effectively. He shows how the pressures on the different players combine with their partial understanding (or misunderstanding) of the situation to lead to scapegoating, turf battles, and finger pointing.

Weinberg presents a breakdown of bug fixing that accounts for a lot of inefficiency and delay in software development projects. Testing can uncover an anomaly, after which further testing merges into another phase called pinpointing, or isolating the conditions under which the anomaly occurs. Failure to agree on who is responsible for pinpointing leads to many conflicts between testers and developers.

Pinpointing leads to locating, which means determining the section of code in which the problem resides. Few organizations regard locating as part of testing, but a desperate developer might be tempted to return the bug to the tester for "more information."

Between locating and fixing comes another important step: determining significance. This is a management function belonging to neither testers nor developers. Many stakeholders can contribute to this process. Ultimately, someone must weigh the costs of both alternatives. Fixing the bug takes time and effort and might break something else. Leaving it as is can result in other costs, such as service calls or lost sales.

One of the biggest problems associated with testing is how it interacts with the schedule. In an ideal development process, testing begins before the project, because deciding whether on not to do a project depends upon information gleaned from some form of testing. Then testing continues through all phases of the project. Unfortunately, most project managers follow a different model.

Typical projects have schedules that include a block of time labeled testing at the end. This block does double duty as a buffer to accommodate slips in the development schedule. In a good project, testers participate in reviewing the usual project artifacts -- market requirement documents, functional specs, user interface specs, detailed design specs, and documentation plans. In a bad project they may be handed a functional spec and told to develop a test plan that tests everything. In either case, unless they work alongside the developers throughout the project, they do not usually find problems until long after those problems enter the product. What could have been fixed easily months earlier now becomes an obstacle to delivery. A project that was "on schedule" until the testers got hold of it is now in danger of slipping. At that point, it's easy to blame the problems on testing.

Another distinction that Weinberg makes is between testing and demonstrating. If you know the "test" result in advance and are only running it to give potential customers a good feeling, you are not testing. Testing always seeks to find new information. Somewhere near the level of Gianni Schicchi in Weinberg's Inferno are the sellers of expensive testing systems who use such fake tests to support unsupportable claims about their products. Vendors who claim that their products will do all the work without receiving extensive explanations about what the programs they are testing are supposed to achieve are simply charlatans in Weinberg's view. Weinberg is firm in his belief that machines will never replace brains when it comes to software testing. Testing is an interactive process of investigation in which each finding leads to new questions.

Like all of Weinberg's books, every page of this one has wonderful insights and advice. If you have anything to do with software development, you should get a copy and read it at least twice.

Freelancing

The Principles of Successful Freelancing by Miles Burke (Sitepoint, Collingwood Australia, 2008, 206pp, www.sitepoint.com, ISBN 978-0-9804552-4-3, $39.95)

SitePoint is a small publisher targeting web professionals. Its books deal with SQL, CSS, HTML, ASP.NET, and similar staples of the browser-based world. Miles Burke is a freelance web developer, and he has written a book for others who wish to enter that arena.

I confess that I was put off by the lax editing and the Australian jargon of this book, but if you can get past that, it has a lot of useful content for people who want to set out on their own. Other books for freelancers (for example, Secrets of Consulting by Gerald Weinberg) focus on advanced topics. This book lays out the basics. If you read this book carefully, you won't make any major omissions as you prepare for your freelance career. In fact, at every stage, Burke puts on the brakes and urges you to move slowly.

Many people jump into freelancing because they have just lost their jobs. Such an approach has little chance of success in most cases, because starting a business requires a financial cushion that few recently fired workers have. Also, those who have never worked for themselves may overlook expenses like self-employment taxes, insurance, sick days, vacations, and the like. Burke shows you how to make the necessary computations and set your rates accordingly. You may have to adjust a little if you don't live in Australia, but many of the points he makes apply in the USA as well.

Burke covers all the bases -- from not using stolen software to negotiating with your family for an efficient work area in your home. He also gives good advice about how to find work, how to relate to clients, and how to balance life and work when they all happen in the same place.

If you are considering a career as a freelancer, especially in the web design area, you should find this book well worth its price.

Hot, Flat, Crowded

This article appears in slightly different form in the January/February 2009 issue of IEEE Micro © 2009 IEEE.

The Italian political philosopher and pragmatist Niccolo Michiavelli once wrote about the difficulties of change. An English version of his much quoted aphorism is as follows:

There is nothing more difficult to plan, more doubtful of success, more dangerous to manage than the creation of a new system. The innovator has the enmity of all who profit by the preservation of the old system and only the lukewarm defense of those who would gain by the new system.

This time I look at a prescription for a new system for dealing with the world's energy needs.

Hot, Flat, and Crowded: Why We Need a Green Revolution and How It Can Renew America by Thomas L Friedman (Farrar, Straus and Giroux, New York NY, 2008, 448pp, ISBN 978-0-374-16685-4, $27.95)

Tom Friedman has been with the New York Times since 1981, when he signed on as an energy reporter. He is now a foreign affairs columnist. His first two books deal with the Middle East. His third, The World Is Flat (see Micro Review, May/June 2005), deals with globalization. In this, his fourth book, he returns to energy, bringing to bear his broad understanding and experience of global issues and the Middle East. Though the book is about global systems, he has written it as an American and addressed it to Americans. 

Friedman uses the phrase "hot, flat, and crowded" to refer to the three themes that are pushing the world into what he calls the Energy-Climate Era: 

• Global climate change.

• The emergence of new, globally connected middle classes in China, India, Russia, and elsewhere.

• Worldwide population growth.

He calls it the Energy-Climate Era, because energy and climate are the principal issues that the world will face for many years to come. These issues underlie and manifest themselves in the following big problems:

• Increasing population and the rise of middle classes have led to a sharp increase in demand for natural resources and energy.

• The huge amounts we spend on crude oil and natural gas have strengthened dictatorships in Russia, Venezuela, and the Middle East and funded some of our worst enemies.

• Vast emissions from energy production have overwhelmed the mechanisms that kept the level of carbon dioxide in our atmosphere relatively constant for a long time. Increasing levels are causing worldwide climate change.

• Access to energy has become increasingly important, but more than a billion people are not attached to a power grid. Many of these "energy poor" are using inefficient, dirty, ad hoc methods to obtain electricity.

• Flattening and crowding have led to destruction and disappearance of critical ecosystems. Species are disappearing at an alarming rate. 

Because many people contend that human activity is not a major factor in climate change, Friedman addresses that issue -- first with a lot of science and then with a Pascal bet. The mathematician/philosopher Blaise Pascal argued that even though you might not be able to prove it, you should live as if God exists. Such a lifestyle, he contends, is intrinsically good, regardless of whether God exists or not. Applying this reasoning to climate change, Friedman says that changing our economy to use energy efficiently and to generate it from clean sources will benefit us greatly, even if "global warming" is a hoax. Of course, getting to that changed economy is not trivial, despite the proliferation of articles with titles like Seven Easy Ways to Save the Earth in Fifteen Minutes a Day. Friedman impresses us with the magnitude of the task.

The current level of carbon dioxide in the atmosphere is 270 parts per million (ppm). If human activity stops tomorrow, the level will continue to rise because of mechanisms already in motion. Friedman believes that a workable goal is to manage the unavoidable and avoid the unmanageable. He translates this into slowing the growth of the carbon dioxide level, so it reaches a maximum of less than 550 ppm around 2050. Robert Socolow and Stephen Pacala of Princeton University have computed the scale of the necessary programs, assuming that those programs start now and are completely in place by 2050. Some of their programs overlap or are alternate approaches. Here are some of the key programs:

• For two billion automobiles, double the fuel efficiency to 60 miles per gallon, halve the number of miles per year to 5000, or change the fuel to ethanol or hydrogen.

• For 800 to 1600 large coal-fired plants, sequester all of the carbon emissions, increase efficiency from the current 40% to 60%, or fuel them instead with natural gas. 

• Replace all coal-fired plants by generating twice as much energy as currently from nuclear plants, 40 times as much from wind, or 700 times as much from solar.

• Cut by 25% the electricity used by homes, offices, and stores.

There are more, but these give an idea of the scale of the problem. 

Friedman explains the current power distribution system. It is a loosely connected network (called a grid) of power companies (called utilities), all regulated by the states in which they operate. The utilities all participate in the following bargain: they can operate as monopolies as long as they provide a cheap, reliable, ubiquitous source of electrons. Unfortunately, the grid provides little support for the kind of innovations needed to reduce power use. The grid is huge, unintelligent, decentralized, unregulated, and poorly integrated. Its main defects are that it has no two-way communication, and there is no way to vary the price by load or time of day. Furthermore, the utilities operate under a business model that rewards them for generating and selling more power but not for promoting or facilitating efficiency. Friedman sees a smart grid and a new business model as essential to a sustainable energy system. The question is how to get there. Friedman's answer is to encourage innovation.

Near the bottom of page 243, Friedman buries the following:

If you take only one thing away from this book, take this: We are not going to regulate our way out of the problems of the Energy-Climate Era. We can only innovate our way out, and the only way to do that is to mobilize [the marketplace].

This is the essence of Friedman's proposal for replacing our current unsustainable energy system with a sustainable one: shape the marketplace so that this is the inevitable outcome, then let the free market system work. For those who don't consider this an example of a free market, Friedman has an answer: our current free market isn't free either. He points out that what he calls the fraudulent hiding of the true costs of resource extraction and consumption shapes the current market. In addition he cites regulatory anomalies like the difference in import tarrifs between Brazilian ethanol (54 cents per gallon) and Saudi Arabian crude oil (3 cents per gallon).

Friedman says that the market is like a garden to be shaped into an ecosystem for energy innovation. The shaping takes the form of an intelligently designed system of policies, tax incentives/disincentives, and regulations. He doesn't want to pick winners, but the entrenched interests are like kudzu. Uprooting them might be the hardest part of the job.

Friedman believes that the simplest way to shape the market is to provide a price signal, and he quotes from his interviews with the chief executive officers of General Electric and DuPont to support that view. A price signal can take many forms, but in essence it is a fixed target that developers of alternative energy sources can shoot at in their long term planning. To understand why this is important, consider the price of crude oil, which is largely controlled by a cartel, the Organization of Petroleum Exporting Countries (OPEC). The price tends to rise and fall by substantial amounts. Its rises stimulate development of alternative energy sources. Then it falls, and the companies developing the alternatives go out of business.

One price signal would be a government guarantee of a minimum gasoline price. For example, when gasoline prices exceeded $4.00 per gallon in the summer of 2008, the government could have said that for each dollar the price dropped, the tax would increase 95 cents. This would have told alternative fuel developers what price to shoot at. Instead, as autumn gasoline prices dropped to below half of the summer levels, alternative fuel developers were forced to change or abandon their plans.

Another kind of price signal, applicable to all energy production, not just vehicle fuel, is a carbon cap-and-trade regime. In this approach a controlling authority sets a limit on the total emissions of carbon from all sources in an upcoming time period. It then distributes individual limits to all possible emitters in the form of credits. The credits add up to the total emissions limit. Emitters who do not have enough credits to cover their emissions must buy them from someone else or pay a severe penalty. At the end of the covered time period, the authority sets a new, presumably lower, emissions limit for the next period and issues new credits accordingly.

An alternative to a cap-and-trade regime is a carbon tax. Emitters pay an amount that depends on the amount of carbon they emit. This is simpler than a cap-and-trade system. Each has advantages and disadvantages, which Friedman explores. Although the cap-and-trade system gives "environmental certainty" by limiting total emissions, Friedman prefers the carbon tax because it is more transparent.

A price signal that some countries (and Texas) already use is a renewable energy mandate. The controlling authority specifies a percentage of total power generation that must come from sources (for example, windmills or solar cells) that do not emit carbon. The authority then increases the percentage over time. European countries have done this to varying degrees, which is why the US company First Solar finds more business in Germany, Spain, France, Greece, and Portugal than in the United States. 

Regardless of the form it takes, a price signal requires our leaders to make a long term commitment to a complex plan. This is why Friedman asks whether democracy can survive complexity. He envies the Chinese government's ability to dictate policies and have a reasonable expectation that all levels of the society will implement those policies. "If only we could be China for a day," he says, "but not for two days." In earlier crises, leaders took control to enable them to act decisively. Lincoln asserted power over the states, and Franklin Roosevelt expanded and strengthened the Federal government. 

US energy policy, like our system of government, is designed to make inaction easy and transformational action hard. Among the cooks stirring this broth are state regulators, the Environmental Protection Agency, the Department of Transportation, the Department of Energy, the Department of Agriculture, the Army Corps of Engineers, and the Federal Energy Regulatory Commission. Friedman characterizes the resulting policy as follows: maximize demand, minimize domestic sources, and make up the difference by borrowing huge sums of money to pay to those who hate us. To centralize energy policy, Friedman proposes a real Department of Energy. He points out that the current department with that name is concerned mainly with watching over our nuclear weapons.

Friedman provides many success stories of green projects. For example, US troops in Iraq used diesel-powered generators to provide electricity for a remote desert base. Transporting diesel fuel to the base exposed them to the danger of attacks that were hard to defend against. By insulating their tents, they were able to reduce their air conditioning power needs to a level they could provide from wind and solar sources. They even had power left over to give to a nearby village. This example shows the power of a price signal. Considering the costs of transporting and defending shipments of diesel fuel, the Army found it easy to justify the costs of becoming energy self-sufficient.

Another example is the conversion of 7.5% of New York City's taxicabs to hybrid vehicles. This example shows the effects of regulation. The city had to remove regulations that forced taxis to be large and heavy. Moving forward from this success, New York is trying to achieve similar increases in the efficiency of limousines.

A main thesis of the book is that transforming our energy system is a key to remaining a world leader in the years to come. If we do not lead, other countries will, and we will fall behind. If other countries, particularly China, try to follow the same path we followed, we have little hope of averting catastrophic climate change. If we lead the way to a green revolution, other countries will follow, and we will remain a world leader. In other words, as Friedman puts it, green is the new red, white, and blue.

Friedman examines what is happening in China. As the world's most populous country and the one with the fastest growing economy, China is the key to successfully transforming our energy system. The Chinese government seems to have recognized the problem and committed themselves to producing cleaner energy, but they have competing priorities. The bargain between the Chinese government and the Chinese people is "we rule, and you rapidly become more prosperous." This means that they can't afford to stop the bus. They have to swap out the dirty engine and replace it with a clean one without slowing down. 

This review focuses on a few key points, scratches the surface of others, and completely ignores parts of the book. Friedman is a thorough and careful reporter. As he showed in earlier books, he is good at understanding how issues interact. He tells compelling stories to illustrate his points. If you want to understand the workings of the worldwide energy system and how it interacts with population growth and climate change, you should study this book.

Effective Java, Thoughtworks Anthology, Javascript: The Good Parts

This article appears in slightly different form in the September/October 2008 issue of IEEE Micro © 2008 IEEE.

Software Development Patterns

This time I look at three books about programming. None of them has the word "patterns" in its title, and none of them claims to deal with patterns or best practices, but that is what they are all about.

Effective Java, 2d ed by Joshua Block (Addison-Wesley, Upper Saddle River NJ, 2008, 368pp, ISBN 978-0-321-35668-0, $49.99)

When I reviewed the first edition of this book (Micro Review, July/Aug 2002), I had never met Josh Bloch. Several years later, however, I was visiting the Google booth at JavaOne, when Josh came up to me and told me that my review had launched his book. He believed this because when my review appeared, his book had already been out for a year with modest sales, and his sales then increased substantially. I don't believe that my review had much to do with it. The book has a lot of substance, and it takes time for people to digest it and spread the word. Nonetheless, I recently took advantage of his gratitude to persude him to give me a copy of the new edition.

The first edition contains 57 short essays, called items. The new edition has 78 items. A few of the original items have disappeared, and Josh has added items to reflect the important changes to Java since 2001. He was a major player on the teams that developed generics, enum types, annotations, autoboxing, and the Java 5 for-each loop and concurrency library. 

In 2004, Josh moved from Sun to Google, where he continues to guide the evolution of the language but now also develops Java libraries that use the new features. Thus he knows Java both as a language designer and a language user. There is probably no other person who could have written this book with the same degree of insight and experience.

If you ever have a chance to hear Josh talk, don't miss it. He is an excellent teacher, because he explains complex points clearly. That talent shines through the pages of this book as well. The items that Josh discusses are quite complex, even abstruse. Yet any experienced Java programmer can understand them.

My review of the first edition highlights many good points that are still good in this one. What I concluded about the first edition remains true of this one. Every bit of this book is essential for Java designers. Reading this book before you start delivering products can easily repay its cost thousands of times.

The ThoughtWorks Anthology -- Essays on Software Technology and Innovation by members of the ThoughtWorks staff (Pragmatic Bookshelf, Raleigh NC, 2008, 234pp, ISBN 978-1-934356-14-2, $38.95)

ThoughtWorks is a software development firm that specializes in fighting software bloat. To this end they have developed techniques for programming, building, testing, and delivering applications efficiently. The essays in this book present several of these techniques.

ThoughtWorks's founder, Roy Singham lays out his vision in the first essay. He talks about "the last mile" of putting a system into production. The expression "the last mile" comes from the telecommunications industry. The high speed backbone of the global internet slows down when it reaches the "last mile" of copper wire that brings bits to and from most individual customers. People have been talking about this problem for decades, but customers have seen few improvements.

For Singham, the last mile starts when the software meets all of its functional requirements. It ends when it is in place, delivering value to the customer. Development teams can bring a project to a conclusion rapidly and efficiently because of their tools and methodologies. Customers then begin a long, slow, expensive process. They must ensure that the new software can handle the expected workload, work well with other systems, interact with existing data without corrupting it, and protect against all sorts of unauthorized use or attacks.

Currently, developers derive functional requirements from end users who, ideally, have continual contact with the developers throughout the project. Singham suggests adding the last mile tasks to the functional requirements and involving the last mile stakeholders in the project from its inception. This should lead developers to automate much of the last mile process, just as they routinely automate builds and testing. 

In telecommunications, the Holy Grail of the last mile problem is a means by which the internet can reach any customer at full speed (for example, fiber to the home). For software, Singham sees the Holy Grail as versionless software, a means by which developers can add features directly to production systems "on the fly," without fear of unintended consequences. Neither of these visions will come to pass any time soon.

Twelve more essays round out this book. They deal with programming, managing development, maintaining web applications and service oriented architectures, and building and testing systems. Having worked with large, convoluted Ant scripts, I especially appreciate Julian Simpson's essay on refactoring techniques for such files.

In the 1970s I worked with the late Rudolph Langer, who built a number of special purpose languages, usually on top of macro assembly languages. One of these, for example, facilitated designing pinball machines. Another made it easy to design optical forms to enable users to enter data visually. Nowadays we might call these domain specific languages (DSLs). Martin Fowler's essay explains how to use Ruby to create a DSL.

One of my favorite essays is Jeff Bay's "Object Calisthenics."  He proposes an exercise in which you must write a 1000 line Java program using the following extremely strict coding standards:

• Use only one level of indentation per method.
• Don't use the else keyword.
• Wrap all primitives and strings.
• Use only one dot per statement.
• Don't abbreviate.
• Keep all entities small.
• Limit classes to two instance variables.
• Use first class collections.
• Don't use getters, setters, or properties.

Bay explains the points that he intends this exercise to make. He wants you to encapsulate data, use polymorphism properly, avoid duplication, and appropriately model the problem that you want the code to solve. 

Like most books from the Pragmatic Bookshelf, this one is full of good practical advice. If you develop software, this one should be on your bookshelf.

JavaScript: The Good Parts -- Unearthing the Excellence in JavaScript by Douglas Crockford (O'Reilly, Sebastopol CA, 2008, 170pp, ISBN 978-0-596-51774-8, www.oreilly.com, $29.99)

Douglas Cockford is a JavaScript architect at Yahoo, where he introduced and maintains the JavaScript object notation (JSON) format. He is deeply steeped in the intricacies of the JavaScript language. The target audience for this book consists of highly experienced software designers and architects who are probably working on Ajax applications. If you do a little JavaScript programming to spruce up your website and are looking for a few tips and tricks, this is not the book for you.

Early browsers used Java applets, but this technology did not catch on. JavaScript went, in too short a time, from a rough draft of a language to the main language recognized by browsers. Subsequent revisions have added features, but the basic language design remains unpolished. Cockford refers to JavaScript as a "beautiful, elegant, highly expressive language that is buried under a steaming pile of good intentions and blunders."

The part that Cockford considers beautiful is what he calls Lisp in C's clothing. By this he means that at heart JavaScript is a functional language like Lisp or Scheme, but its syntax resembles that of C. JavaScript has objects but not classes, which confuses many casual JavaScript programmers. You can create objects from prototypes, but you can also freely add attributes and methods. You can also define objects as literals, simply by listing their attributes and methods.

Cockford sets out to show you how to create JavaScript programs that use the parts of the language that he likes and avoid the parts he considers bad. Unfortunately, you can't avoid all of the bad parts (for example, global variables, the lack of block scoping, the poor way it handles Unicode). 

Cockford's explanations of how to avoid the bad parts entail sophisticated programming. His examples require careful reading, and real mastery probably takes a lot of practice. Cockford characterizes the book as follows:

"This book is small, but it is dense. There is a lot of material packed into it. Don't be discouraged if it takes multiple readings to get it."

Excluding the appendixes and the index, the book is 100 pages long. You probably won't read it in one sitting, but it will repay your effort. It doesn't spend much time explaining the bad parts. If you want to use those, Cockford says, read any of the other JavaScript books.

If you are an experienced programmer familiar with many programming languages but forced to do serious development in JavaScript, this book may save you an enormous amount of trouble.