Life and Work

This article appears in slightly different form in the September/October 2009 issue of
IEEE Micro © 2009 IEEE.

The Declaration of Independence of the United States of America declares that every person has the right to life, liberty, and the pursuit of happiness. These are precious rights, but exercising them takes work. Pursuing happiness entails overcoming many obstacles. The easiest and hardest of these are often in our own heads.

Recently I attended a workshop called Strategic Planning for Your Life, led by Judy Glick-Smith, a former president and Fellow of the Society for Technical Communication (STC). That workshop is not the subject of this review. You can find information about it at www.mentorfactorinc.com. Glick-Smith's method asks you to begin by discovering what makes you happy and casting aside the internal obstacles to pursuing that happiness. She then lays out steps whereby you can arrive at a set of life goals and an ongoing process for pursuing and updating them.

By whatever path you arrive at your life goals, they are likely to include some sort of productive work. If that work is a high-tech job, I can recommend a book to help you.

Land the Tech Job You Love by Andy Lester (Pragmatic Bookshelf, Raleigh NC, 2009, 272pp, ISBN 978-1-93435-626-5, www.pragprog.com, $23.95)

Andy Lester has been writing software since the mid-1980s. He is active in the Perl community, where he maintains more than 20 modules on the Comprehensive Perl Archive Network (CPAN). In the course of his professional life he has seen countless résumés, interviewed many candidates, and done his share of job hunting as well. His frustration at what he has seen led him to write this book.

Many people give job-hunting advice. Most are dogmatic about their recommendations, even though one effective job hunter may give you rules that contradict those of another equally effective job hunter. Lester is no exception. He is sure of his rules, though he recognizes that they may not apply to your situation. Hardly anybody, however, is likely to disagree with the basic principles he starts with:

• Be honest with yourself.

• Be honest with others.

• Think like the boss.

• Be a problem solver.

• Sell yourself.

• Tell stories.

• Be positive.

Lester begins from the premise that life is too short for a job you don't love. Being honest with yourself is a key to ensuring that you don't wind up in a job that you don't love. Lester urges you to think about what you want in a job before you start looking at job ads or writing a résumé. Following one of his own principles, he tells a story of how he had taken a seemingly perfect job and found it to be unbearable. He hadn't aksed himself what he really wanted in the job. Had he done so and given himself honest answers, he would not have taken the job.

Once you decide what you want in a job, it's time to think about a résumé. Lester identifies what he considers to be the basic sections of a résumé and tells you how to gather and organize the contents of each. There are no surprises here, but his systematic coverage can help you ensure that you haven't overlooked anything.

Lester also identifies material that you should not include. Some of this is uncontroversial -- for example, don't include a photo or any information that you can't legally be asked about in an interview. Cautious hiring managers will discard such résumés without looking further. Some of his other rules, however, may cause raised eyebrows. Some experts suggest beginning a résumé with an objective, but Lester considers objectives to be useless fluff, often filled with meaningless phrases like "challenging position" and "contribute my skills." And if you ever apply for a job for which Andy Lester is the hiring manager, don't waste space on your résumé with the phrase "references available on request."

Once you have the basic building blocks, Lester wants you to put them together in a variety of ways. Of course you need a generic résumé on paper and in electronic form, but that's just the beginning. Ideally, for example, you should create a separate résumé for each position, tailoring and arranging the parts to highlight the ways in which you meet that job's requirements. Furthermore, you need versions in various formats. Some hiring managers ask for résumés in Word format, while others want text, and you should also have a generic HTML version on a publicly accessible website. If someone asks for Word format and you send them PDF, they already know that you can't or won't follow directions. Lester has much more to say about résumés, and it's worth reading, even if you don't agree with all of it.

After all of this groundwork, you have to find a job to apply for. Lester wisely steers you away from "job boards," websites that treat candidates and jobs as fungible commodities to be matched with one another. He also distrusts recruiters, because their clients are the hiring firms. You might think that recruiters find positions for candidates, but their actual job is to find candidates for positions. Instead of using job boards and recruiters, ask the people you know -- including your Aunt Edna, but excluding your current co-workers -- to help. Help can take many forms, but Lester advises keeping it lightweight. Ask for pointers and leads that you can follow up on. Don't ask anyone to do your job hunting for you.

Another piece of good advice that Lester offers is to use traditional sources like newspapers or the local Chamber of Commerce. Newspapers can give you a high level view of what's available in the area they serve. The staff of the Chamber of commerce can probably give you pointers to valuable contacts inside firms that need your services.

Once you find some interesting job possibilities, there is more to do before you send anybody your résumé. Lester tells you how to use publicly available sources like a company's website or its Securities and Exchange Commission (SEC) filings to answer the key questions: how does this company make money, how could I help it make money, and would I enjoy doing so? Remember, life is too short for a job you don't love.

Lester suggests using online resources to help you find information about a company. You might discreetly approach a current or former employee who posts to a technical mailing list that you belong to. Or you might look on the Facebook or LinkedIn social networking sites for people whose profiles mention the company.

Once you've found out everything you can about the company, it's time to apply for the job. Many people will tell you to send your résumé with a cover letter, but Lester points out an aspect of cover letters that you may not have thought of. If you can't come up with two paragraphs that explain why you are a good match for the specific job and company, you probably shouldn't apply.

Jack Molisani, a recruiter who often makes presentations about job hunting, tells the story of a résumé he sent to a client who wanted someone with patent experience. The client called him back and asked why Molisani was wasting his time with a candidate who didn't have patent experience. The résumé mentioned the candidate's patent experience, but not near the top, and the client hadn't read far enough to find it. This illustrates several of Lester's points. One point is that if interviewers ask you questions that your résumé clearly answers, don't say "read the bleeping résumé." Molisani knew better than to call his client's attention to the material the client had overlooked.

A second point that Molisani's story illustrates is that you should look at your résumé through the eyes of the hiring manager. Think about what is important to the hiring manager and make that part of your experience prominent on the résumé. I agree with that point, but I disagree with the method Lester suggests for making parts of your résumé prominent. Lester suggests highlighting words of the résumé that are important to the specific job by making them bold. This is a dangerous technique. A skillful communicator might get away with it, but most job candidates are likely to wind up with résumés that look like a ransom notes. Other techniques that Lester also advocates are more effective. Rearrange the sections of the résumé to highlight the most important information, and mention key points in the cover letter. By all means, tailor the résumé and cover letter to the job, but go easy on the bold.

While Lester rightly encourages you to follow directions and give hiring managers what they ask for, he makes an exception for salary history. Never provide this information. It's nobody's business, and there is no benefit to providing it. Lester suggests finessing this request by politely but firmly noting that the information is confidential and that it's not a subject that you discuss with anyone.

When the company invites you to a job interview, you're almost there. Unfortunately, many candidates perform badly in interviews. The hiring manager is likely to prepare carefully for the interview, and you should do the same. This includes preparing logistically. Arrive on time with whatever materials you need to bring with you. Clear your schedule, so you can stay as long as necessary.

There are many ways to look at interviews. They are conversations in which you can lead as well as follow. They are sales pitches by which you show that you are the right person for the job and make the hiring manager eager for your services. They are a first day on the job in which you seek out the areas where the company needs you most. Unfortunately, they can also be an obstacle course in which the hiring manager seeks to trip you up with questions designed to eliminate candidates and simplify the hiring choice. Lester prepares you for all of these aspects of interviews.

One piece of advice that Lester provides applies to many of life's situations -- not just to interview questions. Listen, and seek to understand not just the question, but the reason for the question. Remember that an interview is a chance for each party to decide whether they would like to work with the other. Sometimes understanding the reasons for questions can help you with your half of that decision. And being sure you understand a question before you answer it can keep you from saying something that makes you look uninformed or ill-prepared.

Lester also provides a lot of advice that you've probably heard before, but it can't hurt to keep it in mind. Arrive a little early. Treat everbody like the CEO -- especially the receptionist. Stand to shake hands. Make eye contact (but don't stare). Be positive. Ask for the job.

I do not completely agree with Lester about how to handle a portfolio. I learned about how to use portfolios from Lance Gelein, another former president of STC. To Gelein a portfolio is a set of impressive props to use while telling stories. If the conversation comes around to a specific kind of job, you open your portfolio to a sample of that sort of work and tell the story of how you approached it, what obstacles you faced, and how you overcame them. The portfolio piece is not the entire job, but a snippet that you can point at while telling the story. Your portfolio does not stand alone without you. You never leave it with the hiring manager or let anyone look at it without your accompanying narration.

To Lester, a portfolio is a collection of representative pieces of work that you are proud of. Each stands alone, and you happily leave a copy behind for the hiring manager to peruse after you've left. There is, of course, a place for this sort of work sample, but it doesn't fully exploit the story-telling potential of a portfolio.

Lester devotes an entire chapter to tough interview questions. Again, the key is to understand the reason behind the question, so you can directly, without equivocation, satisfy the interviewer's concern. Lester identifies a number of red flags that interviewers want to uncover. Some of these are two-edged swords. For example, do you refuse to ask for help? Are you unable to work without constant hand holding? Interviewers ask questions designed to detect where you stand with respect to these extremes. If you recognize what they are getting at, you can give a useful answer. Don't try to duck questions like "What is your greatest weakness?" or "Tell me about a project that didn't go well." Lester shows you how to approach such questions.

Lester's advice doesn't stop with the interview. He covers follow-up, dealing with an offer or with rejection, and even how to stay hirable after you start the new job. He has written a short but comprehensive book -- easy to read and full of good advice. Like every other book about job hunting, this one is not perfect, but if you absorb its advice and adapt it to your situation, it should pay for itself long before your first hour on your new tech job is complete.

Twitter

This article appears in slightly different form in the July/August 2009 issue of
IEEE Micro © 2009 IEEE.

The Twitter social messaging service is simple, but hard to understand, at least at first. I poked around its edges for a while, until I found the Twitter Book.

The Twitter Book

The Twitter Book by Tim O'Reilly & Sarah Milstein (O'Reilly, Sebastopol CA, 2009, 240pp, ISBN 978-0-596-80281-3, www.oreily.com, $19.99)

Tim O'Reilly is the founder and CEO of O'Reilly Media. I have never before reviewed a book that he wrote, but I have been reviewing books that he published for about 20 years. Starting with a few how-to books for Unix users in the 1980s, O'Reilly Media has become the leading publisher of books about hot software topics. In recent years, Tim O'Reilly has become a leader in the open source and web 2.0 communities.

Sarah Milstein started out as a freelance writer and editor. She was a regular contributor to The New York Times. Later she joined O'Reilly media, where she led the development of the Missing Manual series. She now travels around the country, lecturing and teaching about Twitter. With fellow Times contributor Marci Alboher, Sarah is cofounder of 20slides.com, which will soon launch a series of career-related online workshops.

The Twitter Book has been my gateway to Twitter. It contains solid basic information about all aspects of Twitter and provides context for a large number of pointers. By following those pointers, I learned everything I know about Twitter. If you invest a few hours reading this clear and comprehensive book, you will be well on your way to understanding how Twitter fits into your life.

Basics of Twitter

To begin using Twitter, go to twitter.com and sign up. Supply your full name, and pick a username that is short and to the point. My username is xrmxrm, which includes my initials and echoes my email addresses and Facebook ID. No username that includes a significant indication of my full name would be short enough to be an effective Twitter username.

Next, set up your profile. Twitter allows you a picture, a 160-character biography and a location. My biography is:

Technical communicator specializing in documentation for programmers -- APIs, web services, programmer guides. Columnist for IEEE Micro.

My location is Berkeley, CA.

Your name, your location, your biography, and your contributions are all that a person considering following your contributions has to go on.

Before using Twitter, you should understand updates, following, @references, retweeting, hashcodes, and searching.

Updates, also called tweets, are your contributions. You can enter a message of up to 140 characters, and everyone who has signed up to follow you will see that message, with your picture, among the tweets that come to them. Anyone can follow your updates, unless you protect updates, which few people do, because there is little point to doing so. You can send updates from a browser or a third party Twitter client, on your computer or your cell phone.

Following is an easily established asymmetric relationship between you and another twitter user. You simply go to that user's page (twitter.com/their_username) and click the Follow button. You can follow other users without their permission and without their following you. Twitter will, if you choose, notify you by email when other users choose to follow you. If you like, you can view their pages and decide whether to follow them. Many people automatically follow people who are following them, but there is no need to do so. Some people follow you as a form of spam. Obviously, you should not follow them. Twitter provides an easy way for users to report spam.

The most interesting uses of Twitter are conversational, so you need a way to refer to other people's tweets. Twitter interprets content of the form @username as a link, called an @reference, to that user's page. By a quirk of Twitter, if you begin a tweet with an @reference, most Twitter users who follow you but do not follow the user referred to in the @reference will not see the tweet. As a result, if you want to pass a tweet to your followers, a process called retweeting, you should place the @reference in the body of the tweet. The standard formats for retweets are to begin with RT @username or to end with via @username.

Twitter has a powerful search mechanism. In addition, it automatically brings to your attention the topics that are most popular at the moment. Search results can contain the tweets of users you are not following, so searching enables you to follow interesting trends and find interesting users to follow.

Often, a group of users interested in a topic choose a search term to include in tweets about that topic. Such search terms usually begin with hash mark (#). These are called hashcodes. As I write this, #iranelection and #michaeljackson are widely used hashcodes. Nobody assigns hashcodes. Users simply start using them. Different versions soon converge, and conflicts work themselves out.

Twitter Helpers

Twitter provides a certain amount of pushed summary information. In addition, a number of third parties follow Twitter, massage the flow of tweets, and make the information easily available. They can notify you about matters of interest to you, so you don't need to read millions of tweets each day. For example, TwitScoop shows hot topics as a tag cloud, in which terms appear in larger or smaller font sizes, depending on their recent frequency.

Another difficult problem for most new Twitter users is deciding which users to follow. Sites like MrTweet provide personalized lists of Twitter users for you to consider following.

If you want others to find your tweets worth reading and retweeting, you should link to interesting material. URLs can be long, and Twitter's limit is 140 characters for your entire tweet, so you need to find a good URL shortener. Such services (for example, TinyURL, Bit.ly, is.gd, and Twi.bz) replace a long URL with a unique, permanent short one that points to the same place. Different shorteners provide different advantages. For example, some give you some control over the resulting URL, while others make it easy for you to track clicks.

Pictures are a compelling component of many tweets. You can post a picture to a site like flickr.com and use a shortened URL to point to it. Others can leave comments at the site or retweet the link with their comments. The twitpic.com site integrates especially well with Twitter.

If you interact with Twitter solely through twitter.com, the primitive user interface soon becomes limiting. You must refresh the page to see new tweets. Simple tasks like replying and retweeting entail tedious manual steps that cry out for automation. Because Twitter provides an application programming interface (API), many third parties have created programs to put a different front end on Twitter. Most such programs run on computers, but others work on smart phones. I have tried Twhirl, SeesmicDesktop, TweetDeck, and DestroyTwitter on my laptop. Each provides improvements over the twitter.com interface, but I keep coming back to twitter.com because I feel more comfortable and less restricted there.

One drawback of the clients I've tried is that they provide little in the way of accessibility. I like to change window sizes and enlarge the text and graphics on my screen. This is easy to do in most browsers. The clients I tried, however, all use Adobe Air. If Air provides that sort of control, these clients haven't figured out how to make use of it. Or if they have, they haven't made it clear to me how I can use it. For that matter, the same criticism pertains to Adobe FrameMaker, which uses Air as the delivery environment for its online help.

What Twitter Is and Isn't

Twitter enables people to share facts, opinions, news, and commentary about everyday life or extraordinary events.

Recently Scott Williams posted an article at http://bigisthenewsmall.com/?p=2170 called Seven Things Twitter Is Not. He makes the point that Twitter is like a river. You can stand on the bank and watch the tweets go by. They are not piling up in your mailbox. Nobody expects you to answer them. If you do toss a tweet into the river, it flows away quickly. Someone may retweet it or reply to it, but the reply floats away too. It may include some context, but 140 characters soon become inadequate to contain an ongoing chat.

Another interesting metaphor is that of a great collective consciousness. As you watch the river, a subject may suddenly attract your attention. For example, on November 26, 2008, reports about the terrorist attacks in Mumbai spiked suddenly on Twitter, drawing worldwide attention before the news media could react.

One frequently asked question about Twitter is, "How can I use it to make money?" Twitter does not seem to have answered that question for itself, but many businesses have found ways to integrate Twitter into their ongoing relationships with actual and prospective customers. The Twitter Book devotes a substantial section to the do's and don'ts of using Twitter for business. No business should jump into Twitter until its management understands the principles that O'Reilly and Milstein lay out there.

You may not understand Twitter at first, but if you try it patiently for a while and get a sense of how others are using it, it will grow on you. Go to twitter.com and give it a try.

Software Architects

This article appears in slightly different form in the May/June 2009 issue of
IEEE Micro © 2009 IEEE.

This time I look at a collection that, according to its editor, is completely different from any other book you've read. That statement is true of any book, I suppose, but this book is unusual.

97 Things Every Software Architect Should Know: Collective Wisdom from the Experts edited by Richard Monson-Haefel (O'Reilly, Sebastopol CA, 2009, 218pp, www.oreilly.com, ISBN 978-0-596-52269-8, $34.99)

Richard Monson-Haefel is a co-author of the O'Reilly books Enterprise Javabeans and Java Message Service. The project grew out of a presentation called 10 Things Every Software Architect Should Know, which he was planning for a No Fluff, Just Stuff symposium. He solicited inputs on a mailing list, and nearly 50 people contributed the "things" that became this book. Each fills a pair of facing pages and includes a contributor photo and brief biography. The regularity of the structure ends there.

The free format allows contributors to formulate their wisdom in whatever ways work best for them. They address a variety of subjects in a variety of ways, but looking at the book as a whole, I see themes emerge. They have very little to say about project retrospectives or managing risk, so I hope they will all read two excellent books from Dorset House. Norman Kerth's Project Retrospectives (Micro Review, May/June 2001) and DeMarco & Lister's Waltzing With Bears (Micro Review Jul/Aug 2003) are essential reading for any software architect.

Communication

Communication receives a lot of attention in this book. Udi Dahan of Microsoft, for example, advises you to stand when you speak to a group. It makes people take you more seriously.

Other contributors stress the importance of knowing both the language of the business decision makers and the language of the software developers. Communication failures between the two groups cause many project problems, and these contributors assign to architects the responsibility for making that conversation work.

Norman Carnovale of Lockheed Martin Professional Services emphasizes the importance negotiating with the business decision makers to ensure that mid-project changes don't lead to unrealistic schedules and ultimately to failure. David Muirhead of the Blue River Systems Group recognizes that the business must drive a project, so he advocates developing the kinds of information that business leaders can use to make good decisions. Yi Zhou lays out the elements that make up a strong business case for a proposed expenditure.

Michael Nygard reminds us that we should remember the difference between a negotiation and an engineering discussion. When business decision makers ask "do you really need x," the answer should be "yes" if that's what the analysis led to. In an engineering discussion of the point, you might say "you could do without it if . . ." but business decision makers won't hear the "if" or anything that follows it.

Another communication theme is that architects must manage their own attitudes so that they can engage in open conversations with developers and customers. They must not let overconfidence or arrogance prevent them from hearing the good ideas of others or listening to the specific needs of customers.

Timothy High of Sakonnet Technologies gives advice that is easy to follow but often ignored. He advocates recording the rationale for decisions. He questions the value of most design documentation because it is hard to keep up to date. The rationale for decisions, on the other hand, doesn't require changes and is a valuable resource for the life of the project.

Requirements

Another theme of the book is requirements. Understanding, examining, and negotiating requirements and turning them into specifications is one of an architect's key jobs. Everyone understands that this process is a negotiation that must balance the needs of customers, funders, and developers. Eben Hewitt puts forth the beautiful metaphor of a consommé -- an extremely clarified broth achieved by repeatedly straining out the solids that cloud it. You must test the language of the requirements again and again until it is clear. This reminds me of the Mary Had a Little Lamb heuristic in Gause & Weinberg's Exploring Requirements (Micro Review, Sept/Oct 2001), another book that all software architects should read.

Many of the contributions boil down to pushing back against stated requirements in various ways. Einar Landre of StatoilHydro advises seeking the value in the requirement. He tells the story of the F-16 Falcon aircraft for which an initial requirement was speeds of Mach 2 to 2.5. By asking why, the contractor was able to change the requirement to an agile aircraft that can accelerate quickly to escape from combat. Similarly, Eben Hewitt tells you not to rush to solve the problem. First interrogate it to see if you can change it into a more tractable one.

Keith Braithwaite of Zuhlke exhorts us to quantify requirements. Fast, responsive, and extensible, for example, are terms that mean different things to different people. Developments based on such terms can lead to serious disputes at acceptance time.

Assumptions, as we all know, can be dangerous. Timothy High advises us to challenge assumptions, especially our own. Urban myths -- or even facts that used to be true -- can lead us away from excellent solutions to design problems.

Not all requirements can or should be satisfied. Good design involves tradeoffs. Mark Richards of Collaborative Consulting recounts the story of the Vasa, an early 17th century Swedish ship that satisfied all of its requirements and sank the first time it fired its guns. Bill de hÓra of NewBay Software says that requirements often come in threes (like fast, cheap, and accurate) and that you should be prepared to pick two and jettison the third. When a requirement seems absolute, be skeptical and ask why. Dave Quick of Thoughtful Arts puts it another way: Scope is the enemy of success. The greater the scope, the worse the chances of success.

Another way to test requirements, according to Stephen Jones, is to stretch key dimensions to see what breaks. This kind of stress testing can identify future problems long before they occur. It enables the necessary changes before the design becomes set in stone.

A number of contributors show concern for end users. Your customer may not be the end user, as Eben Hewitt points out, but your customer won't succeed if the system serves end users poorly. Keith Braithwaite, following Heidegger, uses the german term zuhanden to suggest that systems should be invisible to users. However, Steve Talbott in Devices of the Soul (Micro Review, July/Aug 2007) takes the opposite point of view: Because it is so important to remain conscious of the assumptions and unseen factors that affect us, it's a good idea not to forget about our tools. An invisible tool still embodies the ideals and assumptions of its creator.

Pitfalls

Many contributors offer advice about pitfalls and how to avoid them. Avoiding unnecessary complexity or generalization is a common theme. Neal Ford of ThoughtWorks defines accidental complexity as the complications that arise as byproducts of addressing the essential complexity of the problem. Often accidental complexity comes from adopting frameworks that are too general or fail to match the problem.

Kevlin Henney puts it another way: simplicity before generality, use before reuse. Eric Hawthorne says to choose frameworks that stick to their own domain and don't impose unnecessary requirements. Keith Braithwaite points out that the real world has multiple, inconsistent, overlapping frameworks. Sometimes it works to use each in its own domain. Or as Randy Stafford of Oracle puts it, there is no one-size-fits-all solution. Exercise contextual sense to understand what works in a given domain.

The editor of the book, Richard Monson-Haefel, contributes his own pitfall. Avoid trying to future-proof the system, because you can't predict the future. Trying to guess the technologies of the future can lead to analysis paralysis.

Design and Development Approaches

As you would expect in a book about architecture, design and development approaches receive a lot of attention. Most contributors implicitly or explicitly advocate iterative and incremental processes. Bill de hÓra summarizes this view by saying that great software is grown, not built. Similarly, David Bartlett advocates Martin Fowler's continuous integration (CI) approach, which includes automated builds and testing at frequent intervals.

Dan Chak of CourseAdvisor, however, focuses on designing the underlying database. He feels that the data structures and relationships that underlie applications do not evolve rapidly and require a complex, comprehensive technical design up front. The resulting solid and self-protective data fortress will save application data from the inevitable bugs in the application level.

George Malamidis of TrafficBroker adopts the language of business. He says that every architectural decision is an investment. You should consider the return on investment (ROI). This is true whether you use an agile process or the waterfall model.

George Holpe of Google says welcome to the real world. He implies that most software engineers are clinging to a bygone world of predictably ordered processes controlled by the program rather than by external events. In that world, systems are tightly, rather than loosely, coupled. He quickly tears down that straw man, using the word "scary" a couple of times to characterize the new environment. I get his point, and he's certainly right about the need for distributed, loosely coupled, event-driven systems, but I'd like to point out that we were building event-driven systems in the 1960s, so the idea isn't all that new.

Several other contributors also use uncertainty to their advantage. Kelvin Hanley suggests than when you come to a fork in the road, design so that choosing between the two branches has fewer consequences. Erik Doernenberg of ThoughtWorks suggests trying both branches and deferring the choice until it's obvious which is better.

Performance and Maintenance

Most of the life of a successful software project happens after its initial installation. Many of the contributors address issues about performance and maintenance.

Randy Stafford points out that application architecture determines performance. Rebecca Parsons of ThoughtWorks advocates beginning performance testing early in the development process. This can help you identify problems early. It also gives you an ongoing record that shows which changes helped or hurt performance. Craig Russell of Sun Microsystems takes a broader view and advocates looking at the performance of the developers implementing the design.

Tools and Techniques

I don't have room to go into all of the suggestions about tools and techniques. One that tickled me is Paul Homer's suggestion that you can get a good idea of how a system works by looking at its underlying data structures. In 1967 I attended Butler Lampson's course on operating systems at UC Berkeley. Lampson made exactly that point.

Architects

Many contributions address the necessary qualities or skills of software architects. Many see themselves as leaders of the development project and focus their comments on how to interact with developers. Many say to lead by example or to keep a hand in the development work. The objectives are to maintain the respect of the developers and to experience the design from the point of view of a developer. It's easier to recognize design mistakes if you're the one who has to implement them.

Many stress the importance of maintaining respectful and open relationships with developers. Evan Cofsky of Virgin Charter invokes a taxonomy in Neal Stephenson's Cryptonomicon. Architects are kings and must provide opportunities for elves, dwarves, and wizards.

Many also look at the social or ethical aspects of an architect's work. Barry Hawkins advises valuing stewardship over showmanship. As he points out, you are playing with other people's money. Michael Nygard points out that many decisions restrict or enable end user behavior. A few days of programming time may save end users a few seconds each time they use a feature. With many users and many repetitions, that could be a huge savings for them, but at a cost that you won't recoup. It certainly raises an ethical question.

Yi Zhou tells architects to take responsibility for their decisions. That means putting it in writing and communicating it to everybody it affects. You must also follow through to ensure that developers implement your decision, and you must review the decision periodically to see how it's working out. Taking responsibility also means not making decisions that you're not qualified to make. Delegate those decisions to experts.

I hope this review gives you an idea of the flavor of the book. If you design software, you're sure to find enough nuggets of information in it to justify its price many times over.

Software Testing, Freelancing

This article appears in slightly different form in the March/April 2009 issue of IEEE Micro © 2009 IEEE.

This time I look at two subjects that confuse a lot of people and give rise to a great deal of wishful thinking. Misconceptions about software testing can cause great damage to large and small enterprises. Misconceptions about freelancing can be catastrophic for individual entrepreneurs.

Software testing

Perfect Software and Other Illusions About Testing by Gerald Weinberg (Dorset House, New York NY, 2008, 198pp, www.dorsethouse.com, ISBN 978-0-932633-69-9, $29.95)

Testing is a sampling process designed to obtain information to use in decision making. Some testing activities entail banging on a keyboard, but many others do not. Suggesting a design review, for example, is a form of testing and a developer's reason for not wishing to submit to a review can be a useful test result. Many people see testing purely from the banging on keys point of view. That and other misconceptions lead to painful consequences for them. Gerald Weinberg, a self-described empathic person, has written this book to ease their pain.

Weinberg has been in this business a long time. He was an old timer in 1973, when I first read his enormously influential Psychology of Computer Programming, and he has been an effective and productive writer and consultant in the field of developing and testing software ever since. I have reviewed several of his books in my columns over the years. In addition, many books by other authors, especially those published by Dorset House, show the profound influence of Weinberg's style and approach.

The insights of the late Virginia Satir, a family therapist, inform this book, as they inform much of Weinberg's work about the social aspects of managing software development. Weinberg applies Satir's decomposition of communication into intake, meaning, significance, and response to structure his lessons about how to use testing. He also uses her insights about defensiveness and survival rules to understand the natural immunity to information that people often experience in threatening situations. Nonetheless, there is no new-age mumbo jumbo in this book. Weinberg translates all of his psychological insights into practical terms.

The book's title fallacy concerns the widely held belief among the general public that companies could produce bug-free software if only they tested everything before selling it. This belief falls apart upon cursory examination. Testing "everything" is not possible in a finite amount of time. Furthermore, while testing can identify problems, it cannot correct them. Testing can only provide information. Those in charge of a project can use that information to help them assess the risks associated with various options -- such as shipping the product or sending it back to the developers for more work. This decision is not always easy, because fixing one problem often leads to introducing or uncovering another, sometimes more serious problem.

Because testing can only provide information, you should test only if you need information to help you make a decision. Furthermore, written test reports rarely tell everything that the testers know, so if you want your money's worth from testing, you need to question and interpret the results. It also follows that testers do not produce decisions. Bad managers ask testers if the product is ready to ship. Good managers ask testers what they found, then use that and other information to help them make their own decisions.

Weinberg has seen the insides of far more projects than most of us have and has worked with some relatively dysfunctional organizations. His stories of scapegoating, intimidation, and other misuse of testers are hair raising. If you work under such conditions, you need to read this book right away.

Weinberg seems to have developed a special style just for this book. His chapters average about nine pages. Each ends with a short summary and a long list of common mistakes. The mistakes section reiterates the earlier chapter material in negative form. For example, his chapter on information intake explains why you should choose which information to look for among the large amount of raw data available to you. The first common mistake at the end of the chapter is "Not thinking about what information you're after. Don't just take what comes. You have choices."

As in many of his other books, Weinberg presents some lessons in the form of vignettes. For example, he builds his chapter about major testing fallacies around a few hours in the life of Rose, the new testing manager. The dialog is contrived, and the realistic touches (Rose spears a pickle with her fork and waits for Barry to finish his chili) are lame. Nonetheless, Weinberg uses this form effectively. He shows how the pressures on the different players combine with their partial understanding (or misunderstanding) of the situation to lead to scapegoating, turf battles, and finger pointing.

Weinberg presents a breakdown of bug fixing that accounts for a lot of inefficiency and delay in software development projects. Testing can uncover an anomaly, after which further testing merges into another phase called pinpointing, or isolating the conditions under which the anomaly occurs. Failure to agree on who is responsible for pinpointing leads to many conflicts between testers and developers.

Pinpointing leads to locating, which means determining the section of code in which the problem resides. Few organizations regard locating as part of testing, but a desperate developer might be tempted to return the bug to the tester for "more information."

Between locating and fixing comes another important step: determining significance. This is a management function belonging to neither testers nor developers. Many stakeholders can contribute to this process. Ultimately, someone must weigh the costs of both alternatives. Fixing the bug takes time and effort and might break something else. Leaving it as is can result in other costs, such as service calls or lost sales.

One of the biggest problems associated with testing is how it interacts with the schedule. In an ideal development process, testing begins before the project, because deciding whether on not to do a project depends upon information gleaned from some form of testing. Then testing continues through all phases of the project. Unfortunately, most project managers follow a different model.

Typical projects have schedules that include a block of time labeled testing at the end. This block does double duty as a buffer to accommodate slips in the development schedule. In a good project, testers participate in reviewing the usual project artifacts -- market requirement documents, functional specs, user interface specs, detailed design specs, and documentation plans. In a bad project they may be handed a functional spec and told to develop a test plan that tests everything. In either case, unless they work alongside the developers throughout the project, they do not usually find problems until long after those problems enter the product. What could have been fixed easily months earlier now becomes an obstacle to delivery. A project that was "on schedule" until the testers got hold of it is now in danger of slipping. At that point, it's easy to blame the problems on testing.

Another distinction that Weinberg makes is between testing and demonstrating. If you know the "test" result in advance and are only running it to give potential customers a good feeling, you are not testing. Testing always seeks to find new information. Somewhere near the level of Gianni Schicchi in Weinberg's Inferno are the sellers of expensive testing systems who use such fake tests to support unsupportable claims about their products. Vendors who claim that their products will do all the work without receiving extensive explanations about what the programs they are testing are supposed to achieve are simply charlatans in Weinberg's view. Weinberg is firm in his belief that machines will never replace brains when it comes to software testing. Testing is an interactive process of investigation in which each finding leads to new questions.

Like all of Weinberg's books, every page of this one has wonderful insights and advice. If you have anything to do with software development, you should get a copy and read it at least twice.

Freelancing

The Principles of Successful Freelancing by Miles Burke (Sitepoint, Collingwood Australia, 2008, 206pp, www.sitepoint.com, ISBN 978-0-9804552-4-3, $39.95)

SitePoint is a small publisher targeting web professionals. Its books deal with SQL, CSS, HTML, ASP.NET, and similar staples of the browser-based world. Miles Burke is a freelance web developer, and he has written a book for others who wish to enter that arena.

I confess that I was put off by the lax editing and the Australian jargon of this book, but if you can get past that, it has a lot of useful content for people who want to set out on their own. Other books for freelancers (for example, Secrets of Consulting by Gerald Weinberg) focus on advanced topics. This book lays out the basics. If you read this book carefully, you won't make any major omissions as you prepare for your freelance career. In fact, at every stage, Burke puts on the brakes and urges you to move slowly.

Many people jump into freelancing because they have just lost their jobs. Such an approach has little chance of success in most cases, because starting a business requires a financial cushion that few recently fired workers have. Also, those who have never worked for themselves may overlook expenses like self-employment taxes, insurance, sick days, vacations, and the like. Burke shows you how to make the necessary computations and set your rates accordingly. You may have to adjust a little if you don't live in Australia, but many of the points he makes apply in the USA as well.

Burke covers all the bases -- from not using stolen software to negotiating with your family for an efficient work area in your home. He also gives good advice about how to find work, how to relate to clients, and how to balance life and work when they all happen in the same place.

If you are considering a career as a freelancer, especially in the web design area, you should find this book well worth its price.

Hot, Flat, Crowded

This article appears in slightly different form in the January/February 2009 issue of IEEE Micro © 2009 IEEE.

The Italian political philosopher and pragmatist Niccolo Michiavelli once wrote about the difficulties of change. An English version of his much quoted aphorism is as follows:

There is nothing more difficult to plan, more doubtful of success, more dangerous to manage than the creation of a new system. The innovator has the enmity of all who profit by the preservation of the old system and only the lukewarm defense of those who would gain by the new system.

This time I look at a prescription for a new system for dealing with the world's energy needs.

Hot, Flat, and Crowded: Why We Need a Green Revolution and How It Can Renew America by Thomas L Friedman (Farrar, Straus and Giroux, New York NY, 2008, 448pp, ISBN 978-0-374-16685-4, $27.95)

Tom Friedman has been with the New York Times since 1981, when he signed on as an energy reporter. He is now a foreign affairs columnist. His first two books deal with the Middle East. His third, The World Is Flat (see Micro Review, May/June 2005), deals with globalization. In this, his fourth book, he returns to energy, bringing to bear his broad understanding and experience of global issues and the Middle East. Though the book is about global systems, he has written it as an American and addressed it to Americans. 

Friedman uses the phrase "hot, flat, and crowded" to refer to the three themes that are pushing the world into what he calls the Energy-Climate Era: 

• Global climate change.

• The emergence of new, globally connected middle classes in China, India, Russia, and elsewhere.

• Worldwide population growth.

He calls it the Energy-Climate Era, because energy and climate are the principal issues that the world will face for many years to come. These issues underlie and manifest themselves in the following big problems:

• Increasing population and the rise of middle classes have led to a sharp increase in demand for natural resources and energy.

• The huge amounts we spend on crude oil and natural gas have strengthened dictatorships in Russia, Venezuela, and the Middle East and funded some of our worst enemies.

• Vast emissions from energy production have overwhelmed the mechanisms that kept the level of carbon dioxide in our atmosphere relatively constant for a long time. Increasing levels are causing worldwide climate change.

• Access to energy has become increasingly important, but more than a billion people are not attached to a power grid. Many of these "energy poor" are using inefficient, dirty, ad hoc methods to obtain electricity.

• Flattening and crowding have led to destruction and disappearance of critical ecosystems. Species are disappearing at an alarming rate. 

Because many people contend that human activity is not a major factor in climate change, Friedman addresses that issue -- first with a lot of science and then with a Pascal bet. The mathematician/philosopher Blaise Pascal argued that even though you might not be able to prove it, you should live as if God exists. Such a lifestyle, he contends, is intrinsically good, regardless of whether God exists or not. Applying this reasoning to climate change, Friedman says that changing our economy to use energy efficiently and to generate it from clean sources will benefit us greatly, even if "global warming" is a hoax. Of course, getting to that changed economy is not trivial, despite the proliferation of articles with titles like Seven Easy Ways to Save the Earth in Fifteen Minutes a Day. Friedman impresses us with the magnitude of the task.

The current level of carbon dioxide in the atmosphere is 270 parts per million (ppm). If human activity stops tomorrow, the level will continue to rise because of mechanisms already in motion. Friedman believes that a workable goal is to manage the unavoidable and avoid the unmanageable. He translates this into slowing the growth of the carbon dioxide level, so it reaches a maximum of less than 550 ppm around 2050. Robert Socolow and Stephen Pacala of Princeton University have computed the scale of the necessary programs, assuming that those programs start now and are completely in place by 2050. Some of their programs overlap or are alternate approaches. Here are some of the key programs:

• For two billion automobiles, double the fuel efficiency to 60 miles per gallon, halve the number of miles per year to 5000, or change the fuel to ethanol or hydrogen.

• For 800 to 1600 large coal-fired plants, sequester all of the carbon emissions, increase efficiency from the current 40% to 60%, or fuel them instead with natural gas. 

• Replace all coal-fired plants by generating twice as much energy as currently from nuclear plants, 40 times as much from wind, or 700 times as much from solar.

• Cut by 25% the electricity used by homes, offices, and stores.

There are more, but these give an idea of the scale of the problem. 

Friedman explains the current power distribution system. It is a loosely connected network (called a grid) of power companies (called utilities), all regulated by the states in which they operate. The utilities all participate in the following bargain: they can operate as monopolies as long as they provide a cheap, reliable, ubiquitous source of electrons. Unfortunately, the grid provides little support for the kind of innovations needed to reduce power use. The grid is huge, unintelligent, decentralized, unregulated, and poorly integrated. Its main defects are that it has no two-way communication, and there is no way to vary the price by load or time of day. Furthermore, the utilities operate under a business model that rewards them for generating and selling more power but not for promoting or facilitating efficiency. Friedman sees a smart grid and a new business model as essential to a sustainable energy system. The question is how to get there. Friedman's answer is to encourage innovation.

Near the bottom of page 243, Friedman buries the following:

If you take only one thing away from this book, take this: We are not going to regulate our way out of the problems of the Energy-Climate Era. We can only innovate our way out, and the only way to do that is to mobilize [the marketplace].

This is the essence of Friedman's proposal for replacing our current unsustainable energy system with a sustainable one: shape the marketplace so that this is the inevitable outcome, then let the free market system work. For those who don't consider this an example of a free market, Friedman has an answer: our current free market isn't free either. He points out that what he calls the fraudulent hiding of the true costs of resource extraction and consumption shapes the current market. In addition he cites regulatory anomalies like the difference in import tarrifs between Brazilian ethanol (54 cents per gallon) and Saudi Arabian crude oil (3 cents per gallon).

Friedman says that the market is like a garden to be shaped into an ecosystem for energy innovation. The shaping takes the form of an intelligently designed system of policies, tax incentives/disincentives, and regulations. He doesn't want to pick winners, but the entrenched interests are like kudzu. Uprooting them might be the hardest part of the job.

Friedman believes that the simplest way to shape the market is to provide a price signal, and he quotes from his interviews with the chief executive officers of General Electric and DuPont to support that view. A price signal can take many forms, but in essence it is a fixed target that developers of alternative energy sources can shoot at in their long term planning. To understand why this is important, consider the price of crude oil, which is largely controlled by a cartel, the Organization of Petroleum Exporting Countries (OPEC). The price tends to rise and fall by substantial amounts. Its rises stimulate development of alternative energy sources. Then it falls, and the companies developing the alternatives go out of business.

One price signal would be a government guarantee of a minimum gasoline price. For example, when gasoline prices exceeded $4.00 per gallon in the summer of 2008, the government could have said that for each dollar the price dropped, the tax would increase 95 cents. This would have told alternative fuel developers what price to shoot at. Instead, as autumn gasoline prices dropped to below half of the summer levels, alternative fuel developers were forced to change or abandon their plans.

Another kind of price signal, applicable to all energy production, not just vehicle fuel, is a carbon cap-and-trade regime. In this approach a controlling authority sets a limit on the total emissions of carbon from all sources in an upcoming time period. It then distributes individual limits to all possible emitters in the form of credits. The credits add up to the total emissions limit. Emitters who do not have enough credits to cover their emissions must buy them from someone else or pay a severe penalty. At the end of the covered time period, the authority sets a new, presumably lower, emissions limit for the next period and issues new credits accordingly.

An alternative to a cap-and-trade regime is a carbon tax. Emitters pay an amount that depends on the amount of carbon they emit. This is simpler than a cap-and-trade system. Each has advantages and disadvantages, which Friedman explores. Although the cap-and-trade system gives "environmental certainty" by limiting total emissions, Friedman prefers the carbon tax because it is more transparent.

A price signal that some countries (and Texas) already use is a renewable energy mandate. The controlling authority specifies a percentage of total power generation that must come from sources (for example, windmills or solar cells) that do not emit carbon. The authority then increases the percentage over time. European countries have done this to varying degrees, which is why the US company First Solar finds more business in Germany, Spain, France, Greece, and Portugal than in the United States. 

Regardless of the form it takes, a price signal requires our leaders to make a long term commitment to a complex plan. This is why Friedman asks whether democracy can survive complexity. He envies the Chinese government's ability to dictate policies and have a reasonable expectation that all levels of the society will implement those policies. "If only we could be China for a day," he says, "but not for two days." In earlier crises, leaders took control to enable them to act decisively. Lincoln asserted power over the states, and Franklin Roosevelt expanded and strengthened the Federal government. 

US energy policy, like our system of government, is designed to make inaction easy and transformational action hard. Among the cooks stirring this broth are state regulators, the Environmental Protection Agency, the Department of Transportation, the Department of Energy, the Department of Agriculture, the Army Corps of Engineers, and the Federal Energy Regulatory Commission. Friedman characterizes the resulting policy as follows: maximize demand, minimize domestic sources, and make up the difference by borrowing huge sums of money to pay to those who hate us. To centralize energy policy, Friedman proposes a real Department of Energy. He points out that the current department with that name is concerned mainly with watching over our nuclear weapons.

Friedman provides many success stories of green projects. For example, US troops in Iraq used diesel-powered generators to provide electricity for a remote desert base. Transporting diesel fuel to the base exposed them to the danger of attacks that were hard to defend against. By insulating their tents, they were able to reduce their air conditioning power needs to a level they could provide from wind and solar sources. They even had power left over to give to a nearby village. This example shows the power of a price signal. Considering the costs of transporting and defending shipments of diesel fuel, the Army found it easy to justify the costs of becoming energy self-sufficient.

Another example is the conversion of 7.5% of New York City's taxicabs to hybrid vehicles. This example shows the effects of regulation. The city had to remove regulations that forced taxis to be large and heavy. Moving forward from this success, New York is trying to achieve similar increases in the efficiency of limousines.

A main thesis of the book is that transforming our energy system is a key to remaining a world leader in the years to come. If we do not lead, other countries will, and we will fall behind. If other countries, particularly China, try to follow the same path we followed, we have little hope of averting catastrophic climate change. If we lead the way to a green revolution, other countries will follow, and we will remain a world leader. In other words, as Friedman puts it, green is the new red, white, and blue.

Friedman examines what is happening in China. As the world's most populous country and the one with the fastest growing economy, China is the key to successfully transforming our energy system. The Chinese government seems to have recognized the problem and committed themselves to producing cleaner energy, but they have competing priorities. The bargain between the Chinese government and the Chinese people is "we rule, and you rapidly become more prosperous." This means that they can't afford to stop the bus. They have to swap out the dirty engine and replace it with a clean one without slowing down. 

This review focuses on a few key points, scratches the surface of others, and completely ignores parts of the book. Friedman is a thorough and careful reporter. As he showed in earlier books, he is good at understanding how issues interact. He tells compelling stories to illustrate his points. If you want to understand the workings of the worldwide energy system and how it interacts with population growth and climate change, you should study this book.