The ROI of Test-Driven Development

Leadership wants to know:  "Why should my teams be doing TDD?  What is the return on investment?"

Though TDD is, of course, not a "silver bullet," we've found that TDD solves a number of key software development challenges, including those that are exposed by the frequent releases and high visibility expected of an Agile team.

(For background on these challenges, read about Technical Debt and the Agilist's Dilemma. To grasp the root systemic problem, read my "new" twist on the old Iron Triangle.)

I'll outline some benefits, and some costs.  This is probably not a complete list, but represents the most significant benefits and costs that I've observed.  Where appropriate, I've summarized in a bold, italicized sentence, so you can absorb the main points quickly, or delve into each item in detail, depending on your current needs. 

The Biggest Benefits of Test-Driven Development

 

Defect Reduction

Functionality that doesn't function provides no business value.  TDD allows us to start with high quality, thus providing real value.

Often the most immediate and obvious detectable benefit of TDD is the reduction of defects (60% to 90% fewer, according to this PDF describing the IBM/Microsoft study).

The key to this benefit is the creation of faster feedback loops. If a developer can find a mistake instantly, before check-in, then this is typically weeks (or months) before the bug would otherwise be identified by traditional testing efforts or code reviews. This early detection and elimination of defects avoids the waste of rework involved in finding and fixing a bug, plus having to assign a developer to do the work, and to re-familiarize himself with that region of code.

TDD reduces both (1) defects caused by guessing (incorrectly) that a particular implementation will behave as expected, and (2) defects caused by adjusting design to add new functionality to existing code.  A freshly-written test will catch the former before it's ever checked-in, and the existing suite of previously-written tests will prevent the latter.

When a defect is found (they will still happen), we add any missing tests that describe the defect. We do this even before we fix the defect. Once fixed, that defect can never resurface.

Faster Feature Time to Market (aka Cycle Time)

TDD allows developers to add innovative features as rapidly as the founder did while hacking in her garage, without damaging the original investment.
 
Comprehensive testing and rapid feedback provide complete confidence that any changes to the code (for new features or to improve the design) can be made swiftly, without breaking existing functionality.

When teams don't have to worry about breaking stuff, they can add new behaviors and features much faster.  This gives us a shorter cycle time for innovative features.

I have a number of first-person stories (which I'll share in a future post) where the teams I've worked on (as a contributing XP coach) have been able to turn around a "major architectural change" or "drastic re-purposing" in less than a week.  Had we not been following our TDD engineering practices, we believe the estimates would have been measured in months, not days.

In my experience, most of this benefit is taken for granted (after all, what do we have to compare it to?). Yet even when the great delightful surprises have not yet happened, the whole team (including Product) often notices a subtle shift towards a sense of ease and pride in the team's productivity and the quality of the delivered software.

Improved Focus

TDD helps developers think clearly about the task at hand, without having to continuously hold the whole complex system in their heads. They avoid over-extrapolation and over-design.

"Thinking in tests" allows developers to craft their code by (1) stating a goal for the code (as a new test), (2) confirming that this is indeed something new, (3) implementing just enough to get it to work correctly, (4) reshaping the design to fit well within the rest of the body of code.  This approach gives them opportunity to reflect on needed behaviors and edge cases without over-engineering.

Additionally, developers avoid the frequent context-switching incurred by bug-hunting, and recoup a tremendous amount of time typically spent in the debugger.  During the entire year of 2002, working full-time on a single product, I recall our whole team firing up the debugger only once or twice (averaging once every 6 months).  Compare that to a developer I met who claimed he spent 80% of his time in the debugger prior to TDD.  Even if that's an exaggeration, his rough estimate suggests a large amount of wasted time spent bug-hunting.

Parallel Efforts Without Conflict

Since, with disciplined TDD, each developer (or pair of developers) is expected to run all of the tests in the regression suite and confirm that 100% pass before committing new changes, there is much less opportunity to disrupt or destroy someone else's hard work.

If your team is larger than two developers, then they will likely be working simultaneously on different, but possibly related, areas of the system.  Without a discipline of writing and running comprehensive, pinpoint-accuracy tests, we often damage a prior check-in when we merge or integrate files.  Some teams will instead create a complex tree of code-repository branches, which is really just a way of kicking the can down the street.

The Costs of Implementing a TDD Discipline

As with any investment, you will want to consider costs.  And TDD, like any worthwhile discipline, has its price.

More Test Code

"Twice as much code!" people often protest. Yes, unit-test code is code, and needs to be written by a developer.  Also, test-code tends to be more script-like and verbose. It is, after all, a form of developer-to-developer communication.

Actually, unit-testing code using TDD tends to result in a 2/1 ratio between test code and production code.

"Three times as much code?!" Not exactly...

If you are concerned that so much test code will create a bottleneck, you will be happy to know that TDD tends to reduce the amount of production code it takes to solve a business problem.  TDD reduces duplication by allowing teams to avoid copy-paste coding techniques and stove-pipe solutions.  (A "stove-pipe solution" is one where each activity, page, or feature has its own full set of code at each architectural layer.) Teams that do not do TDD may rely on copy-paste in order to avoid breaking anything that already works, or to avoid interfering with another developer's parallel efforts.

So, though the ratio between test code and production code may be 2/1, it may not really be three times as much code as would otherwise be written.

Besides, in knowledge work, typing speed is not your limiting factor.

Learning Curve

At first there will be a slowing of delivery of new functionality, as the team gets used to using the TDD practice, and as they pay down small high-risk portions of the technical debt.

The Microsoft/IBM study noted an initial slow down.  This is true whenever we begin a new discipline.  Think about starting a new exercise regimen, or start flossing your teeth, or learning the piano: At first, practicing slows you down, it hurts, and it may not seem worth the effort.

After a few months, teams start to rely on their safety-net:  That comprehensive suite of unit tests.  This gives them courage to make bold moves, and to check that those moves haven't reduced the value of the software.

 Developers report useful feedback resulting in an increase in "flow" in short order: usually less than 30 days. (Often, a majority of participants in my 3-day Essential Test-Driven Development course will report that a single surprising test-failure, or a well-tested refactoring, helped them see how TDD is a superior way to craft code.)

Business results can take a little longer: TDD reduces both old and new technical debt, and allows new, even unexpected, functionality to be added rapidly. It may take some reflection on a smoother release to market, or the occurrence of a very short cycle time, before the business is aware of this ROI.

TDD may at first seem like an exotic and esoteric discipline, and possibly quite challenging.  Actually, people find TDD easier to learn than good "test-after" unit-testing. I like to say learning TDD is like learning to walk on your feet.  You've been taught in college and/or in the industry that walking on your hands is "the way it's done in our industry."  I spent 13 years writing code before testing, and 13 years doing TDD. I find TDD to be more natural for my scientific/engineering/inquisitive mind, and far more satisfying.

I've trained and coached hundreds of teams on how to "walk on your feet" using TDD. The approaches that have been most effective:

• My 3-day Essential Test-Driven Development course followed (at some point) by a day of coaching around a team's specific technologies, existing code, and other challenges.
• -or- At least eight days of intensive coaching with a dedicated, enthusiastic (or desperate) team. I work with pairs of developers on their development tasks.  Sometimes we're writing characterization tests for legacy code, sometimes we're testing the "untestables," and sometimes I'm providing a half-technical, half-emotional system of support that allows people to take courageous steps that they would otherwise never take.

Some teams have done it on their own.  They'll hire experienced, "test-infected" developers to cross-pollinate the team over time.  Or they'll read the books and coach each other over time.  These have the added cost of creating extra technical debt until the whole team is doing TDD in a disciplined manner.

Test Maintenance

The test code does have to be maintained, and with the same relentless discipline as the production code.  But if it's done as ongoing refactoring towards maintainability (i.e., writing the next test), it can be managed such that it never grows into a big mess.

Teams that refactor regularly (every two or three minutes) find that both the tests and the code behave themselves; they never grow into monsters.

In fact, teams who have been following the discipline for a year or two have reported to me that they spend a majority of their time and efforts on test maintenance.  Though this sounds horrific (and expensive) on the surface, they are really saying that they need only expend minimal effort on refactoring or designing their production code.  These teams are, truly, "Test-Driven."

A Good Investment?

Each manager/exec/leader will need to do the cost-benefit analysis for her own teams.

In my 26 years of experience--half with teams building software prior to TDD and half with teams doing TDD wholeheartedly--I find this one discipline to be the most potent practice in the Agile Engineering toolbox. I am thoroughly convinced that a focus on quality leads to real productivity, and that "thinking in tests" ("test-driven" and "behavior-driven" are other ways of saying this) leads to clarity of requirements, just enough engineering, and the highest levels of quality.

Let your teams try walking on their feet again. You'll be pleased with the results.

Gentle Discipline: Making Agile Happen in the New Year

Update #1 for 2013:  When this was first posted, I had a few people point out that I was talking about self-discipline.  True enough!  I'm not sure I can conceive of any other form of successful discipline.  It's possible that I'm biased because I'm an Agile Coach; or--more likely--that I've become an Agile Coach because of this bias.  The "User-Friendly Discipline" is my name for an observed phenomenon, not for something I thought up. It is the only form of real discipline I've ever known to work, in my own life as well as the lives of those I've encountered, over nearly five decades, and on three continents.  The assumption that you can force other people to do something as a discipline is, to me, disproven by my observations of successful disciplines (as described below). YMMV, but I doubt it. -- Rob

It's 03 January 2011, and I'm going to wait one more day before going to the gym. If you've ever had a gym membership, you know exactly why: Right now, every gym is a zoo of humanity. Right now, hundreds of new people join the gym and try so very hard to live up to their New Year's Resolutions.

But, as the regulars and the staff are aware, most of those newcomers eventually give up and stop coming. Though I'll be happy to have things back to normal, I always feel a little sad for the people who weren't able to establish an ongoing discipline.

The D Word
In my courses, I often ask people what comes to mind when they hear the word "discipline." Images arise of strict teachers with rulers, or boot camp drill sergeants. The word seems to imply rigorous, strenuous rules, extra effort, and--above all else--suffering.

Many people stop going to the gym because they start out too intense. Often their fit (and younger) personal trainer has constructed a regimen suitable for a Marine; or they just push too hard the first day, and still can't lift their arms above their head a week later. Or they tried to dedicate all their free time to this one endeavor, and life (laundry, house-cleaning, overtime, friends) intrudes; or at least provides an excuse.

Discipline: The User-Friendly Definition

The way to establish a consistent, painless discipline is embedded in my formulation of the "acceptance criteria" for a discipline.

1. A discipline is a habit...
2. ...that you come to rely on, even during difficult times...
3. ...because you know that it benefits you, personally.

Let's examine each of these further, and I'll make some suggestions on how to implement this.

A Habit...
A habit is something we do almost unconsciously, and frequently. There are good habits as well as bad. Think about the numerous habitual disciplines you've already established in your daily routine before going to work. Most people shower, put on clothes, and brush their teeth. If you think back, you had to be taught all of these, and some of us probably put up a fuss over these disciplines as children. Do they still seem painful? Would you give them up?

I recall learning (perhaps many decades ago) that to establish a habit, psychologists suggested we repeat the behavior for about 30 days. Whether this rule of thumb is still canon or not, I've found that we must remind ourselves, and perform the activity, repeatedly until we know we can discard the reminder.

A sticky-note on your mirror could remind you to floss. A daily alarm on your calendar software could remind you of the stand-up meeting. A big, visible chart graphing the number of unit-tests checked in to the repository could encourage the team to write tests (first! ;-).

...We Rely On...

For a practice to become a stable discipline, enough people on the team (it could be a team of one) have to be doing it habitually so that it will not die out due to some minor random event (e.g., when two new developers are added, or while three people are out with the flu).

A Tale of Two Teams

I've observed at least two Extreme Programming (XP) teams as they discovered they needed to rush out a release. One organization decided at the last moment to show the product at a trade show. On another team, there was a customer anxious for an early Beta release, and willing to pay.

The reaction of these teams suggested their levels of discipline with various practices. The less-experienced team gave up TDD, pair-programming, refactoring, code-reviews, object-oriented goodness, all manner of sanity; and just cranked something out. The product failed to "wow" anyone, to say the least.

The other team acknowledged the crunch, took a little time to re-prioritize and break down the backlog, then sat down in pairs and practiced their testing and engineering disciplines as though nothing untoward had happened. Their release may have been short on bells and whistles, but it was also very short on defects. (And they sold many licenses and the VCs lived happily ever after, The End. There be Agile Dragons, remember?)

A key lesson from these events was that a discipline isn't ever something we think is "a great idea, in an ideal world, but we're too busy right now in the real world, so could you bug us about it later?" You have to know its true value, in your bones.

Frequency Over Intensity

Lather, Rinse, Repeat. Always Repeat.
-- Homer J. Simpson

To get there, it's important that you repeat the activity rather consistently before a crisis occurs, and through some minor daily crises. Even when, especially when, it's painful to do so.

Here's one of the user-friendly parts: Early on, frequency is more important than intensity. You can tone down the practice until it becomes a habit.

An example for the resolute exerciser: If your muscles are sore, just go do your cardio. (If your regimen is purely cardio, then go play lightly on the weight equipment.) Just go have fun with it.

An example for the blossoming meditator: No time to practice? Sit down and breathe slowly and deeply three times, then get up and go to work. (And try not to curse at traffic lights, or other drivers. ;-)

For the Agile team participant: Try a new practice wholeheartedly for a month (only one to four iterations). You know you can ask the team to reexamine and tweak the practice at every retrospective; so don't evade, grouse over, or sabotage the team's efforts. Do it as though you believed in it, and that your project or career depended on doing it well.  (It does.)

My friend Bob Hartman once helped establish a team's pair-programming discipline by asking the team to pair up whenever they were fixing a bug. "If one person broke it, why do you think one person can fix it?"

...Because It Benefits Us
We won't even attempt a new discipline without some indicator that it's valuable. We observe people who are healthy and happy, and they exercise in some way (among other things). We observe teams who produce high-quality software, and they're using fast, automated testing (among other things). We want that!

At the team or organizational level, we want fewer defects, faster time-to-market, more profit!

Weigh Costs, Too

If the individual, team, or organization picks up a new practice, the benefits are probably well-documented. What may not be clear from the literature are the costs. Both have to be considered in order to select the right set of practices for the organization (or organism).

Unfortunately, the costs--or aches and pains--are always more intense when you first start out. So, there's frequently early resistance ("emotional re-evaluation"), which could stifle the burgeoning discipline.

It's Personal

The biggest hurdle hiding within the acceptance criteria may be even more daunting: In order for it to be a discipline, each participant must have experienced (past tense) at least some small personal benefit from the discipline.

Effective disciplines are not followed out of altruism, nor are they followed because people are getting paid to do it the "right" way. Even Mother Theresa admitted to personal benefit and pleasure in helping others. And I've worked with many large organizations who were paying good salaries to people who didn't really do much: They were mostly trying to stay hidden, and waiting for their pension to kick in.

Even those employees are not very happy in their careers. People need more than a big paycheck to feel happy. What's sad is that many of them don't know this. They think "that's the way things are, and there's no way to change the system."

Every good discipline has something in it for you, the practitioner! The benefits of my sample disciplines (exercise, flossing, meditation) are clearly personal, and entirely obvious, right? ;-)

How about Agile disciplines? "What's in it for me?" Here's just a sampling, regarding some of the most peculiar Agile practices ever devised:

Paired programmers report that they are more confident in their solutions. Their defect-density drops, and they spend far less time in the debugger. They feel that they--even the old-timers--are learning from their peers at an exhilarating rate. They are more comfortable around teammates, and more confident in their discussions with management. They have fewer carpal-tunnel-related complaints. Ultimately, the feel more productive.

Disciplined TDD developers report an uncanny level of confidence in the quality, design, and adaptability of their code. At first the costs seem prohibitive: Extra test code needs to be written; and since the design emerges over time, time is spent continuously refactoring (which is the design mechanism in TDD). But after only a few months, they are able to add new behavior into the system in days, rather than weeks; and they use tests to seek out, isolate, and crush defective scenarios (bugs). They report that they recoup nearly all the time they would otherwise spend in the debugger. (I've had developers tell me that they used to spend half their time debugging. What would you do to recoup almost 4 hours of each day?)

Once people get a taste of these benefits, they have a much easier time continuing the practice, and improving upon the required skills. For them, it has become a real discipline.

Additional Suggestions
Stop Lamenting the Past

If your blueprint for success contains (or implies) "Step 1: Build Time Machine" then you need to rethink your plan.

In order to establish a discipline, you must stop dwelling in the past. You're out of shape? You can't leap into strenuous exercise. You have a million lines of untested code? The optimal path is not to take a year off to cover all that code with characterization tests. Hundreds of severity-one defects? Funding a complete rewrite isn't going to help.

There isn't a single discipline on Earth that involves wishing things were otherwise, or fixing everything that's broken.

Stop Blaming

Never punish or berate yourself or your team for the occasional relapse into those old 2010 days.

Acknowledge the lapse, yes. Perform root-cause analysis and see if you need to jiggle your circumstances to prevent a future relapse.

Let me ease your concerns here and now by assuring you that lapses will happen. So, now that you already know, you won't have to argue with yourself about perfection and responsibility. With each lapse, you simply have to re-commit to practicing now and in the future. (And, yes, you will have to pay your personal trainer for the missed session.)

But you can't try to make up for it. One of the worst mistakes I see in the gym is working out twice as long after missing a session, or after eating too many holiday treats. That path leads to further discouraging circumstances. Sure, if there's a minor mistake that can be corrected (e.g., by adding unit-tests around new, untested code), then do it. That's correcting a mistake. "Correcting" a relapse involves extra time practicing the discipline, and that invariably eats into the time needed by some other practice. Tomorrow is another day.

Does this conflict with my suggestion to focus on frequency over intensity? No, because I was not suggesting that if you go a day without (flossing, meditating, a daily stand-up meeting) that you have to start the 30-day clock all over again in order to build the habit. When you fall down, get up, dust yourself off, and get back on the bike.

Stop the Excuses

Don't over-plan your efforts towards a new discipline. A complex plan is full of opportunities for reversal. What we're really doing when we over-think the practice is creating a mental model of our lives where the practice is ultimately not worth the effort.

That is, we invent our own cosmic excuses. This takes a certain level of cleverness. If we are to try something that doesn't come with a guarantee of instant gratification, this cleverness must be tempered with courage.

You're intelligent. You're courageous. What are you planning to try this year? Add a comment and share your experiences.

Update #2 for 2013: I've noticed another impediment to creating a healthy new discipline:  Embarrassment, or peer pressure. "Why must you _____?!" people will ask (or imply), incredulously. I have a number of very old habits that a few people have (openly and vocally) suggested were professionally lazy: Exercising three times per week, meditating for 1/2 hour per day, and getting eight hours of sleep on most nights.  I don't tell you about these now to be self-aggrandizing (or self-effacing) but as factual examples:  For me, they are so routine that when I have to drop one, I become cranky, and much less effective at work. They are all wonderful, productive, professionally supportive, healthy habits. But disciplines that are counter to prevailing culture will meet with resistance.  If you can use your own facts and experiences, you will be able to politely and clearly convey the benefits to those who are skeptical. That's how culture is slowly changed. Have courage!

Have a happy and prosperous New Year!