Productivity Versus Quality in the Iron Cage of Death

The Classic Iron Triangle

(Re-)Introducing the Iron Triangle

You and your team, division, tribe, or organization are given a task. You must complete the whole project, you don't have extra money to hire more people, and it has to be completed by a specified date.  This is known as the "Triple Constraint" or the "Iron Triangle":  All variables have been pinned down: They're not really variable by the time you see them.  

If the project does succeed, the next project will be constrained further.  "Good job! Now let's see you do more with less."

Despite the fact that we've known for thousands of years that this arrangement is likely doomed to fail, the Iron Triangle still survives within many corporate cultures.

Why don't we see the fallacy behind the Iron Triangle?  (1) Because the failures occur after the project is completed. And (2) because the Iron Triangle itself is a flimsy model.

Performance as a Function of Time, Effort, and Size

Performance According to the Five Core Metrics

I must have misread this book.  I recall reading about time, effort, and size, and then reading about how performance is a function of the other three.  Let's see, how many metrics is that? 1 + 3 = 4.

Then the 5th metric is introduced: Quality!

I recall being stunned that it wasn't included in the performance equation. Were the authors really suggesting that, as long as we meet the Triple Constraints, we were providing value even if we were producing total crap?

I must have misread this book.

Remember Quality

I used to start this client-coach conversation with "slow down to speed up."  The problem with "slow down to speed up" is that it's as valuable as "buy low, sell high": True, but absolutely useless.

The Wobbly Iron Rhomboid

A better metaphor is that of the chef who sharpens her knives before cooking. This simple practice makes chopping safer, and thus the chef can chop with more confidence that the knife is going to go where she wants it to go. All chefs learn this, and those who choose not to follow this simple advice often end up with a mangled finger, a trip to the ER, or worse. Skipping this essential practice is a false "short cut" that leads to wasted time, effort, and money.

Software development is chock full of false short cuts.  When we ask our teams to deliver everything, on-time, at cost, we have pinned down the three corners of the Iron Triangle.  Of course the team will find ways to achieve your goals, consciously or unconsciously.  And there's only one place where the Wobbly Iron Rhomboid can wiggle if those three corners are pinned:  Quality will suffer.

"Yay! We've released version 1.0!" A resounding victory, until the customers start to use the product.
 

Lead with Quality

I consider myself experienced enough to know why start-ups need to rush to market.  And yet, having worked on both highly successful software, and at least two disastrous VC-funded dot-COM start-ups, I have noted this:  Companies and products do much better over time if they deliver fast, and they deliver less than everything, with high quality.

What do you get when you add features to buggy code?  More buggy features.  If the feature doesn't work, the product hasn't provided the intended value, and the customers wander off. When we push teams to deliver more and more functionality, without giving them the routine breathing room to pay down the technical debt created by rushing, we risk product melt-down.

The team must be encouraged, very early on, to establish disciplines that create high-quality features, and preserve the quality of those features over time.

Teams who do this frequently discover that these same disciplines improve their ability to add features later on, reducing the time it takes for an innovative new feature to reach production. In other words, disciplines that establish and preserve high quality also improve the throughput of value.

So, as with every "Iron Cage Death Match," everyone wins. By building quality in, productivity--the real throughput of value--is also increased.

How does a software development team "sharpen the chef's knives"? Stay tuned!

The Agilist's Dilemma

Let me tell you a dirty little secret: An Agile approach to crafting software comes with its own challenges.

I'm guessing you already knew that.  Does the following tale seem familiar?

Fields of Green

Imagine you start developing fresh new "greenfield" software today using the basic Scrum process framework (for example), and little more.  The developers design and build just enough to complete the commitments of the first iteration; the testers test it, and the demo goes smoothly.

This continues for a handful of iterations.  The team starts to "gel," gives itself a name that has little to do with the current project, holds meaningful meetings, and thrives in its collaborative team space, and even starts to handle its own personnel issues without escalation. The Product Champion (Scrum "Product Owner", XP "Onsite Customer") is engaged, enthusiastic, and encouraged by the increments of real value demonstrated in each Demo. All is sunny!

Then a subtle shift occurs. Defects are found. New functionality starts to take longer and longer to build. Developers discover that with each new story, they're having to reshape their design to accommodate new functionality without breaking old (i.e., they must "refactor"). But folks outside of engineering can't see this: Product folks may worry about the cost of this "refactoring" stuff. Testers know they have to test the new stories, but also every old story as well. But they're understaffed (as always) and start to accrue a separate backlog of stuff to test.

Eventually, real throughput (valuable new functionality with an acceptable level of quality) in each iteration comes to a dead stop.

The results of the Agilist's Dilemma

That's what I call the Agilist's Dilemma: The very fluidity that we espouse has become an impediment. "Had we only planned and designed for this! If only we knew what all the test cases would be! We should go back to a phased approach!"

Okay, calm down, everyone.  Let's not get all crazy. Remember the good old Death March? The all-night bug-fixing parties? Remember when the classic phased approach destroyed tech companies at a break-neck pace? Ah, yes, the Good Old dot-BOMB Days.

The dot-COM fiasco actually happened after the inception of Scrum and Extreme Programming, but people viewed Agile methods as strange, chaotic, and fringe.  That those methods had conceptual roots in Deming's work from the 1950's actually didn't help our case at all:  People love to find excuses to dismiss Deming.  Plus there are always high-profile examples of organizations that are very successful without adopting any particular form of process whatsoever. There are always the lucky few.

We know that Agile is about not trying to predict the unpredictable future. It's about seeing what's happening early, so these problems can be repaired while they're small. And it's about winning (creating value) within this iteration (or release), while setting ourselves up to win in the next iteration (or release). It's all supposed to provide the ability to deliver a predictable stream of value while remaining adaptable to the changing tides of business and customer demands.

And that's where our imaginary team is now:  If they (developers, testers, product, and leadership) are sensitive enough to the signs, they'll catch this early, and try something to fix it. That's exactly why Agile methods work:  We detect trouble early enough to do something about it.

The Elusive Silver Bullet

We know there are no silver bullets; no quick fixes. Yet most teams and organizations spend a lot of time covering surface symptoms with band-aid techniques, with the hope that they can just hold it all together through the next release. After that, "we'll take the time to fix everything that went wrong." Some of these techniques: hiring more testers with less experience with the product, mandating overtime for developers, and shipping on time and hoping the customers won't notice the quality issues. All of these band-aids are counter-productive, and subtly lay the blame on the people doing the work.

I think this occurs because of two fundamental problems:  Most people are unaware of the subtleties of the system within which they find themselves, and most are driven by metrics that don't relate directly to the health of the whole system. Even at the highest levels of management, often people are partially blinded by their own "performance-based" bonus check. When people are forced to compete within the organization for perceived scarce resources, the whole organization suffers.

I'm not suggesting the fault lies with management, either.  I've heard that the Lean approach is to "blame the process."  I find that blaming is like hoping:  A total waste of time. What all the Lean/Agile/Scrum literature is really suggesting is that we realistically examine and adjust the system.  Scrum gave us perhaps the most pithy and memorable phrase: "Inspect and Adapt."

So we take the time to dig a little deeper to locate those "root causes": Those things that are generating the myriad symptoms.  "Why did that happen?  What's really going on here? What can we do (or stop doing) to help resolve this?"

So we stop blaming (or discouraging) Product folks for wanting the world, or testers for finding and reporting defects, or developers for not being able to type 500 WPM. We model what's currently happening, locate where value is really getting bottlenecked, and then take a single step towards alleviating that bottleneck. Repeat.

And?

I do have some changes to suggest: Techniques that have worked for numerous teams, and work on collections of root causes that commonly plague software development.

I'm headed somewhere with this post, obviously, but I'm trying something new here (new for me):  Using a blog post like a short chapter or section, rather than having each blog post represent a complete, stand-alone article.  Maybe I'll get more written that way. ;-)

So, please "stay tuned."

Five Key Ingredients of Essential Test-Driven Development

There are any number of ways to think about TDD.  For example, I use one metaphor when describing the business value of TDD with the organization's leadership, and another when describing the personal value to the members of the development team.

There are also formulations of the actual steps performed in TDD (the most brief being "Red, Green, Clean").

This list of "Key Ingredients" that follows is yet another way to think about Test-Driven Development.  I keep this list in mind when I'm training and coaching teams, to make sure they have all the fundamental tools in place in order to successfully adopt this elegant, but challenging, discipline.


Test-First

The practice:

We write a test that expresses what we want the software to do, before coding the change to the software that implements that behavior.

This practice is one of the original dozen Extreme Programming (XP) practices. It's nothing new: For as long as we've had IC chips, chip designers have written tables of required outputs based on given inputs, before designing the circuits.  Colleagues at SQE tell me they found a view-graph (aka "slide") from the mid-80's that says "Test, Then Code."

Benefits:

Analysis through experimentation: You think about what you want the software to do, and how you would know it's right, rather than jumping directly to how the code will solve a problem.  You write, in effect, a small aspiration for the software.

Better interface design: You are crafting code from the client's perspective, and you are designing the interface first, rather than designing the solution and then exposing a clunky interface that suits your implementation.

Communication of understanding: Now and in the future, you and your teammates agree on the statement of the problem you are trying to solve.

Most defects are found immediately: You know when you're done.  You know when you got it right. And the aggregate of all prior tests keep you from breaking anything you or your teammates have ever coded since the inception of the product (if you are one of the fortunate few who started with TDD).

Easier to add behavior later: You are crafting a comprehensive safety-net of fine-grained regression tests. This allows you to make seemingly radical additions to functionality very rapidly, and with an extremely high degree of confidence.  This is perhaps the biggest benefit of TDD overall. We get productivity through maintainability, and we protect the investments we've made in robust software. You can think of each test you write as a little investment in quality. By writing it first, you are investing early.


Merciless Refactoring

The practice:

Once each new microtest and all of its siblings are passing, we look for localized opportunities to clean up the design towards better readability and maintainability.  Small steps are taken between test-runs, to be sure we are not breaking functionality while reshaping code.  Both tests and production code are subject to scrutiny.

Another of the original XP practices. Again, nothing new.  Programmers have always reshaped their modules to incorporate unanticipated new requirements.  Refactoring--any change to the code to improve maintainability without altering the explicit behavior of the code--is the formalization of a professional thought-process that has been with us since Alan Turing. 

Benefits:

Emergent Design: Your design is--as James Shore so eloquently says it--reflective, rather than predictive.  You don't try to anticipate the myriad variations you may need to support in the far future. Rather, you craft the code to meet the needs expressed in the tests today (and, again, since inception of the code base).

By removing "code smells" such as duplication, you allow the real variations expressed in your product's requirements to become encapsulated on demand. When you refactor mercilessly, what emerges is often a common Design Pattern, and not always the pattern you would have predicted, had you spent your time and mental energy trying to predict the optimal solution.

Emergent Designs tend to be simpler, more elegant, more understandable, and easier to change than those you would have otherwise created in UML before coding.

Nothing is wasted: There is no extraneous functionality to maintain and debug; there is no phase where you try to get it all right before receiving feedback from the system itself.

Ease of testing: You know that good object-oriented designs result in objects that are extremely easy to test. Often you refactor towards making the next test easier to write or pass. The test-refactor cycle thus supports and improves itself: It's easier to refactor with the presence of the tests, and the next test becomes much easier to write, because it's now obvious to you which class should own the behavior.


Automation

The practice:

We write the tests using a tool that makes them immediately executable.

This one may seem obvious, but its explicit definition offsets and clarifies the others. For example, can you do test-first without automating the tests? Yes. And is there benefit? Yes.  Is it TDD? Not quite.

Programmatic microtesting has also been with us for a very long time.  In the C programming language, we used to create (and later discard) a separate little main() program with some printf() statements. Now we capture these bits of testing, and effortlessly re-execute them for as long as necessary. Since they remain separated from production code, we need not wrap them in #ifdef TESTING statements.

Benefits:

Repeatability:  You know that each time you run the tests, all tests are the same, and not open to human (mis-)interpretation.  Testers, even very talented and insightful testers, can make mistakes or succumb to biases.

Faster feedback: Computers are good at doing these dull, repetitive tasks much more quickly than humans.  Subjecting testers to repeatedly running pages of manual test suites is cruel. You help testers recoup time to spend on exploratory testing and other testing activities that provide higher returns.

One life-critical ("if you break it, a patient may die") medical product I worked on had 17,000 automated tests after two years in development.  They all ran in less than 15 minutes.  We could add functionality, fix a bug, or refactor a bit of awkward code right up to the hour of deployment, and still know we hadn't broken anything. And if we did see a single failing test, we would not commit our changes to the repository. To do so would have been unprofessional and unethical.


Mock Objects

The practice:

When creating microtests, we often test an object's response to external situations at integration boundaries (e.g., "What does my DAO do on Save() if the database is down?").

We do this by replacing the interface or class that represents the external subsystem (e.g., it makes the calls to JDBC or ADO.Net) with a "Mock" or "Fake" object, which has been instructed (aka "conditioned") by the microtest to operate in a way that suites the scenario being tested (e.g., the mock simply throws the appropriate "Database is down" exception).  When the microtest calls our object under test, the object under test interacts with the mock, just as it would in production.

Benefits:

Predictability: You use mocks to decouple the code you are writing from databases, filesystems, networks, third-party libraries, the system clock, or anything else that could alter the clear scenario you are trying to test. The test will pass or fail depending only on what your production code does within the scenario arranged by the test.

Faster tests: By decoupling from anything slower than the CPU and memory access (e.g., hard-drives, network cards, and other physical devices), you and your teammates create a regression suite that can run thousands of tests per minute.

You may have a subset of tests that actually interact with these external systems.  Think of these as "focused integration tests" (another term I think I got from James Shore).  For example, on many systems I've worked on, we've wanted to test that we could actually read from and write to database tables (prior to Hibernate, Rails, etc.).  I recall those 17,000 tests included 2% to 5% functional and focused integration tests, which took perhaps 50% of the 15 minutes.  That's okay.

Nowadays, the database itself can often run in memory.  Or, you could develop on a solid-state laptop, such as the MacBook Air or Asus Zenbook. Different paths to the same goal.

The rule of thumb is to keep the comprehensive suite from taking much longer than a "coffee break".  A "lunch break" (1/2 or 1 hour) is far too long, because then the full suite would only be executed intentionally a few times per day.  On a small Agile team, we need to see the full suite run green a dozen or more times per day.

It's this fast feedback loop that enables "Agile" teams to stay agile after months of development.


To-Do List

The Practice:

We keep a list of things to test during this TDD session immediately available, so we can add test scenarios that occur to us during the session.  We keep this particular To-Do List around only until we complete our engineering task and commit our changes.

When performing the TDD steps (briefly described as Red, Green, Clean), often we see an opportunity to test a new corner case, or refactor some bit of stale design.  These to-do items go immediately onto the To-Do List, without interrupting the flow of Red-Green-Clean.

On teams I've coached, one responsibility of the pair-programming "Navigator" is to maintain this list on a 3x5 card.  We often started with the back of the task card (Note: not the story card, because our stories were subdivided into multiple engineering tasks).

You don't have to use a 3x5 card. Some IDEs allow you to enter "TO DO" comments into the code, which then appear automatically in a "Tasks" tab.  This is especially helpful if you're not pair-programming (you unfortunate soul).

Please, just never ever ever ever ever commit a "TO DO" comment into your repository. Checking in a "TO DO" comment is effectively saying, "Well, someone will do it, but we don't have time right now." The "Tasks" tab will fill with un-prioritized to-do items that "someone" (no one) is going to do "someday" (never), and the tab will become unusable as a To-Do List.

Benefits:

Idea repository: As you write tests, implement code, and refactor, you will notice other things you want to test or refactor. Some may be rather elaborate, but all you need is a brief reminder. Once that's recorded, you can allow your creative energy to return to the test or refactoring at hand.

Memory aid: You can take a break and, when you return, you have a short check-list of items you need to complete before you're done with the task.

Pair-Programming aid: The list gives the Navigator and Driver points to discuss, decide upon, or defer, and keeps them focused on real issues rather than long philosophical debates ("Opinion-Driven Development").

Jim Shore and I refined the practice one day, after the following exchange (from memory, plus some much-needed embellishment, after 10 years):

Jim (navigating): [Writes something on the To-Do List]

Rob (driving): [I stop what I'm doing] "Whatcha adding?"

Jim: "We're going to want to test what happens when that Connection parameter is null."

Rob: "Oh? How could it end up null?"

Jim: "Well, we're getting it from StupidVendorConnectionFactory, and so..."

[long discussion ensues]

Rob: [frowning] "I forgot where we were..."

Jim: [Sighing heavily] "Rob, Rob, Rob...Just re-run the tests!"

The next time Jim thought of a missing test...

Jim (navigating): [Writes something on the To-Do List]

Rob (driving): [I stop what I'm doing] "What's that?"

Jim (navigating): [Hides the To-Do List] "I won't say. As the driver, don't worry about what I'm adding to the list.  Let's focus on the test we need to get passing right now, then we'll review the list together."

Rob: "Can I add stuff to the list?"

Jim: "Of course! Just say it as you think of it, and I'll write it down."

Rob: "Okay.  Let's add 'fix, wrap, or replace StupidVendorConnectionFactory'"

Each item on the To-Do List must either get done, or be "removed."  We designated this by drawing a little checkbox before each item. We would check those we completed, and X out those we decided were not worth doing, or that had been replaced or subsumed by another to-do.

Jim: "I feel better, now that we have ConnectionPool wrapping StupidVendorConnectionFactory."

Rob: "And since ConnectionPool has been tested, and has no paths that could possibly return null..."

Jim: "...we don't need to check for null every time we pass a Connection. I'll take it off the list." [puts an X in the box]

Rob: "Are we starting to finish each other's sentences?"

Jim:  "Creepy.  Let's talk about hiring some developers locally so I don't have to pair with you every day."

Essential Test-Driven Development: Just Say No to Unit-Testing

Wait...Is He Kidding?

A while back I was working with a large client to plan their Agile training curriculum for teams across the country. They wanted to have ready a few courses on Agile development practices, and I recommended my Essential TDD course as a starting-point.  The head of the program said, "Let's not train them on anything that advanced.  Let's start with a unit-testing course."

I answered, "I find it easier to train and coach people on good unit-testing practices through the practice of Test-Driven Development. TDD makes unit-testing easier: Easier to do right, and easier to do in time to make a difference."

Unit-testing, as a stand-alone practice, is frustrating and dull.  The fun work of designing the solution and writing the code is over, and now we must prove that we did it right.  Usually, this means we have to re-read our own code, remember what it's supposed to do, and retrofit unit-tests onto it. The tests we write after the fact are often difficult to write. Often, they give us bad news (that would have been more helpful sooner). Plus our own fallible human brains will often fall back on justifications for why any failing test is wrong rather than an indicator of a mistake (i.e., Confirmation Bias).

Many developers find Test-Driven Development to be easier, more enjoyable, more effective, and more logical.  For me, it was like discovering that I'd been walking on my hands since graduating from college: Yeah, after 10 years I was pretty good at painstakingly writing simple, quality code without unit-tests, but once I got used to the "weirdness" of TDD, I saw how my old ways had limited my creativity, productivity, and understanding. I was back on my feet.

Just Like Science, Only Totally Different

Software Developers are natural problem-solvers.  This can be to their detriment. When we write software "on our hands" we mentally note the problem, then select the most interesting solution of the half-dozen that pop instantly into our heads. "That smells like a Composite Pattern.  Yeah!  I'll implement a Composite here." Yep, I used to design up-front.  Only it turned out not to be a Composite, and I wrote too much prematurely-generalized code, giving my product capabilities that no one asked for, yet someone paid for.

I also had extra code to maintain. Every time I added new behavior to "my" code, I had to carefully ponder the effects of my changes on the myriad intricacies of my existing beautiful design.  The more the software grew, the more the complexity problems grew, and the greater chances I would introduce a defect.  Walking on one's hands is a cool trick, sure. It's just not the most professional strategy.

Fortunately, in 1998, XP Coach extraordinaire Fred George came to visit our team, and turned my world upside-down.

In college (1984) I started out as a physics major, and thrived in all environments where scientific experimentation and discovery were the norm. Creating a hypothesis and designing experiments is somewhat akin to writing a unit-test first: State the problem first, then use that to prove the solution. Then refine the problem/solution pair through Merciless Refactoring. When I realized this, I was hooked.  TDD feels more like scientific investigation, and less like hacking.

Science isn't the perfect metaphor:  After all, you can't make a false hypothesis true by tweaking the Universe.  But you can make a test pass by making small changes to the code (Bob Martin calls them Transformations).

And It Works!

The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15–35% increase in initial development time after adopting TDD. 

-- Nagappan et al, © Springer Science + Business Media, LLC 2008

I used to deliver TDD training at Microsoft for Net Objectives, and I recall talking to two attendees from different teams who, while working on the course labs together, had discovered through conversation that they were assigned to very similar products.  Microsoft is one of the only organizations large enough to do a good study of similar-sized projects, with teams who have the freedom to choose aspects of their methodology.  Up until this study, most of our evidence of TDD's efficacy was either academic or anecdotal.

A few things I'd like to point out from the IBM/Microsoft study quoted above:

First, 40-90% decrease in defect density? Um...wow! (Need I say more?)

Secondly, note that there is a cost associated with adopting the TDD discipline.  This should not come as a surprise: TANSTAAFL, y'know.  Worst case, you take a 35% hit in development time, and reduce defect density by 40%.  Remember what defects mean to us:  Time spent debugging, reworking, and re-testing.  Defects, a form of technical debt, are expensive!  I'll take the subjective 35% initial hit to avoid incurring that debt.

Thirdly, I'd like to focus on that one word: Initial. Any discipline requires time in order to develop a level of comfort, and to see early benefits. Disciplines are, in fact, painful to adopt, and TDD is no exception.  "...Twice as much code...!" developers cry. "No!" I tell them, "Closer to three times the code!" Also, on any program (green-field or existing), you will wonder "Where do I start?" It takes time before the ubiquitous, domain-specific language starts to solidify, and writing the next test becomes effortless.

Fortunately, it really doesn't take that long to see the benefits.  One developer on a team that I trained this summer reported back to me after about a month: "I thought it was stupid.  I'd write the simplest, stupidest little test--yes, first!--even though I knew the implementation was code I could write in my sleep. I had to focus on not falling back on old habits, even though I would occasionally curse your name.  Then, barely two weeks later, one of those stupid-simple tests was failing, in an area of the code that (I thought) I wasn't even touching! That one stupid test saved my bacon.  It was a simple mistake, and if I had checked it in, I could have messed up some very important customers..." His company's customers are legal firms. Like the conference T-shirt says, "TDD Saves!"

I've experienced a number of first-person events where the safety-net of comprehensive unit-tests plus the flexibility and maintainability provided through Merciless Refactoring resulted in astounding payoffs.  In no less than three cases (on two disparate programs), a simple "user story" contained what most teams would call "major architectural changes."  In each case, we were done in less than a week.  I had never experienced these extremely short cycle-times until I worked with teams who embraced TDD.  (I will share these first-person stories in a future blog post.)

Of course, these benefits could perhaps be obtained by simply having a comprehensive suite of unit-tests, so how is TDD better than unit-testing?

Nutrients-First

When I was a kid, I hated eating my vegetables.  My Mom would insist that I couldn't leave the table, or have dessert (if there was any) until I had consumed all those overcooked nuggets of yuck.  I finally learned a trick (no, not feeding them to the dog when Mom wasn't looking - that was my siblings' trick! ;-). I discovered two very important things about vegetables:  They taste better when they're hot, and they taste a lot better when you're hungry.  The trick:  Eat your veggies first!

TDD is like that:  Write the test first. It's healthier.

We (1) describe the expected interfaces and outcomes, (2) confirm that we've asked for something new and unique, then (3) make the code meet the challenge.  The next step, not to be skipped: (4) we clean up the design through refactoring, in preparation for the next tiny step towards the delivery of value.

Once developers grasp why we're writing a single unit-test and watching it fail before we write the code that makes it pass, they eventually cannot resist this mode of thinking. When you write code Test-First, you cannot write untestable code. When you write Test-First, you cannot miss a test; you cannot forget to cover a behavior with tests.  When the test does pass, you're likely done with that test for the rest of time, but it serves as a tiny bit of "executable specification," and it serves as a tiny investment in the future.

The alternate route, adding unit-tests to already-written code, often invokes our "Confirmation Bias":  I will have a nearly subconscious tendency to trust the code more than the tests, and I may tweak a test to match what the code is already doing.


Backups are Free. Restores? Now Those Will Cost You!

I recall just such a confirmation-bias disaster.   In the mid-90's, our Enterprise Backup software's UNIX ports had a chunk of code based on "tar," the UNIX tape-archival command.  On restore, tar would mask out the super-user execution bit. I recall seeing this code (hadn't written it...the architect had lifted it from tar), and assumed that a smart, security-minded thing to do in tar was likely a smart thing for our product, too. Except that our product ran as root anyway, and was expected to restore a whole system to a bootable state.

Oops!  The architect, executive developer, and UNIX developer (me), had all unknowingly conspired to ruin a customer's day ("further ruin" since the customer was trying to restore the root volume on a large server...).  All for want of a clear test scenario prior to the development of this code. All the test scripts we had written around this functionality assumed that the code (and the architect) was right.

How human of us, to assert only that we couldn't have possibly made a mistake. That would be like designing a flawed experiment to prove a pet theory.  You're hurting yourself. You're jeopardizing your career. You're walking hot pavement on your hands. Stop that!

Go Home Already!

Yesterday morning I decided I needed a break. Not that life had been excessively stressful recently (quite the contrary!), but sometimes you just need to step back, take stock of your surroundings, and move forward again based on a broader perspective. So this wasn't the kind of break where I set work aside, completely, but rather a reassessment of circumstances, and of the choices that are before me.

This reassessment was refreshing, and left me with some energy to write about more of these Agile practices that foster creativity and sustainability.

Slack

Definitions of "slack," and related practices, abound. To me, "slack" means not allocating 100% of the team's time. The team commits to a realistic amount of work, rather than falling for the "planning fallacy" that everything will go according to a rigorous plan.

Teams who don't plan out 100% of their iteration tend to provide more value. This may seem paradoxical, until you consider human nature: If people go into a sprint (a.k.a. iteration) sensing that they are committed to 100% capacity, they start the sprint...sprinting! They are stressed from day 1, and will lose enthusiasm for the work long before the end of the iteration.

Related Practices

Velocity: Team velocity, measured by "yesterday's weather," can help. If the team tends to do 12 points in a sprint, we schedule 12 points. Velocity--when it's not constantly debated, gamed, or artificially increased--already takes many unforeseen difficulties into account.

Leave Room on the Plate: I've seen teams who set "stretch goals" or add "low priority" stories to the trailing end of the iteration plan. If trust is strong, this can work, but it could lead to overcommitment and disappointment. Instead, I suggest allowing the team to pull small, high-priority stories in at the end of the sprint, if there's time left.

Of course, all other stories must first be "done done." Is there any further testing needed? User documentation to alter? If so, there is still work to be done, and the team should not pull in more work yet.

You know the phrase "too much on my plate"? Be sure you're not asking the team to "super-size" their iteration plan.

Creative Time: Some teams set aside actual "slack time." During this time, the team may:

• Explore a new technology that would help improve flow later.
• Refactor that old area of code that bothers or worries us. Perhaps it doesn't need to change very often (else we'd refactor prior to expanding its responsibilities), but we just have some energy and desire to clean it up now.
• Brainstorm on some upcoming technical challenge.
• Run a "spike" or experiment.

I've worked with two or three teams who adopted this practice, and they used guidelines like these:

• Thursday afternoon was set aside for "play."
• Each person could use the time to explore some area that was of interest, and that had potentially positive effects on career skills or the team's well-being.
  
• The team was asked not to work on stories from the iteration. If they had, this would throw off the velocity and thus erode this practice over time.
  
• Examples of acceptable activities: Spikes (timeboxed experiments). Refactoring away some technical debt. Trying out a new development or testing tool. Holding an extended brown-bag to talk about Design Patterns. Anything that didn't break the previous guidelines.
• We asked the team to describe how they had used their slack time at the next stand-up meeting, and "it wasn't very productive" was an acceptable answer, with no stigma attached.
• Full-time employees could even simply play video games or surf the web if (a) they sincerely needed the down-time and (b) they gave a report at the Friday stand-up (and "I played Dragon Age and got my mage to level 17" was perfectly acceptable).

External coaches weren't to play games (obviously!), but we derived plenty of joy and relaxation from refactoring the snot out of some bit of nasty code. Once, frustrated with some quirks of our old Java IDE (note: It was neither Eclipse nor IDEA), we evaluated IDEs and recommended (either Eclipse or IDEA) the next day. The team agreed to use the new IDE in the next iteration.

I suggest Thursday afternoons rather than Friday afternoons, because on Friday afternoons people will naturally just escape after lunch for the weekend, and little team value will be obtained.

Sustainable Pace and Energized Work

Different people have different amounts of daily sustainable energy for doing certain parts of their jobs. It often appears that younger developers have more energy to work long hours on a problem. I've since realized that what most knowledge-workers of any age tend to do is to push themselves beyond their own rational limits, into an "irrational loop." They're too wrapped up in a problem to realize that they're tired and hungry, so they keep pushing on that problem. In the irrational loop, they become more and more frustrated, and are prone to make simple mistakes that cause the solution to elude them further, resulting in more time spent looking for a solution.

I sometimes quip that most bugs are written in the last two hours of a 12-hour day. It's probably not accurate as stated, but it gets the point across: Work hard while you have the energy, then stop.

I cannot count the number of times over my 25 years when I've had to be the one to say "Y'know, this problem will wait for us until tomorrow morning." In fact, it's now one of the most fulfilling parts of my job as coach to say "You're stuck? Roll back to the last green bar [i.e., all tests passing] in your local repository, go get some food, some rest, and some time with your families. Go home!" Nearly always, the next day someone walks in having realized what the roadblock was, and with a good solution rattling around in her skull.

There's another benefit: People will sometimes avoid going home because there's even worse stress waiting for them there. But those troubles won't fix themselves, while we hide at the office answering megabytes of e-mail. Letting people have the time to spend with families, on hobbies, and on a full night's sleep pays off almost immediately: Fewer sick days, fewer chronic family issues, higher retention, and alert, relaxed teammates!

In my own experience, I know I can teach my courses with four hours of sleep and a double-shot espresso, if I have to. But I'm effectively on autopilot, playing back the recorded messages in my brain when asked a familiar question. Instead, I prefer a good dose of actual sleep, and no caffeine: I'm more focused and more in tune with the needs of the attendees.

Related Practices

Go Home: Once you know approximately where your limits are, set a time when you plan to go home, and stick to it. Let colleagues know when you're starting to run down, and that you've got about an hour left of productivity in the office. Schedule time for yourself at the end of the day: A yoga class, dinner with the spouse, or just a "go home" reminder on your calendar. When asked if you can attend that 6:30pm meeting, say "No."

One Failing Test: For TDD or XP teams, this is a favorite for lunch breaks or at the end of the day. Leave your (small amount of) uncommitted changes on your dev machine with a single failing unit test to mark where you were. It will take no time at all to figure out where you left off. Much less context switching! Of course, "all-green and committed" is an alternative, but there's nothing quite as concrete as a single (uncommitted!) failing test to get you back into the Zone.

We appreciated this practice so much, we'd start a new task and write the first failing test with only a few minutes left in the day.

Continuous Learning

On a day-to-day basis, people are energized by intrinsic motivators such as interesting, valuable work; pride in quality work; and interesting learning experiences. This does not have to be limited to a yearly trek to a trade conference. Those are great, but if they're the team's only source of fresh mental stimulation, all those spongy brains are going to be surviving on minimal subsistence the other 51 weeks of the year.

Related Practices

Sharpen the Knives: You've heard the old adage about the lumberjack who--in order to cut a tree in an hour--will take 45 minutes to sharpen the saw?

This old analogy really needs updating: Not many of us are all that familiar with the logging industry. Right now, reality cooking shows (Top Chef, Iron Chef, ...) are hot, and I've just learned the right way to sharpen cooking knives (one way for me to contribute in the kitchen, besides cleaning it).

So, why do chefs sharpen their knives? Because they can go faster, work more accurately, and create higher quality. We wouldn't tell a chef to "just keep cooking! Go, go, go! Never mind quality or technique or safety..." Would we?

Most teams need to "slow down to speed up" using "Slack," and this practice takes slack one step further: We improve throughput of value by allowing the team the time and authority to learn an effective new skill.

Pair Programming: Two people collaborate on critical efforts, so teammates constantly learn from each other. This learning is not the result of a teacher-student or master-apprentice relationship: Peers can share a variety of things, including efficient tools and techniques, domain knowledge, product knowledge, and appreciation for the skills and responsibilities of other teammates.

Summary

"Measure twice, cut once."

"Sharpen the saw."

"Slow down to speed up."

"It's a marathon, not a sprint."

In all endeavors, at all of life's stages, the best action to take is often counter-intuitive. When you feel that old chaotic wind spinning around you, step back, get more of the big picture, and try out one of these practices, or one of your own invention. They are all essentially the same: You are taking the time necessary to complete your task with a high degree of quality, integrity, and craftsmanship; and without ruining your health.

Okay, I'm outta here...!

Gentle Discipline: Making Agile Happen in the New Year

Update #1 for 2013:  When this was first posted, I had a few people point out that I was talking about self-discipline.  True enough!  I'm not sure I can conceive of any other form of successful discipline.  It's possible that I'm biased because I'm an Agile Coach; or--more likely--that I've become an Agile Coach because of this bias.  The "User-Friendly Discipline" is my name for an observed phenomenon, not for something I thought up. It is the only form of real discipline I've ever known to work, in my own life as well as the lives of those I've encountered, over nearly five decades, and on three continents.  The assumption that you can force other people to do something as a discipline is, to me, disproven by my observations of successful disciplines (as described below). YMMV, but I doubt it. -- Rob

It's 03 January 2011, and I'm going to wait one more day before going to the gym. If you've ever had a gym membership, you know exactly why: Right now, every gym is a zoo of humanity. Right now, hundreds of new people join the gym and try so very hard to live up to their New Year's Resolutions.

But, as the regulars and the staff are aware, most of those newcomers eventually give up and stop coming. Though I'll be happy to have things back to normal, I always feel a little sad for the people who weren't able to establish an ongoing discipline.

The D Word
In my courses, I often ask people what comes to mind when they hear the word "discipline." Images arise of strict teachers with rulers, or boot camp drill sergeants. The word seems to imply rigorous, strenuous rules, extra effort, and--above all else--suffering.

Many people stop going to the gym because they start out too intense. Often their fit (and younger) personal trainer has constructed a regimen suitable for a Marine; or they just push too hard the first day, and still can't lift their arms above their head a week later. Or they tried to dedicate all their free time to this one endeavor, and life (laundry, house-cleaning, overtime, friends) intrudes; or at least provides an excuse.

Discipline: The User-Friendly Definition

The way to establish a consistent, painless discipline is embedded in my formulation of the "acceptance criteria" for a discipline.

1. A discipline is a habit...
2. ...that you come to rely on, even during difficult times...
3. ...because you know that it benefits you, personally.

Let's examine each of these further, and I'll make some suggestions on how to implement this.

A Habit...
A habit is something we do almost unconsciously, and frequently. There are good habits as well as bad. Think about the numerous habitual disciplines you've already established in your daily routine before going to work. Most people shower, put on clothes, and brush their teeth. If you think back, you had to be taught all of these, and some of us probably put up a fuss over these disciplines as children. Do they still seem painful? Would you give them up?

I recall learning (perhaps many decades ago) that to establish a habit, psychologists suggested we repeat the behavior for about 30 days. Whether this rule of thumb is still canon or not, I've found that we must remind ourselves, and perform the activity, repeatedly until we know we can discard the reminder.

A sticky-note on your mirror could remind you to floss. A daily alarm on your calendar software could remind you of the stand-up meeting. A big, visible chart graphing the number of unit-tests checked in to the repository could encourage the team to write tests (first! ;-).

...We Rely On...

For a practice to become a stable discipline, enough people on the team (it could be a team of one) have to be doing it habitually so that it will not die out due to some minor random event (e.g., when two new developers are added, or while three people are out with the flu).

A Tale of Two Teams

I've observed at least two Extreme Programming (XP) teams as they discovered they needed to rush out a release. One organization decided at the last moment to show the product at a trade show. On another team, there was a customer anxious for an early Beta release, and willing to pay.

The reaction of these teams suggested their levels of discipline with various practices. The less-experienced team gave up TDD, pair-programming, refactoring, code-reviews, object-oriented goodness, all manner of sanity; and just cranked something out. The product failed to "wow" anyone, to say the least.

The other team acknowledged the crunch, took a little time to re-prioritize and break down the backlog, then sat down in pairs and practiced their testing and engineering disciplines as though nothing untoward had happened. Their release may have been short on bells and whistles, but it was also very short on defects. (And they sold many licenses and the VCs lived happily ever after, The End. There be Agile Dragons, remember?)

A key lesson from these events was that a discipline isn't ever something we think is "a great idea, in an ideal world, but we're too busy right now in the real world, so could you bug us about it later?" You have to know its true value, in your bones.

Frequency Over Intensity

Lather, Rinse, Repeat. Always Repeat.
-- Homer J. Simpson

To get there, it's important that you repeat the activity rather consistently before a crisis occurs, and through some minor daily crises. Even when, especially when, it's painful to do so.

Here's one of the user-friendly parts: Early on, frequency is more important than intensity. You can tone down the practice until it becomes a habit.

An example for the resolute exerciser: If your muscles are sore, just go do your cardio. (If your regimen is purely cardio, then go play lightly on the weight equipment.) Just go have fun with it.

An example for the blossoming meditator: No time to practice? Sit down and breathe slowly and deeply three times, then get up and go to work. (And try not to curse at traffic lights, or other drivers. ;-)

For the Agile team participant: Try a new practice wholeheartedly for a month (only one to four iterations). You know you can ask the team to reexamine and tweak the practice at every retrospective; so don't evade, grouse over, or sabotage the team's efforts. Do it as though you believed in it, and that your project or career depended on doing it well.  (It does.)

My friend Bob Hartman once helped establish a team's pair-programming discipline by asking the team to pair up whenever they were fixing a bug. "If one person broke it, why do you think one person can fix it?"

...Because It Benefits Us
We won't even attempt a new discipline without some indicator that it's valuable. We observe people who are healthy and happy, and they exercise in some way (among other things). We observe teams who produce high-quality software, and they're using fast, automated testing (among other things). We want that!

At the team or organizational level, we want fewer defects, faster time-to-market, more profit!

Weigh Costs, Too

If the individual, team, or organization picks up a new practice, the benefits are probably well-documented. What may not be clear from the literature are the costs. Both have to be considered in order to select the right set of practices for the organization (or organism).

Unfortunately, the costs--or aches and pains--are always more intense when you first start out. So, there's frequently early resistance ("emotional re-evaluation"), which could stifle the burgeoning discipline.

It's Personal

The biggest hurdle hiding within the acceptance criteria may be even more daunting: In order for it to be a discipline, each participant must have experienced (past tense) at least some small personal benefit from the discipline.

Effective disciplines are not followed out of altruism, nor are they followed because people are getting paid to do it the "right" way. Even Mother Theresa admitted to personal benefit and pleasure in helping others. And I've worked with many large organizations who were paying good salaries to people who didn't really do much: They were mostly trying to stay hidden, and waiting for their pension to kick in.

Even those employees are not very happy in their careers. People need more than a big paycheck to feel happy. What's sad is that many of them don't know this. They think "that's the way things are, and there's no way to change the system."

Every good discipline has something in it for you, the practitioner! The benefits of my sample disciplines (exercise, flossing, meditation) are clearly personal, and entirely obvious, right? ;-)

How about Agile disciplines? "What's in it for me?" Here's just a sampling, regarding some of the most peculiar Agile practices ever devised:

Paired programmers report that they are more confident in their solutions. Their defect-density drops, and they spend far less time in the debugger. They feel that they--even the old-timers--are learning from their peers at an exhilarating rate. They are more comfortable around teammates, and more confident in their discussions with management. They have fewer carpal-tunnel-related complaints. Ultimately, the feel more productive.

Disciplined TDD developers report an uncanny level of confidence in the quality, design, and adaptability of their code. At first the costs seem prohibitive: Extra test code needs to be written; and since the design emerges over time, time is spent continuously refactoring (which is the design mechanism in TDD). But after only a few months, they are able to add new behavior into the system in days, rather than weeks; and they use tests to seek out, isolate, and crush defective scenarios (bugs). They report that they recoup nearly all the time they would otherwise spend in the debugger. (I've had developers tell me that they used to spend half their time debugging. What would you do to recoup almost 4 hours of each day?)

Once people get a taste of these benefits, they have a much easier time continuing the practice, and improving upon the required skills. For them, it has become a real discipline.

Additional Suggestions
Stop Lamenting the Past

If your blueprint for success contains (or implies) "Step 1: Build Time Machine" then you need to rethink your plan.

In order to establish a discipline, you must stop dwelling in the past. You're out of shape? You can't leap into strenuous exercise. You have a million lines of untested code? The optimal path is not to take a year off to cover all that code with characterization tests. Hundreds of severity-one defects? Funding a complete rewrite isn't going to help.

There isn't a single discipline on Earth that involves wishing things were otherwise, or fixing everything that's broken.

Stop Blaming

Never punish or berate yourself or your team for the occasional relapse into those old 2010 days.

Acknowledge the lapse, yes. Perform root-cause analysis and see if you need to jiggle your circumstances to prevent a future relapse.

Let me ease your concerns here and now by assuring you that lapses will happen. So, now that you already know, you won't have to argue with yourself about perfection and responsibility. With each lapse, you simply have to re-commit to practicing now and in the future. (And, yes, you will have to pay your personal trainer for the missed session.)

But you can't try to make up for it. One of the worst mistakes I see in the gym is working out twice as long after missing a session, or after eating too many holiday treats. That path leads to further discouraging circumstances. Sure, if there's a minor mistake that can be corrected (e.g., by adding unit-tests around new, untested code), then do it. That's correcting a mistake. "Correcting" a relapse involves extra time practicing the discipline, and that invariably eats into the time needed by some other practice. Tomorrow is another day.

Does this conflict with my suggestion to focus on frequency over intensity? No, because I was not suggesting that if you go a day without (flossing, meditating, a daily stand-up meeting) that you have to start the 30-day clock all over again in order to build the habit. When you fall down, get up, dust yourself off, and get back on the bike.

Stop the Excuses

Don't over-plan your efforts towards a new discipline. A complex plan is full of opportunities for reversal. What we're really doing when we over-think the practice is creating a mental model of our lives where the practice is ultimately not worth the effort.

That is, we invent our own cosmic excuses. This takes a certain level of cleverness. If we are to try something that doesn't come with a guarantee of instant gratification, this cleverness must be tempered with courage.

You're intelligent. You're courageous. What are you planning to try this year? Add a comment and share your experiences.

Update #2 for 2013: I've noticed another impediment to creating a healthy new discipline:  Embarrassment, or peer pressure. "Why must you _____?!" people will ask (or imply), incredulously. I have a number of very old habits that a few people have (openly and vocally) suggested were professionally lazy: Exercising three times per week, meditating for 1/2 hour per day, and getting eight hours of sleep on most nights.  I don't tell you about these now to be self-aggrandizing (or self-effacing) but as factual examples:  For me, they are so routine that when I have to drop one, I become cranky, and much less effective at work. They are all wonderful, productive, professionally supportive, healthy habits. But disciplines that are counter to prevailing culture will meet with resistance.  If you can use your own facts and experiences, you will be able to politely and clearly convey the benefits to those who are skeptical. That's how culture is slowly changed. Have courage!

Have a happy and prosperous New Year!

I Have Written Bug-Free Code Without TDD

There's an excellent blog post by Felix Geisendörfer which gives his experience report on using TDD. Kent Beck tweeted about it, so it's hopefully "going viral" right now. Read it here.

Reading his post, and some of the comments afterwards (when will I learn?!), got me to thinking about my own experiences with TDD, and I decided to share two of my own stories.

Once Upon a Time

I used to write bug-free software without TDD! (Look at me: I am so smart...)

My boss and I would play this game called "Who Wrote the Bug?" in which he would claim that a bug was in my code, and I would come back (after painstakingly debugging the whole product) and point out where it was in his code. (Hmmm...who was smarter?)

Humility Sets In

The software I was writing was a protocol stack that needed to convert between 8-bit and 9-bit bytes, or 16-bit and 36-bit words. I used the command-line/printf approach (System.out.println for the Java readers; Console.WriteLine for C# readers) to test stuff. The debugger, COFF, was okay, but testing through a debugger is such a pain.

If you haven't figured it out from the clues I've dropped, this was over 20 years ago. In retrospect, this was pretty tame stuff. I could keep track of what was going on in my head.

One can assume that software projects these days are more complex. Perhaps too complex for one person to keep track of all the little moving parts. It's enough for the team to have a grasp of the overall concepts, goals, and metaphors.

I would argue that even on a team with only three or four developers, eventually they're going to bump into each other's changes. They need a suite of microtests in order to maintain quality code.

The test frameworks like JUnit, NUnit, and RSpec essentially give you a way to let the computer check the "printf" output (figuratively speaking) for you. You record your assumptions, rather than manually checking them each time, and you can let those assumptions grow upon each other. You record them separately, rather than interspersing the following infamous abomination:

#ifdef DEBUG
printf("foo=%s", foo);
#endif

Two Years of Investment

In 2002 I worked on a team of four J2EE developers on a life-critical (i.e., "You break it, someone may die") application. After two years, we had built some pretty sophisticated reports and heuristics that helped hospitals determine who was most likely to survive an organ transplant.

We had over 10,000 tests. They all ran in about 12 minutes.

Imagine: All aspects of a complex, life-critical, team-built application could be thoroughly checked in 12 minutes. We could (we did) bring in someone new, and essentially say "today we will make whatever changes we need, even mere hours before the next release, as long as we run all the tests and see a 100% pass-rate!"

Twelve minutes, and we knew we hadn't broken any of our careful, 8-(plus-)person-year investment.

Rise to the Occasion

If you're sitting alone in your basement writing a game to sell to a VC, you may not care if it's maintainable. By all means, please continue using only manual testing techniques to verify your app. I'm not being snarky: This may be the optimal win-win approach for you and the VC. (The team that eventually maintains your app will curse your name and throw darts at your picture, but do you care?)

On the other hand, if you're part of a team, writing mission-critical software for a large or complex domain, and it needs to get beyond version 1.0; then please: Make sure you can always test everything, and quickly. TDD is the best--the only--way the industry has invented so far for doing this.

Anything else would be unprofessional.