There Be Agile Dragons!

At recent conferences such as Agile 2009 and SQE's Agile Development Practices conferences in Vegas and Orlando, I've been running into more and more actual decision-makers. This is exciting, because it's one more indicator that Agile is "crossing the chasm" into regular, large-scale corporate use.

These decision-makers usually want to know, "Will this Agile stuff really help, or is it just another process fad?"

I always converse enthusiastically with people interested in "going Agile," but I no longer limit myself to "Agile" methods. Often, I'll avoid even using the term (though I have no plans to change the name of my company just yet...).

I tell them I'm interested in helping teams examine and alter their development processes by identifying bottlenecks, looking for root-causes, finding ways to ease those constraints, and delivering valuable software.

XP practices, Agile management, Lean leadership, and an overall "Theory of Constraints" approach; all these should be in the software-dependent organization's toolbox. What we call "Agile" is a subset that applies to a variety of constraints, but not always to the limiting constraint.

Dramatis Personae
Over the decades, I have picked up techniques to uncover people's key concerns. I can talk until I lose my voice; describe the latest findings, and pause to listen as they come to their own conclusions.

Decision-makers usually have a healthy dose of skepticism towards new (or seemingly new) techniques. They pick things up somewhere between the early adopters and the late adopters. It may even be a corporate survival trait. Skeptics are always fun to talk to.

But then there are the naysayers, who seem to believe their team cannot do anything right without constant oversight and carrot-and-stick incentives. Naysayers will listen, but then they simply reiterate their position, and will try to keep you in the conversation until you concede (whether or not you realized you were in a debate).

I think I know why: These are leaders burned by some past failure, and are avoiding failure by avoiding change. They will try to build a mental model of some new thing (Agile), and then run virtual scenarios in their minds, rather than committing themselves and others to real risk. Unfortunately for them, their mental model cannot be completed through mere conversation.

To them I now say: "Meet The Dragon."

The Boy and the Dragon

There's a story (very old, but I can't find a handy reference, so I'll bring it up-to-date) about a boy who loved dragons. He drew pictures of dragons, wrote poems and stories of dragons, and read books about dragons. He even dressed as a dragon for Hallowe'en, and his Twitter-name had "dragon" in it (there, now the story is updated).

One day a great, wise dragon heard about the boy, and thought that she would delight the boy by visiting him at his home. "Imagine," she thought, "how excited he will be to finally meet a real dragon!"

She appeared one night in his room and began to speak...

The boy saw the dragon in his room and was so terrified, he could neither move nor reply. His eyes were wide, his heart pounded, and he was in such a state of shock that he could not even scream. The terror was so great that he died of fright right there and then!

(Okay, okay, the dragon gives him CPR and he lives. In fact, the dragon, let's call her "Puff,"and the boy--I think his name was "Jackie Paper"--become fast friends. Better? Sheesh! Remember the good old days when children's stories used to be truly scary? ;-)

DM: Dear @AgileCoach, Do Dragons Really Exist?

Yep. There are many successful Agile teams around, actually doing Agile stuff because they want to; because they produce high-quality, high-value software quickly; because they enjoy working in an environment that encourages them to learn and to succeed; and because it seems the most professional way that we know of--so far--to build real software.

I have met many real Agile dragons of all shapes and sizes. I have assisted dozens (perhaps over 100) of transitions, and many of those have adopted Agile far enough to leap beyond mediocrity. I have been a contributing team member (long-term, full-time coach/TDD developer/project manager) on at least four exceptionally successful teams: Small teams, larger teams, applications handling millions of dollars, and one life-critical ("you break it, someone may die") application.

"I'm Not Your Life-Coach, But Since You Asked..."
Do you have a friend who wants to be a great actor, a supreme-court judge, an astronaut? (Not someone simply greedy for fame and fortune, but a sincere, interested, even talented individual.) How many of his friends are supportive, but secretly think he's foolish? Do they encourage him with vaguely positive platitudes, yet privately think he should take that accounting job at his father's firm?

I would tell this person (if asked): "The odds are against you, and you may have to 'settle' for less than a bullseye, but there are indeed famous actors, supreme-court judges, and astronauts in this world. Who is to say you won't be one of them? Seeing is believing, so [quit telling me about your fantasies and] go find one to talk to!"

But assembling an exceptional software team is so much easier. You're not competing for a finite number of roles/seats/launches. There are plenty of problems that software can solve, that it hasn't. You don't have to have only the best programmers. You don't have to have Steve Jobs as the Product Champion and Ken Schwaber as the coach.

I'll give the aspiring actor and the aspiring Agile team the same advice: Identify your weaknesses, work to reduce their impact. Identify your strengths, and leverage them. And don't be discouraged by an apparent lack of progress. You may never get to where you though you wanted to be, but you will be much happier with the results than if you just kept doing what you were doing. You'll "get what you need" (Mick Jagger, of course).

Visit the Dragon's Lair

I think Pivotal Labs in San Francisco will give you a brief tour, and will describe what they're doing. I know that Menlo Innovations in Ann Arbor will give a two-hour tour, and in fact their tours are so popular that they've published a book and posted video on the web for those who cannot make it out to Ann Arbor. Another in the Bay Area, I'm told, is Sarah Allen's Blazing Cloud.

A few Wise Dragons to choose from. At each of these places, you can watch real software being made using Agile management and development techniques.

And rather than falling for the cynical belief that it can never work at your shop, let me bring in my executive coaches, Lean/Theory-of-Constraints experts, and Agile Mentors (me and others), and we'll help you reveal the Wise Dragon lurking within your teams.

You don't have to believe, you just have to get over the shock.

Innovation isn't Dead, but Remember to Feed It

In mid-July, I attended a 20-year college reunion of close friends who graduated mostly from Northern Arizona University's Computer Science/Engineering college. We call these, affectionately, "Geek Reunions." We get together about every five years in Flagstaff to reminisce, eat at our favorite college restaurants (Alpine Pizza and Macy's Coffee House, both still open after decades of abuse), and to marvel at the wonders of computer science.

Except that, in twenty years, what do we have? Some cool web-apps (the Internet already existed) like amazon.com and Google Maps. We have the iPhone, and we have [insert your favorite high-tech whatzit here rather than flaming me for forgetting it]. And that's about it. (Okay, to be fair, here are others that I can't imagine living without, though we did alright without them in the 80's: Microwave ovens, cell phones, GPS devices, hybrid engines, LED flashlights, organic produce, PDF documents and ubiquitous e-mail [together negating FAX machines, which were always an insult to human intelligence], flat-screen TVs, sticky-notes, much-improved solar cells...)

I haven't read the book Where's My Jetpack? yet, because I think it could trigger a panic attack. But I heard the author on NPR describe various inventions and leaps of human progress (both far-out and practical) that were predicted to exist, but still don't. Some that I lament: Bullet-trains crisscrossing the nation, Moon Base Alpha, fusion power, a Martian colony, a world-wide food program, world peace, the elimination of cash (or at least those darn noisy/heavy/barbaric coins), ubiquitous personal digital signature authentication (eliminating check-writing, contract-signing, and--sadly--the USPS), solar-power collection farms dotting the nation...

Among the participants at this reunion were at least four previous NAU Association of Computing Machinery student-chapter presidents, including myself. The current NAU ACM President, Leah Shanker (her blog), offered to give us our usual nostalgic tour of the newly remodeled Engineering building on South Campus. We noted, too, that the Business building had undergone remodeling, and was now much taller...blocking the view of the San Francisco Peaks from every vantage point in the EGR building and on the popular grassy "quad" below it.

There has always been a friendly rivalry between the Computer Science/Engineering program and the Business College's Computer Information Systems program. But this was an affront to nature! We immediately discussed ways to regain a view of the Peaks: From mirrors to wormholes to particle accelerators to bulldozers...

Where Have All The Nerds Gone?

Apparently the NAU student chapter of the ACM had fallen into obscurity after our network of friends had left. In fact, the whole computer science program had suffered an extreme drop in enrollment.

When I graduated, the artificial bubble known as "Star Wars" or SDI (Strategic Defense Initiative) was just collapsing. Some of us had moved to various cities and signed one-year leases, only to get laid off the next week. (I can't complain. Had I taken a job with Lockheed-Martin or McDonnell-Douglas or Skynet [makers of the T-100], I would likely not be an agile coach now. For me, it was a lesson in turning a bad situation into an opportunity. Of course, it took ten years and 20/20 hindsight...)

For a while (and possibly throughout the dot-COM era?) a CSE degree was not seen as quite as valuable, or as interesting, as a MBA. And, even today, the NAU CSE fourth-year class has only two women enrolled. That's about 1 percent, and far lower than when I was there. Leah is one of the two.

Return of the Nerdi

But there was a very bright side to the visit. Leah and her cadre of mischievous engineers were doing all the things we had done: Going to programming contests, returning with trophies, spending free time in the computer labs, socializing in the labs, working in the labs, falling in love in the labs, hacking campus computers (Leah did it as part of a report on computer security, and the dean had to intervene to keep her out of trouble!), trying to interface computer software with robots, and writing computer games. They were exploring their medium.

I had to ask "so, you're not trying to invent the next killer web app?!" Leah cocked her head skeptically, frowned, and pointed out the window...at the Business building.

"Rob, Aren't You an Agile Business Coach?!"

Of course, business thrives on both technical and business innovation. I see amazon.com as originating from business innovation. Put the whole bookstore (and quite a few other stores) right in front of people, give them a way to rate products for each other (and appeal to their ego at the same time), and tell them about products that they may be interested in (with an all-too-easy one-click purchase option).

It's like a mall crossed with Consumer Reports crossed with a Personal Shopper. Amazon.com takes far too much of my money from me each year, but I get these little packages of joy delivered to my door, even when it's not Christmastime! It's brilliant.

Now amazon.com has made much of its sales engine available to developers via their web-services API. They're using a technical innovation, and that's paying well, too.

If we consider the original web search algorithm, Google was a technical (mathematical, algorithmic, scientific) innovation. I recall wondering how they were going to turn that into a business success...until they came out with those short, text-only, search-related advertisements. That was a clever business innovation, later measured as the most effective use of advertising dollars on the internet.

I knew that Leah's reference to the Business building was her way of doing her part to keep the CSE vs. CIS rivalry alive, and I was encouraged by that. We need both forms of innovation, and keeping them somewhat separated is probably beneficial.

Imagine if the Google founders had waited until they had all the infrastructure ready to make money before enabling search on the Web. Imagine if amazon.com had waited until the advent of standardized web services before launching their site.

I wonder how many companies have failed because they refused to launch until they had the market all tied up in a neat little ball? (Remind me to tell you the sad story of Cymerc.com. They relied on Cysive to build the product, and Cysive was not agile. And now they're both dead.)

Some companies (particularly start-ups) rely mostly on one or the other form of innovation, but most midsized or larger companies these days need to consider how to stimulate both.

Seeds of Innovation

The unbounded, un-purposed exploration that's again happening in my alma mater is a key ingredient to innovation. Whether or not some great invention will appear from the halls of our Universities; this is not my concern. But software development teams need an environment that is similarly conducive to exploration, and perhaps tolerant of a touch of mischief. The mindset of exploration is the driver of innovation, and cannot be forced or coerced, even with bags of money.

I dearly miss Flagstaff, the NAU campus, daily interaction with those friends, and that feeling of exploration. Yet I left the reunion with a happy heart: Software innovation is not dead!

First Steps Towards Agile

Yesterday I received a note from a fellow NAU alumnus who found me on LinkedIn and asked where to start with a small-team transition to Agile.

Aside: I sent a reply; mostly an unedited stream-of-consciousness, but I thought it was good enough to post here. I'm not sure I'd give the same advice to everyone, but it seemed to make sense for his circumstances as I understand them. I got the sense that he's in need of a do-it-yourself Agile transition, perhaps due to his remote location. (Though I suspect he knows that I'd travel to his particular region of the country without hesitation. I give discounts for certain personally-nostalgic locales! :-)

Feel free to comment with other suggestions. What would be the first two or three steps you would take in his circumstances?

The Letter

I was wondering if you wouldn't mind helping me? I lead a team of developers, analysts and a webmaster...read a lot about agile and think it's potentially a great way to go. How do I even get started?

To date I started doing some small team-ups for projects we have. I have assigned tasks to different team members. I know this isn't agile but up until this time our team developed everything silo'd. They all had their own projects and never worked on someone else's work... So I figured it would be a good start to at least expose them to teaming...

From here I am really not sure what to do nor do I deeply understand the concepts in agile. I have read a ton on the internet but there is nothing like experience to really help.

Effectively, I suggested he start with a retrospective...

My Reply

Hi,

I think I have a great answer for you, but I want to say a few words beforehand so you don't think I'm just handing you more to read.

I was glad to hear that you're getting them used to working together as tiny teams. I suggest that small organizations try to partition the teams by separate code-bases. Products tend to map one-to-one with a code repository, and teams work best when they map one-to-one with a product. People often try to have multiple teams with their hands in one code-base, and they often discover that they're inadvertently impacting each other. If someone can impact your team, that someone is effectively on your team. But I'm wandering into dangerous territory since I'm not familiar with your set-up.

When a team starts to do things differently, it is difficult to figure out what to do first, or to just jump in with both feet (or head-first, perhaps?). So keep that in mind: Ultimately, you have to decide what's the right first step for change.

It may sound odd, but the first thing that needs to be in place is a mechanism for getting anonymous feedback on how the process changes are being experienced, and what the team's biggest problems truly are. I would start with a "retrospective" meeting. I suggest that teams do this every two weeks. It doesn't have to take more than an hour:

1. Get everyone in a room together and have them write troubles on sticky (Post-It) notes and have them post these stickies randomly on a wall. This gives people the ability to anonymously voice what they see as the biggest pain points. No limit to the amount of troubles.

2. Then have them go up in small groups and rearrange the notes. Not in rows/columns, but in clusters by how closely related the troubles are to each other. Duplicates are very close, similar topics farther, and wild suggestions farther. Let the stickies clump organically, so to speak.

3. Then look for the biggest cluster. Help the team find a way to fix that.

If your team is too small to distinguish a cluster after step 2, replace step 3 by giving each person three stars or colored sticky dots to use to vote for their top three problem areas. We usually require that they cannot vote for the same problem more than once. You'll probably find that they all agree on one or two major problems.

That's one of the first things I would do with your team. But since I don't know what they would come up with as the biggest problem, I can't suggest further action except to go out and obtain a copy of The Art of Agile Development by James Shore and Shane Warden, and read it as though your career depends on it. I worked closely with Jim Shore in 2001 and he knows how to solve problems. Most of what they have recorded there is stuff that we were doing in 2001, and it's still some of the best "agile" advice, all collected in one place.

Lastly, be aware of the two primary reasons why agile projects fail, and how to prevent them:

1. The Product Champion (Scrum "Product Owner" or XP "Customer"), the person who decides what features go in and what features don't, and also chooses which order they go in, must be readily available, willing to work on creating "stories" with enough detail and acceptance criteria so the team knows what is needed, how to test it, and how complex it is (for estimation). And this person must be ready to decide what the top priority detailed stories should be done first. Without this, the project will probably fail. "Agile" is not just a developer's process, it tends to impact the way the organization builds software product.

2. Too few automated tests. I call this the "Agilist's Dilemma" because without tests it becomes obvious to the developers (whether they'll tell you or not) that it's getting harder and harder to change the code to accommodate the next set of stories. At the very least, encourage developers to try Test-Driven Development practices wholeheartedly, right away, for two 2-week iterations. They'll need to learn how to do it right, and with some discipline, to get the full value. Dave Astel's book is good (Test-Driven Development: A Practical Guide), Kent Beck's book is good (Test Driven Development: By Example).

My courses are even better, because you get hands-on experience plus team-specific guidance! I'm also available for coaching, of course.

Serendipity At StarEast

This same question (effectively "What are the first steps?") came up after Lisa Crispin's talk at StarEast today. I like her answer, too. She responded with these two (I'm paraphrasing):

1. Get some Agile training for the team.
2. Talk with managers about how their roles will change, and let them know that the team will need their support and confidence.
Summary

So perhaps I'd alter my list to be:

1. Get some Agile training and/or hire an external coach.
2. Run a retrospective on the current, pre-transition process. Refer to Art of Agile for potential solutions. Try the chosen solution for two weeks. Repeat.

...And include management in both!

Techniques for Agile Trainers

The following are skills/techniques/style tips that I sent out to a friend recently. I cc'ed some of my training mentors (Scott Bain of Net Objectives, and David Bernstein of Techniques of Design), and David suggested I blog this and get feedback/additions from other trainers (Agile or otherwise, really).

Originally I tried to categorize them, and the categories became a hindrance. So here they are in no particular order:

Projection: I had to learn to "project" my voice. To bellow out words without sounding like I'm yelling. It's like singing loudly in your car, but you have to do it when people are looking.

Immersion: The best courses (for everyone) are those where the instructor says less, and the attendees do more. Build creative exercises that give the attendees the chance to really build the right skills (while avoiding out-of-course-scope skills, e.g., don't ask testers to write code).

Diplomacy: This one is Scott's suggestion to me: If you get someone who wants to argue, give yourself the chance to really listen, and try to see how that person came to those conclusions, then make them right: "Yes, you could implement the pattern using a switch statement. And it would still be The Pattern! So, what do we gain and lose by using classes?"

Mimicry: Borrow techniques, phrases, and jokes from your favorite instructors. I can't teach just like Scott (I tried many years ago), but I do borrow from his style. I'm also still borrowing heavily from Fred George after 10 years, and two of my favorite college professors from 20 years ago.

Authority: You may want to tap your inner extrovert. Sometimes it feels like I'm wearing a mask, or playing a role, except that it's still me. I have to talk with authority, but avoid rambling excitedly about some fascinating (to me) detail that will lose people. Strangely, the "Helpful Guide" gets a far better reaction than the "Curious Scientist" persona. It's a shame, because I'm really the latter at heart. But I do let my inner geek out to play from time to time.

Honesty/Humility: No one appreciates a know-it-all. Allow yourself to say "I don't know. I'll see if I can find out for you before the end of class." (If you are worried about losing credibility, try "That's fascinating! I haven't run into that before. I'll definitely want to look into that and get back to you.") Or utilize the giant pool of knowledge sitting in the room: The attendees. I've taught some very bright developers, and they often know much more about the latest software tools. I couldn't possibly keep up with them all!

Also, allow yourself to make mistakes. Try not to point them out if it looks like they're trivial (like having two slides out of order, or Java code in a C# slide...), but if you misspeak, correct yourself.

Focus: Keep the course on-topic. If there are questions about another topic, offer to do a lunch-and-learn, or stay late, or come in early, to hold a brief talk about the other topic that's worrying them. Don't try to throw everything into a single course. It may be entertaining, but the necessary skills will be lost. It's also poor marketing, as they'll assume they don't need you back.

Pragmatism: Teach what you know, and what you believe in (or have seen work in the past), and continue to get real experience in what you teach (that's a tough one).

Others? (With thanks!)

Mitigating Mediocrity in Middle Management

James Shore just wrote a post on "mediocrity" which nicely explains why Agile doesn't always scale well: It isn't Agile that's being scaled, but corporate inertia. If an organization isn't going to commit to doing things differently, they have succeeded only at mediocrity.

Some of the follow-up comments got me to thinking about the problem. People were asking how to get middle management on-board. I think a few of the questioners realized that finger-pointing, and arguments, and directives to "Just do it!" are counter-productive. It's not that managers don't see the value in Agile methods, but it's not clear how their day-to-day behavior is supposed to change.

We need a little root-cause analysis here. Businesses are run by people, and people are motivated by a variety of things: Self-interest, compassion, fear, joy, confusion. Change is unnerving, particularly when one is not the instigator of the change.

I find that people resist change at work when work itself is a source of stability in their lives (e.g., they would rather be at work than at home), and/or they see their job as just a job, and not a productive, change-generating career. In traditional organizational structures, middle management is detached from the two most dramatic regions of the org chart.

So if I'm a middle manager, and I like my stability, I resist the change. If I would like to be part of the change, I see my own involvement as limited. And, do I even have a role (and a job!) in this new world order???

The good news is that, on a truly agile project, there is plenty of interesting work, and learning, for everyone. Our first step is to stop thinking of them as managers, and start thinking of them as leaders.

I once heard a very good quote which helped me out of a career rut. I'll paraphrase: "Not everyone gets to do what they love [C'mon, I'd be co-authoring novels with Larry Niven and Vernor Vinge. In Maui. Shaka, Bra!]; but we must love what we do."

That's a non-mediocre goal, without being an impossible stretch towards perfection. You help the team by helping yourself grow, and you help yourself by helping the team.

I can't count the number of times I have met a manager who says, somewhat forlornly, "I used to write code..." Leadership involves some wonderful skills that we can learn and be proud of, but we may want to reconsider the "one ladder" approach of moving our most talented technical people into pure management roles.

Nor should we necessarily get our managers directly from a pool of young, non-technical, certified PMIs. We need to be picky, and find people who are familiar with technical endeavors and are willing to lead people towards a common goal.

Maybe that's why "coach" has such appeal for me: It's technical, and people-oriented. I help other people (developers, testers, managers, directors, VPs, everyone) find ways to enjoy what they do while successfully building quality software products. I do love what I do!

Do you? (If not, we can help.)

Further Reading

Jim's post: http://jamesshore.com/Blog/Stumbling-Through-Mediocrity.html

Agile Bob's comments regarding Jim's post: http://www.agileforall.com/2009/03/30/new-to-agile-dont-settle-for-mediocrity/

A Fresh Perspective on Shu-Ha-Ri

Shu-Ha-Ri is a learning model. It is one of those timeless formulations that is both pragmatic and universally applicable. Like many good, ancient models, it is elegant and simple. Of course, misapplication of such a model can lead to oversimplifications and missed details.

I recently stumbled upon a passage in a book that made me think of Shu-Ha-Ri, although the author did not mention it explicitly. I'll get to that, but first...

What is Shu-Ha-Ri?

Shu-Ha-Ri is a very old way of looking at the way we learn and teach new things. Shu-Ha-Ri divides understanding into three levels, Shu (hold), Ha (break), and Ri (leave).

In more detail (and perhaps colored by my own interpretation):

Shu: The beginner, or apprentice, needs (and usually wants) to be given the rules, so that he or she can be productive. The Shu practitioner follows the rules, and wants to know when the crayon is moving outside the lines. Creative interpretations by a Shu practitioner can lead to disaster. At this level, finding a good teacher is essential.

Ha: The explorer, or journeyman, knows the rules, and when and how to bend them. There is more creativity involved. Exploration "outside the lines" by the Ha practitioner usually results in a high degree of craftsmanship. Self-learning can occur, but the practitioner is also thirstily seeking varied mentors.

Ri: The expert, or master, is not hindered by the rules. In fact, the master practitioner can create new rules, and sometimes these rules seem to conflict with old rules. Some have suggested, tongue-in-cheek, that this level can be summarized as "Rules? What rules?" This is not accurate: The master of a practice can see the underlying truths behind the rules, and thus does not need to consciously think about the rules. This does not imply that a master can do whatever she wants. The master's actions embody the truth discovered within the context of the practice. Being an expert at something does not make one perfect at anything.

No One is Perfect

No one is an expert at everything, and only a newborn is a beginner at everything. Each of us has a level of proficiency in a variety of skills. It can take a lifetime to reach mastery for even a single practice.

No one stays at one level. A beginner can have a creative insight that turns out very well. An expert can forget and relearn the simplest and oldest rules. I'm told that if you think you've mastered something, you probably haven't; and if you say you've mastered something, you certainly haven't.

A Strange Loop
Experts (Ri) may have a hard time teaching beginners (Shu). (I think it was James Shore who pointed this out to me while we were building a course together.) Training comes from at most one level "higher" than the intended audience. Masters (Ri) can provide excellent examples by how they live their craft, but are not always able (or willing) to communicate effectively with an apprentice.

So those who can train beginners in a practice have learned another skill: Training. It's not enough to be an expert alligator wrestler if you want to train others to wrestle alligators.

Since becoming an "Agile Trainer," I've realized that there is a whole new dimension to my interests. I try to absorb anything I can regarding software-development, process, and teaching.

Learning to be a teacher... Is that Shu-Ha-Ri applied to Shu-Ha-Ri?

Taking Our Places
I recently read a pleasant little book called Taking Our Places: The Buddhist Path to Truly Growing Up by Norm Fischer. In it, there's a section about teachers and teaching, and he outlines three levels:

Literal: We follow the rules to the letter.

Compassionate: "...we may find it necessary to go beyond rules, laws, or customs, motivated...by compassion, which sometimes causes us to transcend the literal level of good conduct for the purpose of helping others." [Fischer, p. 182]

Ultimate: "...we come to appreciate that the [rules] are deeper than we had ever imagined; so deep that they can never be completely understood." [Fischer, p. 182]

Sounds a lot like Shu-Ha-Ri, yes? Fischer never mentions Shu-Ha-Ri, but I suspect the parallels are not coincidental.

I'm quite inspired by the implications: I notice that the level that may be most suitable for trainers like myself (Ha) maps to Fischer's level where compassion is the primary motivation.

Further Reading
There has been a lot of "tweeting" recently regarding Shu-Ha-Ri:

http://search.twitter.com/search?q=shu-ha-ri

Wikipedia has a (very) brief introduction:

http://en.wikipedia.org/wiki/Shuhari

Ward's good old (original) wiki has one of the best introductions I could find on the web:

http://c2.com/cgi/wiki?ShuHaRi

Alistair Cockburn made some valuable clarifications:

http://alistair.cockburn.us/Shu+Ha+Ri

Ward Cunningham's Debt Metaphor Isn't a Metaphor

Last month, Ward Cunningham posted an excellent video on YouTube regarding refactoring and "debt". If you haven't seen it, I have it for you right here...




But is it simply a metaphor, or is he describing a real fiscal debt associated with software development activities?

The form of technical debt that Ward is referring to is noted by developers as poor, brittle design. Since a poor design is one that's difficult to change or maintain, it will take developers longer to add features, and their changes will more likely result in defects. They will tip-toe slowly and cautiously within (or around) untested code, and even the best developer will inadvertently introduce the occasional defect.

Isn't developer time expensive? Of course. So, in effect, Design Debt is real debt. I wouldn't suggest you create a liability account in QuickBooks: It's difficult to measure the actual dollar value, and it manifests itself in subtle and even contradictory ways. For example, remember the old story about the manager who proclaimed, "if only my developers could type faster!"? That's akin to saying "if only I could use my credit card faster!"

Design Debt feeds into another form of debt which is easier for the business to see: Quality Debt. This can be quantified as the number and severity of defects experienced by users. This metric has a fairly direct relationship with the financial impact of lost sales, time spent on support calls, developer time spent debugging the product, and on and on (and on).

Yet how many teams just assume that Quality Debt is "the nature of software development"? Well, doctors were using rattles and snake venom 50 years into the profession of medicine. Aren't you glad they're not still clinging to those archaic practices?

Unfortunately, what we usually try to do may just make it worse...

The functionality grows, the complexity grows, and it all has to be tested. We hire more testers, or give ourselves longer for the test "phase" of an agile iteration. (Yes, this "mini-waterfall" approach has been noted on many "agile-ish" teams. If waterfalls are bad, why are more of them better???)

And the stack of manual test-cases grows and grows, until the poor test team is working late at night to get them all tested: Point-click-type-point-click-type... and even our infallible testers make mistakes!

Or, the team runs only the "critical" tests each iteration, and saves the whole suite for the pre-release shake-down cruise. But which tests--which product features--are not critical? And waiting until the end...well, that sounds a lot like waterfall again, and also sounds like a Home Equity Loan, if you get my meaning.

The third form of debt is Testing Debt. The anecdotal measurement of Testing Debt is the height of the stack of printed manual test-cases. A better measure would be the time it takes to run the entire regression suite. How long after a change does it take the team to know that they haven't broken anything they've already built?

On the University of Michigan Transplant Center's two-year "OTIS2" rewrite of their aging Organ Transplant Information System; and after weekly reminders that "a mistake could kill a patient"; the answer was 15 minutes.

My goal isn't to make you feel bad in comparison, dear reader, but to point out what's truly possible without any technical debt.

Is Testing Debt real financial debt? Well, there's testers' time and salary. Also, Testing Debt again feeds into Quality Debt. If you don't assure that there are no bugs, your customer will find them for you.

But It's 0% APR, Right?

There is interest. You know what you get when you add features to buggy code? Yeah, more buggy features.

Thanks! You've Ruined My Day. Now What?

First, in the immortal words of Douglas Adams: Don't panic. How does anyone get out of debt?

1. Make regular payments.
2. Stop accruing more debt.

Examples of how agile teams can pay down technical debt:

Design Debt: When tasking and estimating stories, the team should consider the refactorings necessary to get any new story designed correctly. Those refactorings become explicit tasks.

Quality Debt: Set aside time within each iteration to explore and fix high-priority defects, or create a story for a group of related bugs. On a well-running agile project, each reported defect becomes a story of its own.

Testing Debt: Set aside time within each iteration for testers to automate those "critical" tests, or a particularly onerous set of tests. If you're not ready for that, create a story for the team (testers and developers working together) to explore automation tools, including record/playback tools as well as ATDD tools such as Fit. Yes, you're going to have to slow down the introduction of additional stories/features.

Ways to avoid new debt:

Design Debt: Developers, adopt Test-Driven Development (TDD) and refactor in tiny increments as you identify seemingly insignificant trends towards poor design. Whenever a task or story requires a new variation in your design, first refactor the appropriate piece of the design into one that follows the Open-Closed Principle for that variation. (That's as concise as I can be without going into gory detail here. Explaining this technique requires lots of UML, at least, and I'd probably rather use a video medium.)

Quality Debt: Pin down desired behavior immediately with fast, automated tests. Adopt TDD and ATDD (Acceptance-Test TDD) practices to keep away any new defects. Avoid feeding Quality Debt from the other two forms of debt.

Testing Debt: Stop writing manual test cases, perhaps today. As soon as practical, build acceptance tests for stories using something allowing test-first automation such as Fit, Robot Framework, or Cucumber. It's also up to the team to find ways to make those tests run fast.

Let 'Em Eat Cake

The Case of the Missing Piece of Birthday Cake

Recently I spoke with a friend (let's call him Pete) who manages a software project, and he was telling me that his division had recently stopped paying for employee birthday cakes. More accurately, they had discontinued the funding of petty cash expenses. He apparently told his team "Would you rather have jobs, or cakes?"

Pete's small, profitable division of a struggling gargantuan company had been a small, profitable company of its own. It was sold, and the founders had all departed with well-deserved wealth.

Employee birthday cakes were a carry-over from the good old independent days. So the thinking in upper management may have been this: "It's unfair to the other divisions to be giving this perk only to the new acquisition, and we can't afford to give such a perk to all divisions."

Pete's solution was to collect a small bit of funds from each person on the team, so that the tradition could continue. It sounds reasonable, and Pete seemed pleased with himself. But then he continued...

There is one guy on the team (Carl) who conveniently forgets to pay in, yet eats a piece of cake, "or possibly two!" On one recent occasion, Sally (another team member) had purchased the cake with her own funds, and "now she is still going around trying to collect from everyone else," including Carl the Cake Thief.

Pete finished his story with: "Carl is lazy, and he's been in trouble before, and people talk about him behind his back, but he has a lot of product experience, and I can't let him go..."

Biting My Tongue

At the time that Pete and I had this conversation, I was acting as host, not consultant, and kept quiet. Pete was on vacation, and simply needed to vent. But my response would not have been as simple as this post's title would suggest.

It's true that I would encourage Pete to find a way to continue the birthday cake tradition. Sharing a meal is a natural team-building activity, and small, simple rituals are good for morale. But imagine how Carl's perceived pettiness is generating bad feelings. And surely Sally has some critical task, more important than collecting dollar bills and gossiping about Carl's second slice of cake.

And did Pete really say "...jobs, or cakes?" to his team, during a difficult transition and a very deep recession? I can't imagine how one could deliver those words without it seeming like an ultimatum, and thus putting everyone on the defensive.

I see two management anti-patterns at work here. First is the "Penny Wise, Pound Foolish" approach taken by upper management: cutting off a cost without regard to the long-term effects on profitability. How often do we need to see the proof that employee morale has a strong impact on corporate profitability?

The second misstep is Pete's attempt to make things better by "Treating Symptoms."

Go Lean

I'd start with two principles straight from the Lean playbook: Look to eliminate the greatest waste, and respect your people.

Had upper management worked to identify the largest waste in the whole organization, cakes would probably not have been at the top of the list. And if they were, what about the costs of chipping away at a profitable division by removing its small rituals?

Whatever is at the top, the next step is to use Root-Cause Analysis: A heavy phrase for a simple technique of repeatedly asking "Why?" and looking deeper each time until you find the underlying "root cause." Perhaps these cakes were originally Black Hound cakes overnighted from New York...

Then we try one or more of the proposed solutions: For example, Pete (who is a fine baker) could start baking the cakes from a boxed mix.

Much of the burden of change probably falls upon poor Pete. He needs to bring any issues that will affect the performance of the team directly to his superiors. Honest communication must flow in both directions in order for a team to succeed.

For a manager from an acquired company, during a monster recession, telling the boss that a proposed change carries risks may seem like political suicide. Rightfully, Pete is concerned about keeping his own job. But for the long-term health of his team, his company, and his own career, this is the very point where Pete needs to take a few risks and show off his leadership skills. He needs to keep the team running smoothly and generating valuable results, not quietly bickering over a slice of cake.

One of his first tasks should be to talk privately to Carl, the cake-eatin' code-cowboy. Is Carl having tax-time troubles after exercising his stock options? Does he need to hang on to every dollar, and eat every calorie he sees because there's no food at home?

Was Carl the top technical guy before the acquisition, and now he's buried somewhere in a 20-layer org chart?

There are people who are truly, simply lazy, but more often people are just exhausted, depressed, bored, or unnerved by all the organizational change. Pete needs to find out what's motivating Carl, and Carl should be encouraged to find his place in the new organization, or perhaps look elsewhere.

Touchy-Feely Psycho-babble?

I tend to give people who are causing trouble the benefit of the doubt. When people are able to see how their actions impact their organization, the people around them, and their own careers, they tend to behave helpfully and professionally.

Has it ever backfired? Oh, yes. I've worked with people who repeatedly ignored requests to alter unprofessional behavior. (Yes, we fired them. We also adjusted our hiring practices, but I'll save that story for another post...)

The successes will always far outnumber the failures. Getting a key player to enjoy his work again is much more valuable in the long-run. Were I to focus on the failures, I might stop trusting people, and immediately fire poor performers. Results: The team would lose all trust in me, and would stop providing critical information regarding the project. Deeper failure lies in that direction.

Mitigation

What if Pete talked with Carl and gave him a second chance, but Carl doesn't try to improve, or abruptly quits? Remember, Carl was a key part of the product team prior to the acquisition.

Well, it would be too late for Pete, then, unless he had the foresight to mitigate the impact of Carl's departure.

No single team-member should be able to effectively ruin a team by quitting. To allow such a situation to arise is what I would call "Believing in Superheroes," another managerial anti-pattern. When the team first discovers that there's a player who is pivotal to the success of the team, that player needs to take on an apprentice. (That simple act, making Carl a coach, could be all Carl needs to regain his enthusiasm.)

On Agile teams, we have people working closely together, and sharing ideas and techniques. No one has to be able to do everything. In fact, the team doesn't even have to have two experts for a particular topic. Just one expert, and someone else who knows where to look, what to learn, and how to fix minor problems. In a crisis, the team can pick up the pieces and move on.

Leadership Emerges

Within days, I had another visitor with a similar story: A Business Analyst visiting from the Midwest (we'll call her Betty) was discussing the economy, and she told me that her team had added up the cost of "Wednesday's Bagel Breakfast" and had voted to stop expensing it. The next Wednesday, there were still fresh bagels. Apparently the managers had decided to pay for "Wednesday Bagels" out of their own pockets.

Perhaps management was impressed by the team's self-organizing attempt to reduce waste. And perhaps the managers realized that their success was by virtue of their team's success.

Uh, Just Checking

Rob (paraphrased): "Bagels every week?"
Betty: "Or other pastries."
Rob: "Is the team successful?"
Betty: "Yes!"
Rob: "You know this?"
Betty: "Of course. Why work on an unsuccessful project?"
Rob: "Do you like your job?"
Betty: "Yep!"

I'm not surprised. Pete, are you listening?