Jean Tabaka and others devised this statement, and I love it. I heartily endorse it. To my mind, community provides the best mechanism for continuous learning, for continuous improvement, and for whatever safety we can expect from the world. Were I asked, I would add the meme of Network Weaving as an essential evolutionary mechanism for any community of thinkers, but that’s just the hippie in me.

Well done Jean, Liz, Eric, and others. Good stuff:

A Community of Thinkers

I am a member of a community of thinkers.

I believe that communities exist as homes for professionals to learn, teach, and reflect on their work.

I challenge each community in the software industry to:

• reflect and honor the practitioners who make its existence possible;
• provide an excellent experience for its members;
• support the excellent experience its members provide for their clients and colleagues in all aspects of their professional interactions;
• exemplify, as a body, the professional and humane behavior of its members;
• engage and collaborate within and across communities through respectful exploration of diverse and divergent insights;
• embrace newcomers to the community openly and to celebrate ongoing journeys; and
• thrive on the sustained health of the community and its members through continual reflection and improvement.

I believe that leaders in each community have a responsibility to exhibit these behaviors, and that people who exhibit these behaviors will become leaders.

I am a member of a community of thinkers. If I should happen to be a catalyst more than others, I consider that a tribute to those who have inspired me.

”A Community of Thinkers” by Liz Keogh, Jean Tabaka and Eric Willeke is licensed under aCreative Commons Attribution-Share Alike 3.0 License. Please attribute to the distributor of your copy or derivative.

Background

By way of even more background, my original post on this topic is here.

Patterns are Emerging

In response to several comments and conversations with people in the industry, several of us, most recently Chris Beale, Gary Baker, John Huston, and Daryl Kulak, have been converging on a definition of this new role, this new variety of leadership that we want agile teams to have. Many others are also involved in the conversation by now; too many to list here. The more we talk about it, the more comments we receive, the more all of us can see what we want. We are not in complete unanimity, by any stretch. Some folks think we are smoking crack. Not the first time. But most of us in the conversation can see useful patterns emerging.

This is a Big Role

We can also see that this is quite  a lot of skill, experience, leadership, and sheer work to expect of one person. So part of my premise goes like this: the Agile Team Lead is the person that is ultimately accountable for ensuring that this work gets done. Depending on how the skillsets, preferences, team maturity, and other factors shake out for a leader and his/her team, the Agile Team Lead may well end up delegating much leadership to others (and other, subordinate leaders) within the team (the key phrase there being “within the team.”). So an Agile Team Lead might very well delegate to a Tech Lead, to an Agile Test Lead, to a Project Tracking Lead, to a Continuous Deployment Lead, etc., to cover off concerns. Whatever. The main point, of course, is that the buck stops with the Agile Team Lead. That person is, for whatever duration they are lead, the single wringable neck (for all responsibilities owned by the team itself).

This Role Must Emerge From Within the Team, and Must Satisfy All Stakeholders

So, while I am decidedly talking about DESIGNATING a person to act as the team lead, I am decidedly not talking about a traditional manager being imposed, permanently, from above, by fiat. There can be no Agile Team Lead designated FOR a team that does not know, admire, trust, respect, and like that person. Really, the final say in who leads the team should be the team’s.

This Role Should Likely Rotate

This may be something we want several team members to try out, to get good at. Team members could rotate through it, perhaps on a fixed schedule.

This Conversation, Um, Leads to a Few Others

I’d like to postpone to another series of blog posts, (already underway, however), the thorny issues around the enterprise structural implications that arise from this new role, and related observations. Suffice it to say, for now, that I personally don’t know how to make agile teams work, nor how to make this role of Agile Team Lead work, unless everyone in the team room reports directly to the Agile Team Lead, and no-one else. I mean that that one person is accountable for all of the incentive structure, performance evaluation, and (in large part) the hiring and firing for everyone on the team. This does not, again, mean that they own it entirely, nor that they do it permanently.

One other premise: we are deliberately attempting to erase old agile team management role and responsibility labels, like Scrum Master and Project Manager, which we increasingly believe to be fundamentally broken. We find the controversy around this topic to be perfectly matched to the stakes involved for the enterprise: our current patchwork of agile team management and project management and product ownership and process improvement management consists of more holes than cheese. Too many team-level concerns go unmet; too many stakeholder concerns go unmet; too many user and market concerns go unmet. We have been in denial about this shortfall. Let’s give up that denial.

Love it or hate it, we need a paradigm shift around agile team leadership. We need that leadership to be better consolidated, better focused, better defined, and better matched to the dialectic between internal team needs and external stakeholder needs. We need one person who is held accountable the extent to which an agile team’s authority and responsibility are, in fact, congruent.

And we need none of that to violate the principles of self-organizing teams, leadership arising within the team, and teams owning the team’s concerns. The team owns all of these things, but the team lead is accountable for the team actually delivering it. Like I said, thorny.

Summary: What we Want in Our Agile Team Lead

Most of the following is in summary, bullet-point form, with a bit of explanatory text. We’re still mainly starting this conversation; much is left to be done to think this through from the perspective of the team itself, from the perspective of “continuous improvement,” from the perspective of individual Releases/Projects being executed well, from the perspective of Product Owners and Portfolio Owners, from the perspective of all of the different natural roles, responsibilities, skills, learning styles, and personality types that the team needs. These is just another draft sketch of the responsibility categories of this specific leadership role.

I suspect there are still several holes in our thinking, several bits of overlap, and perhaps some false premises. No matter. It’s out here now for us all to discuss. Better in front of your eyes here in public, than lurking, not yet “perfect,” on someone’s laptop.

Continuous Leadership

• Understand the goals of the organization, and the exact ways in which the team furthers them
• Provide a single point of leadership for the team; continuously wins the team’s implicit and explicit trust and admiration. Builds a “safe container” for the team within which work can be done creatively, collaboratively, honestly, accountably, and joyfully.
• Helps continuously build trust and respect between the team and its stakeholders (in a way that buys more and more freedom to speak up, freedom to experiment, freedom to “fail,” freedom to learn).
• Provide a single point of team accountability to the team’s stakeholders (in a way that ensures a single, coherent message to stakeholders about any team commitment about what went well, and what did not go well, how it happened, and why).
• Helps continuously improve internal team-cohesion: this includes mutual respect, mutual trust, morale, rapport, celebration, accountability, fun, and affection (yes, we have to like each other).

Continuous Planning

• Ensures that the team is making more and more accurate and precise commitments.
• Ensures that the team is delivering more and more predictably on its commitments.
• “Gets the lights on,” in every area of process and practice, and ensures everything is “big and visible,” both tactically within the team, and for purposes of publishing to external stakeholders.
• Identifies, gathers, and acts tactically on meaningful performance/progress metrics
• “Makes the plan the bad guy“: ensures that Definitions of Done, short-term scope, long-term scope, deadlines, quality measures, and all other commitments are expressed in “plans” that remove moral judgment and blame from continuous improvement. Plans include code, tests, cards, and other artifacts. The goal is to create an environment in which mistakes are expected, any shortfall is cheap and easy to learn from, and everyone’s reflex is to refine the “plan,” not to point fingers of blame.
• Drives Commitment-making for and with the team. This includes ensuring that the Definition of Done for any small or large commitment is precise enough and deterministic enough to reduce risk of misunderstanding, scope creep, and under-delivery as much as possible.
• Ensures that the plan changes with demand/supply.

Continuous Execution

• Monitors/manages team velocity/throughput. This includes work breakdown practices, estimation practices, tracking practices.
• Reports meaningful metrics/data within the team, tactically, and likely publishes different metrics/data outside the team to external stakeholders.
• Gets the team the resources it needs to deliver on its commitments; removes roadblocks
• Escalates blockages effectively. When the Agile Team Lead cannot unblock something, he or she escalates the issue to someone with the authority to remove that blockage, and makes clear to all stakeholders what the risks are of leaving that issue unblocked.
• Keeps flow, momentum and focus in the team.

Continuous Risk Reduction

• Continuously identifies short-term risks (both Sprint-scope and long-term, Release-scope and Portfolio-scope) across the full spectrum of people, process, technology, structure, and culture.
• Makes risks and potential impacts big and visible to the right people. This is another example of “Making the Plan the Bad Guy.”
• Ensures that risk reduction is integrated into planning and execution
• Measures the effectiveness of risk avoidance and the impact of unmitigated risk.

Continuous Improvement (Agile Coaching)

• Drives the improvement the overall Definition of Done (at each level of commitment scope, from the Story through the Release, and perhaps beyond). This has the effect of improving the quality of commitments, building a habit of overdelivering, building stakeholder trust, smoothing the flow of work (with fewer and fewer scope misunderstandings, production defects, and other “unplanned work phases”).
• Senses where performance breakdowns are occurring
• Creates big and visible means for measuring breakdowns
• Interprets data to identify biggest opportunities for improvement.
• Works with team to identify and implement improvements. This obviously includes owning a retrospective practice, and also includes owning the effectiveness and accountability culture around a reliable pattern of Inspection and Adaptation. It includes things like helping the team learn the difference between ego-based whining and team-based servant leadership, and the difference between vague negative generalization and concrete improvement proposal.
• Investigates and learns emerging practices from other teams, organizations, and communities.

OK, so once more, some disclaimer. This is work in progress. Expect more posts, other venues of conversation. Meanwhile, we welcome, indeed we desperately need, your input.

The Old Atomic Unit of Production: the Project

I came up in a software industry in which the atomic unit of production capacity was The Programmer. Programmers were largely assigned, as needed, to ephemeral entities called Projects. Yes, it had a Taylorist mechanistic undertone, and often still does.

Cut to scene: The Project Manager whisks into the Development Manager’s office and declaims, rather like a Hollywood casting director, “Well, you need to get us Joe and Terri for the Melathobner Project. We can’t do it without them. It just won’t work without them.”  (Lots of action films have the word “Project” right in the title!  Coincidence?  I think not.)

As a film crew comes together once, against tremendous odds fought by various producers and directors, to birth a film production, so have I seen most projects run. As if after this project, the world might end. As if after this project, we could all retire. As if deep down in their hearts, stakeholders and managers secretly believe this: “The idea that this project might actually make it out the door on-time, under-budget, with few defects?  That’s hilarious.”

Then, at miraculous Project conclusion, all cast and crew return to their structural silos (their true homes). Then off to the next Project. Emergency permeates the whole mechanism. Whoa!  This next Project might be the one that finally kills us all!

Project: Wrong Atomic Unit of Production?

Even if a product or system might persist for years, each of its major releases (sequels, in fact) is typically still a Project. Most software projects still have this one-time, heroic, desperate, fire-and-forget feel to it. This very narrow focus, this “goal obsession” on the part of project managers was recently described well here. Certainly there are Project Managers who are better than that, and focused more broadly on notions of business value flow, portfolio, market differentiation, team productivity, context switching costs. They are the exception, partly for reasons I cover below.

One or the root cause issues here is this: as an atomic unit of production, Projects can be inherently bad. And, as I’ll discuss later, fundamentally different than an alternative granule: the Release. First, let’s go to a military metaphor, which is always galvanizing.

Corporate Silos: 14th Century Italian Civil War Between City States

Meanwhile, what, in the context of software Projects, is the corporate atomic unit of budget authority, responsibility, and power? The structural silo. Ouch. We have somehow evolved default software development structure and process that seem to quietly mimic 14th Century Italian feudal warfare.

Our structural silos, under their various VPs and Directors, are capable of behaving like city states. People who report up through Architecture, Development, Production Deployjment/Support, DBAs, QA, Facilities, etc — are typically most loyal to their role and responsibility silos. How could they not be? That is how they have been incentivized and empowered. They represent abstract notions of “quality” and “architecture” and “programming,” divorced from the very real, complex, organic, end-to-end needs of the enterprise (or worse; more below).

None of these silos owns the proper flow of business value through the organization. You could say that our corporate Italy has insufficient representation and incentive for its larger, nation-sized concerns. It’s OK with our corporate Firenza if Italia perishes, as nonsensical as that seems at first blush.

None of the city states owns whether we are reducing costs and increasing revenues for the entire enterprise. Indeed, I have frequently seen these city states incentivized to combat cost reduction and revenue increase, in order to gather more resources and power to themselves. QA is incentivized to find bugs, and find them they do; their weapon of choice: the bug report system. Architecture is incentivized to represent the bigger picture, the technology future; their weapons of choice: Visio and PowerPoint. And so on. It’s a fabulous buck-passing machine at best and, (segue alert! New metaphor) at worst, as if some corporate departments are like parasites feeding on the enterprise as host organism.

Note that I am not saying that the enterprise does not need to address the concerns and skills and worldviews traditionally represented by architects, testers, production deployment and configuration experts, project managers, etc. I am not against all specialization.

But meanwhile, again and again, the Project Managers assemble these temporary Waterfall initiatives, using the best go-to condottieri (think “Ronin”) they can find, and lead them through Waterfall-phase silo-gauntlets, one at a time. As the project winds down, the temporary allegiances dissolve. Back everyone goes to their little city-state.

So, as I’ll discuss, what I increasingly question is the idea of empowering structural silos that represent sets of skills, concerns, and worldviews, using budgets, hiring authority, etc. If that’s how the money flows, into these little city states, I think you should expect your own little replay of the 14th Century Italian civil wars.

Voila, Disruptive Change:
The Agile Team as New Atomic Unit of Production Capacity

So we have this existing corporate socio-economic fabric: Projects being pushed, against powerful odds, through this fractured system of city states. Real business value is not created as a direct result of this fabric, but despite it. People strike backroom deals. A barter economy of favors breaks out. Grudges arise, and are repaid. It is, to my mind, more ugly than pretty.

And then along comes this notion of a self-organizing, empowered, cross-functional, high-discipline, highly-skilled Agile Team, as the new fundamental unit of production capacity. We’re going to mow down all these cube walls, bring all of these people in from their various city states, several of whom quietly loathe each other, and ask them to form a cohesive team. We’re going to ask them to sit with each other, deliver measurable business value in a steady flow together. Form, Norm, and Storm. We’re going to ask them to treat this new team space, not like a brief Project-scope visit to a kind of summer camp, but as their permanent home. We’re asking them to break their powerful, traditional allegiance to their city state. Remove that tattoo; forget that secret handshake. Even though that’s how their performance gets evaluated. That’s how they get paid. No wonder it so frequently looks like a bunch of Hippie Kumbaya.

No wonder the corporate anti-bodies swarm (oops, yes, another metaphor; should have warned you). No wonder the first several retrospectives feel like the teacher asking the class who put the thumbtack on her chair.

Yes, I am Over-Simplifying

To make a point, yes I am. The truth is, I have seen the above scenario more than once. The larger the organization I am working with, the worse it is, and the likelier it is to fight agility, and fight smooth business value flow.

But let’s at least find a common pattern language or metaphor set for this fundamental issue of “cultural change” in the enterprise. In future blog posts, which will be about how all the Hippy Kumbaya agile team culture can actually help the entire enterprise evolve in a way that results in true competitive advantage at the enterprise level, I’ll propose new terms and newish structure (not entirely my invention).

Meanwhile, though, here is my premise:

Your entire corporate socio-economic fabric might be fundamentally busted, seen through the lense of software development “Projects.” You might be recreating civil war in Italy in the 14th century. This might make a great film, but it could be a terrible context within which to try to optimize how the collaborative creation of software systems (be they products out in the market wild, or internal web services) increases your revenue, or decreases your expenses.

Ever Had a Client Stuck in a Nasty Situation?
Ever Had a Bird Trapped in Your House?

I’ve had both. In both cases, I learned not to mess up in the following way: trying to grab the bird and toss it out of the house. Birds hate that, and are really skillful at resisting the attempt. You or the bird can also get badly injured that way.

As a Shu-level, novice coach, I did the classic wrong-headed stuff, and still work hard to resist temptations to do the same stuff: pushing, pulling, badgering the client from a bad situation to a good one. It really is like chasing a bird around the house, brandishing a tennis racket. (Note to self: No!)

Releasing Attachment to the Outcome

We cannot truly control whether, indeed, the bird leaves the house, much less how fast or how well. There are no shortcuts for us or the bird. We can, however, strongly influence how the whole thing unfolds.

It really does have to be the bird’s idea to leave the house, if it will ever in fact happen. This may take a few seconds, a few minutes, a few hours (a few weeks! a few months!  a few years!). Take your time.

Those of us who have (as I have) released birds, bats, and squirrels from the house have learned (A) how to open and close doors, and leave seed trails, in a way that invites the unhappy bird or squirrel out of the house; and (B) patience. With one bat, who flew along walls at 2″ from the wall, straight to each corner, then turned an abrupt 90-degree turn at the next corner, over and over, in perfect silence, it really took awhile. That bat took 30 minutes to notice that there was a door open along one of those walls (if the door had opened in, instead of out, it might have taken 1 minute).

When he finally flew through it, I pounced up and shut that door, preventing regression. I had been sitting there, drinking a beer, and tracking him as he flew these perfectly rectangular room circuits (not easy: bats indoors are flying about as fast as your neck muscles will allow your head to pivot).

Once he had found that first door, he seemed to sense that other open doors were good things, and flew straight through the remaining rooms out to freedom, bugs, and no doubt a well-earned nap. In classic coaching fashion, I had reached that tipping point where I had won enough trust to say goodbye.

Close Some Doors, Open Others

If you close doors to the inner part of the house (example: removing folks from their cubes and placing them in open workspaces, or not engaging in thermonuclear email exchanges about the New Agile Initiative),  you can invite clients to not worsen their situation.

If you open other doors and windows to the great agile outdoors, you invite clients to try inviting, healthy, addictively fun, cool new things (example: tweaking the card wall to reveal blockages or gaps in value flow; helping clients write their first few storytests).

Asking Gently Provocative Questions: Opening Windows in the Mind

By this, I don’t mean “Where the hell are all of your unit tests”?  (BTW, you know the question I REALLY don’t mean?  It’s this one:  ”Don’t you realize testability is vastly more crucial than encapsulation, you dolt?”)

I mean, instead, questions like “Has Rob paired with Lily recently?”  or “How did method-level cyclomatic complexity change this past iteration?”  or “How do you think that business verification went?” or “What do you think about the size of that test?” or “What do you think of that class name?” or “Did we see this retrospective improvement item in the last retrospective?” or “What do you really enjoy doing most on the team?”

If the Bird Blogs About How Cleverly I Let Him Out of the House,
I Probably Messed Up

The best of my personal coaches model for me, and counsel to me, the art of the client never discovering that the great idea was not originally theirs. Smart subordinates have been leading their managers from below in this fashion since the dawn of command and control, no doubt.

Not only is it OK, in a coaching position, to be a peer-to-peer influencer, as opposed to a top-down manager, it is typically preferable. People who I coach end up taking leaps of faith and trying scary new things, in good faith, not because anyone, including me, commands them to do them, or to try them. They do so because they trust me, like me, respect me, and happen to have had this cool idea pop spontaneously into their heads, unbidden.

The coaches I most respect, and most seek to work with, are those who help me as I refine my Ha and Ri in this coaching by question, coaching by seed trail, coaching by patient, engaged, gently provocative observation.

If you see me badgering someone to just shut up and do this Super Advanced Best Agile Practice Thing, please bust me on it. I will likely thank you (eventually, anyway).

My pal Abby Fichtner, the HackerChick, blogged today about emerging agile team leadership patterns and practices, as had been reinforced for her by a recent presentation by David Spann. She lists beliefs and behaviors that, according to Spann, when held by whomever actually manages an agile team, tend to help or tend to hurt. These resonated with me strongly.

Premise: It Ain’t Project Management

Meanwhile, coincidentally, I had been in conversations for the last week with stakeholders about how, in an agile transition, old patterns of Project Management are not a good fit. Traditional Project Managers are typically insufficiently team-focused and insufficiently portfolio-focused, while too top-down and imperative. Agile teams and systems have long lifecycles and broadening authority and responsibility that traditional Project Managers cannot typically see. With all due respect to Spann (with whom I think I agree nearly entirely), I would claim that, adaptive or not, neither the adjective “Project” nor the noun “Manager” helps the person trying to lead an agile team. The old culture and terminology are equally misplaced in an emerging agile world.

So, true, the agile community has not had great answers to questions about who leads and manages agile teams, much less how. So, last week, when I was asked by my best Pillar pal and mentor what the main responsibility, the main tools, and the guiding principle of agile team leadership might be, I realized, after some reflection, that I have been baking these in my head for awhile, semi-consciously. David Spann and Abby Fichtner helped me bake them a bit further.

Premise: It Ain’t Tech Leadership

The Agile Team Lead (for lack of a better role term) is not the tech lead. Many teams need formal technical leadership; many teams may grow out of this, into a pattern of entirely emergent tech leadership and architecture. In any case, the tech lead role is not the right role (nor is it typically, as Abby implies, the right skills match or personality-type match) for team leadership overall.

Premise: It Ain’t Scrum Master

As Abby claims at the start of her post, focus on methodology alone, even while there is frequently a temporary place for a certain style of such coaching, is again not the right focus for an Agile Team Lead. This is another focus that is too narrow, and too short. The team should outlive its Scrum Master by years; that may be a measure of a Scrum Master’s skill: the ability to leave the team in a cohesive and self-improving state. No, process coaching is not exactly agile team leadership either.

Look!  A Man with Nine Legs! Oh Bloody Hell, He Ran Away

We need something “completely different,” however rare, in our Agile Team Lead. Here are the things I now want:

Overarching Responsibility: Optimizing Flow of Business Value

In whatever ways a team improves: technical, interpersonal, process, practice, attitude, morale — we can measure and express them in how business value is delivered more and more efficiently, easily, steadily, skillfully, and (I’ve seen it) joyfully. The Agile Team Lead can trace all improvements back to how well the team delivers ROI and reduces code asset TCO, and back to how delighted the customer and team both are. The Agile Team Lead is in the business of knocking the customer’s socks off, over and over again.

Four Main Tools

The Agile Team Lead optimizes business value flow mainly with four varieties of tactic:

1. Continuous Team Building — this includes all of the soft, fuzzy means Spann lists, whose ends are team cohesion, high ambient trust, high ambient respect, high morale, good humor, equanimity, close interpersonal connection. In Jazz, musicians talk about a band having so much groove that everyone can read everyone else’s mind. Reinforcing that is the Agile Team Lead’s link inward toward the team (bear with my link metaphor here for a few more paragraphs).
2. Continuous Planning — for current purposes, I mean by this all of the agile practices and patterns around eliciting, analyzing, scoping, dividing, prioritizing, iterating, planning, testing, tracking, publishing, and delivering. The Agile Team Lead does not perform all of these practices solo, but leads their performance. This is how the Agile Team Lead is held to account by all managers and stakeholders outside the team. This is the outward link from the team to the world.
3. Continuous Unblocking — teams continuously need outside help, resources, more team members, fewer team members, and other emergent support that they, as a team, lack the authority to achieve. The Agile Team Lead does the heavy corporate lifting to get these deals swung, to keep the team in steady flow. This is the link from the outside world back to the team.
4. Continuous Improvement — The Agile Team Lead guides, without pushing, the zillion varieties of retrospective improvements that the team does own the authority to make. This does include holding the team accountable for owning the ever-changing lists of things to be improved, and concrete improvement proposals and plans. Note that this is not the same as coaching, which may need to happen separately. Teams had better outgrow coaching, while they are never likely to outgrow a need for leadership. These are the links from the team back to itself.

Guiding Principle: Servant Leadership

The leaders most admired in the agile community and software community, as in the world at large, are those who lead by giving and supporting. This is classically the root cause of a graduation from “good” to “great.”

And we are learning that it is simply the most effective variety of leadership and management: collaborative, listening, deeply present, deeply respectful, enthusiastic, inspiring, supportive, celebratory. And certainly trusted and adaptive, as Spann claims.

The Servant Leader attaches identity to the team’s abilities and accomplishments, not just to his or her own. Spann has captured several of the beliefs and concrete behaviors that we can only get from these leaders who lead to serve, not to rule. Indeed, an agile team’s Agile Team Lead must earn the team’s consent to be lead. Mature self-organizing teams eventually outgrow structural, imperative command and control, imposed by fiat.

Doesn’t it seem sometimes that the whole technical world is trying to go collaborative, trust-community, peer-to-peer? Aren’t these same patterns emerging everywhere, in different bits and bobs? Certainly it feels that way to me. Flat is the new hierarchy; village is the new city; conversation is the new document; trust is the new currency; small is the new big.

Thoughts?

That’s my sketch so far: the Agile Team Lead, through the lense of Servant Leadership, uses Continuous Team Building, Continuous Planning, Continuous Unblocking, and Continuous Improvement to optimize how business value flows through the team.

I am still baking this one; your feedback is appreciated.

In my rank procedural coding days, I might have written through-the-web-app-GUI test code that looks like this (we’re using TestNG here, though at first it may look like Junit 4):

@Test
public void canCreateRightTriangle() throws Exception {
    //selenium setup
    SeleniumServer jettyProxy = new SeleniumServer();
    jettyProxy.start();
    DefaultSelenium selenium = new DefaultSelenium("localhost",
        4444, "*firefox", "http://www.google.com");
    selenium.start();
    selenium.setSpeed("0");
    selenium.setTimeout("90000");
    selenium.open("http://virtualdevlabs.com/triangle/");
    selenium.waitForPageToLoad("30000");
 
    selenium.type("//*[@id='side1']", "3");
    selenium.type("//*[@id='side2']", "4");
    selenium.type("//*[@id='side3']", "5");
 
    String fieldText = selenium.getText("type");
    assertEquals(fieldText, "Right");
 
    //stop Selenium
    selenium.close();
    selenium.stop();
    jettyProxy.stop();
}

This code runs fine. And with a bit of probing (including reading comments), one can sort of tell what it does. It sets up some Selenium machinery, asks selenium to enter some values in 3 fields identified via xpath, then verifies that another field (looks like a text field) contains an expected string. It then tears down the Selenium machinery.

If one knows Refactoring in a procedural sort of way, which is to say, knowing enough to eliminate rank setup duplication within a TestCase class (to use the Junit term), one might then extract the blocks of setup and teardown code to appropriate methods, plus a single private helper method, leaving us, in the test method, with this:

@Test
public void canCreateRightTriangle() throws Exception {
    selenium.type(getFieldWithId("side1"), "3");
    selenium.type(getFieldWithId("side2"), "4");
    selenium.type(getFieldWithId("side3"), "5");
 
    String fieldText = selenium.getText("type");
    assertEquals(fieldText, "Right");
}

The Expressiveness Problem

Aside from the test method name, we still cannot much tell what this test actually tests on the page. And in this case, the test method semantics do not help us understand. All we have are the Selenium semantics for marching around the DOM: (“Put this value in this field; check that this other field’s value is such and such.”).

There is a very useful Domain Specific Language trying to emerge here. It’s the same one we use colloquially when we describe to each other how to use the web app:  “You go to the triangle testing page, then you enter the values for the sides of the triangle, and it immediately tells you what kind of triangle you have.”  So far our test expresses very little of that.

Identifying and Separating DSLs

In Java, Ruby, C#, or whatever, we can use  a little bit of new-fangled perspective on some good old-fashioned OOD. We can create  through-the-web-app-GUI test code that is reasonably extensible, and much more expressive. We do this by understanding the DSLs inherently involved,  by striving to keep them separate, and by carefully managing the dependencies between them.

Highest-Level DSL: Operating the Web App as a User

Here is a test method that accomplishes exactly what the one above does, but with very different semantics:

@Test
public void canCreateRightTriangle_2() throws Exception {
    String firstSideSize = "3";
    String secondSideSize = "4";
    String thirdSideSize = "5";
    assertTrue(trianglePage.canSpecifyTriangle(firstSideSize,
        secondSideSize, thirdSideSize));
 
    assertTrue(trianglePage.triangleTypeLabelReads("Right"));
}

It’s clearer at first reading what we are testing. The semantics used by the test method are those of the highest-level problem-domain DSL used by people when they talk about operating and testing the web application: “First check that we can enter the 3 sides for a Right triangle, then verify that the triangle label indeed reads ‘Right’.” The trianglePage object and methods accessible to the test describe the behavior of the web page involved. We might quibble about how much or how little of the behavior on the page has been verified, but the semantics are now at the right level of abstraction.

Second Level DSL: Operating the Controls on the Web Page

As with any OOD, this highest-level DSL encapsulates and hides details that are not useful in the test method code. Here is a peek at the second DSL layer that provides this encapsulation. Under the test method’s covers, there is a TriangleTestHomePage class:

public class TriangleTestHomePage extends BasePageContainer {
    public static final String TESTING_TRIANGLES_URL
        = "http://virtualdevlabs.com/triangle/";
    private TextField triangleSide2TextField;
    private TextField triangleSide1TextField;
    private TextField triangleSide3TextField;
    private TextLabel triangleTypeLabel;
    private TriangleImagePathTag triangleImage;
 
    public TriangleTestHomePage() throws Exception {
        goToPage(TESTING_TRIANGLES_URL);
        verifyTitleIsCorrect("Triangle Tester Exercise");
        triangleSide1TextField = new TextField("triangle_side1");
        triangleSide2TextField = new TextField("triangle_side2");
        triangleSide3TextField = new TextField("triangle_side3");
        triangleTypeLabel = new TextLabel("triangle_type");
        triangleImage = new TriangleImagePathTag();
    }
 
    public boolean canSpecifyTriangle(String firstSideSize,
            String secondSideSize, String thirdSideSize) {
        assertTrue(triangleSide1TextField().
            canEnterText(firstSideSize));
        assertTrue(triangleSide2TextField().
            canEnterText(secondSideSize));
        assertTrue(triangleSide3TextField().
            canEnterText(thirdSideSize));
        return true;
    }
 
    public boolean triangleTypeLabelReads(String expected) {
        assertTrue(triangleTypeLabel().reads(expected));
        return true;
    }
       //etc
}

I’ve omitted a few private methods for brevity and clarity. Here is our second-level page-control DSL, hiding inside methods  ”canSpecifyTriangle()” and “triangleTypeLabelReads()”: a TextField “canEnterText()”. A TextLabel “reads()” as expected or not. (Note: these are not distant third-party-library page control classes. They are our own facades; more on this below.)

Notice a few other things about this class. One is that our TriangleTestHomePage knows nothing about Selenium; it is mostly decoupled from it (we could easily sever the rest of the coupling with a PageControl interface).

Another point is that this page class is autonomous and self verifying, to at least some degree. It knows its own URL and navigates Selenium there. It knows its own expected page title, and the base BasePageContainer class contains a verifyTitleIsCorrect() method that asserts that the actual page title matches the expected one passed to it.

Third Level DSL: Selenese for Manipulating Page Controls

Here is the TextField class:

public class TextField extends PageElementWithIdAttribute {
    public TextField(String id) throws Exception {
        super(id);
    }
 
    public boolean canEnterText(String entry) {
        selenium.type(id, entry);
        return true;
    }
}

Within the canEnterText() method, we finally begin to speak in the semantics we began with in the first procedural example up top: the down-and-dirty “selenese” of walking the DOM, changing its state, and verifying that state: selenium.type(id, entry). In the real world of web app testing, this selenese is frequently fraught with xpath and regex ugliness. However pretty or ugly, we want such low-level semantics as far from the test method semantics as possible.

Summary: Three DSL Tiers

We end up with a hierarchy of three little DSLs:  a test method expresses itself in the DSL semantics of web page operation from a user’s standpoint. It does this using facade-like classes that contain methods describing page-specific operations. Some of those page methods are on the page classes themselves, while others are on an underlying BasePageContainer class.

The page classes contain methods that in turn express themselves in the DSL semantics of page controls, such as typing in entries (and verifying that such operations work), and checking the values of labels.

Finally, the methods on those page control classes (TextLabel, TextField) express themselves in the nitty gritty SeleniumRC selenese semantics of traversing and manipulating a page DOM, and checking its state.

Each of these DSLs can and should be organized into their own package hierarchies. And while we might start with simple 2-tier object hierarchies per package/DSL, they might easily grow new tiers. (Groups of wep page facade classes, might, for example have some duplicate portlets that deserve an intermediate class).

Why Bother, For Crying Out Loud?

The extensibility that all of this buys us can be the difference between a test suite that gets maintained, and one that gets abandoned. With these three tiers of DSLs, we can have very expressive, succinct web page flow tests that contain the least duplicate code, and are readily extended. The web page classes and page control classes under the covers can be just as clear about their page-control-level operations, while they hide details and squeeze out duplication.

Our test methods can be concise even with very lengthy page flows. We need not duplicate any selenium code, nor anything but actual page flow operations (which is unavoidable in such tests). So let’s now look at this in the context of some non-toy code.

Meanwhile, in the Real World

I am currently evaluating the following test code. Based on our above guidelines, what condition do you think it’s in?

public void testSimpleAuthorSearchTestCase() {
     int searchResultIndex = 1;
 
     assertTrue(openAdvancedSearchPage());
     typeSearchTerm(authorName);
     selectSearchType(searchType);
     search();
     assertTrue(verifyResultsAreReturned());
     storeSearchResult(searchResultIndex);
     assertTrue(verifySearchResultDataNotEmpty());
     openSearchResultDetailedView(searchResultIndex);
 
     assertTrue(verifySearchResultTitleMatchesDetailedViewTitle());
     assertTrue(verifySearchResultPubMatchesDetailedViewPub());
     assertTrue(verifySearchResultDocumentTypeMatchesDetailedViewDocumentType());
     assertTrue(verifyAuthorInDetailedView(authorName));
     assertTrue(verifySearchTermIsFoundInDetailedView(authorName));
     assertTrue(verifySearchTermIsHighlightedInFullTextInDetailedView(authorName));
 }

So these private helpers are more expressive than selenese, granted. But where do all of these private helpers end up living? In a single web page TestCase “God class”?  An unrelated set of such classes? In a bit of TestCase object tree? This particular app has several more pages, and more kinds of expected and actual values to be compared to each other. Will all of the underlying selenese be duplicated?

With this design paradigm, even with a bit of object tree here and there, we still have a more elaborate version of the refactored procedural example we began with up top. We’ve squeezed the selenese out of the test code, but we are still conflating DSLs. We still have duplication down in the details, and the objects in play are not nearly as expressive and extensible as they could be. We are still on a slippery slope.

I’ve been refactoring this test method, while stubbing out the supporting DSLs it needs to be expressive in the way we have been discussing. Here is what I have so far:

@Test
public void testSimpleAuthorSearchTestCase_UsingDSLTiers() {
    AdvancedSearchPage searchPage = new AdvancedSearchPage();
    AdvancedSearchResultsPage resultsPage =
        searchPage.search(searchType, authorName);
    assertTrue(resultsPage.verifyResultsAreReturned());
 
    int searchResultIndex = 1;
    SearchResultDetailedView detailedView =
        resultsPage.getResultsDetailedView(searchResultIndex);
    assertTrue(detailedView.allExpectedResultsMatchUp(authorName));
}

And for example, my first stub pass at the SearchResultDetailedView class looks like this:

public class SearchResultDetailedView {
    private TextBlock documentTitle;
    private TextBlock documentType;
    private TextBlock publication;
    private TextBlock actualAuthorName;
    private TextBlock searchResultTitle;
    private TextBlock searchResultPublication;
    private TextBlock searchResultDocumentType;
 
    public SearchResultDetailedView() {
        actualAuthorName = new TextBlock("ReplaceMe");
        documentTitle = new TextBlock("ReplaceMe");
        documentType = new TextBlock("ReplaceMe");
        publication = new TextBlock("ReplaceMe");
        searchResultTitle = new TextBlock("ReplaceMe");
        searchResultPublication = new TextBlock("ReplaceMe");
        searchResultDocumentType = new TextBlock("ReplaceMe");
    }
 
    private boolean searchTermIsFound(String searchTerm) {
        // TODO Auto-generated method stub
        return false;
    }
 
    private boolean searchTermIsHighlightedInFullText(
            String authorName) {
        // TODO Auto-generated method stub
        return false;
    }
 
    public boolean allExpectedResultsMatchUp(
            String expectedAuthorName) {
        assertTrue(documentTitle.equals(searchResultTitle));
        assertTrue(publication.equals(searchResultPublication));
        assertTrue(documentType.equals(searchResultDocumentType));
        assertTrue(actualAuthorName.equals(expectedAuthorName));
        assertTrue(searchTermIsFound(expectedAuthorName));
        assertTrue(searchTermIsHighlightedInFullText(
            expectedAuthorName));
        return true;
    }
}

So this could doubtless use another pass or two; and lacks some actual implementation here and there. And our TextBlock class would likely need its equals() method overridden, or would require a matches() method of some sort. Its entire semantics might need more thought. How much expected state should be passed all the way from the test method? Good question.

But we are at least making steps in the right design direction. Our test method now uses a user-centered DSL that reads like this: “From the advanced search page, search for this author name using this search type. On the resulting advanced results page, verify that all returned actuals match expected values.”

And as we push down from there, we evolve the underlying DSLs, keeping them SRP-compliant and DRY.

Web App Test Code is OO Code Too

Most quasi-procedural functional test code mixes up at least three DSLs. In many situations, you’ll have more than four DSLs in play. For example, for one recent team and app, the customer wanted a BDD-style FitNesse test page that ran green when a large tradeshow conference demo page flow had successfully been verified by SeleniumRC. The FitNesse fixtures used a “given/when/then” semantics that were their own DSL, and had to be crafted to reuse two of the other DSLs shared by TestNG. It turned out that all of this was fairly straightforward to introduce in a way that was extensible and expressive, and DRY.

We want to minimize the Total Cost of Ownership(TCO) of our test code, just as we do for any other code asset. A couple of short examples may not show it, but mixing these DSLs eventually guarantees duplicate code, lack of test clarity, lack of problem domain clarity, lack of expressiveness generally, with every new TestCase and test method. This inevitably increases our TCO for that code.

Whether you are an agile tester, or a programmer with agile testing responsibilities, and whether you are using Selenium or HtmlUnit or WATIR or RobotFramework or whatever, learn to recognize the DSLs you are given by your tools and your problem domain, and the testing DSLs you are creating. Keep them separate to keep your code DRY, expressive, and extensible. And have fun. Outside the hellish realms of QTP and VBS, automated functional test code can be enormous fun.

Note: you can play with code very similar to the first examples above, by checking it out from its google code repository using any Subversion client, here. The codebase is an exercise we use to hire potential agile testers, as a first gate to measure their coding ability. You can read more about that here.

Premise: No, Software Dev Ain’t Like Manufacturing

So it’s pretty old news that software development is not manufacturing. The reason many of us have questioned manufacturing metaphors  is that software development involves inherently much more variation. Compared to a factory line for a sports car, building the average corporate CRUD web application (not to mention much more interesting app types) is just a completely different animal.

Once a factory line for a sports car is designed, built, tuned, and humming along, it includes no equivalents of test-driving an object model, or storytest-driving requirements. In real manufacturing of sports cars, using approaches like Lean, you can reduce away nearly all of your variation in the actual manufacture. You want to eliminate muda. You can get to a point where the exact steps required to build any one of your sports cars are very nearly identical. This kind of perfectly repeatable, fully-automated process is just not possible for converting feature requirements into actual running, tested features. Sports cars and software system features are completely different animals. Fair enough.

Yet We Do Eliminate Wasteful Variation & Complexity in Software

Nevertheless, a lot of what we do in the world of agile development, and agile coaching and mentoring, is exactly this: reducing or eliminating wasteful, useless variation. Things that complicate our lives unnecessarily, that make the whole system less repeatable, predictable, and more expensive. A fleet of best practices and their literature, I would claim, are about exactly this: squeezing the wasteful variation out of our development.

This doesn’t make us mechanizing, reductionist, draconian Taylorists. We are not being less humane, but more, when we squeeze out wasteful variation. The difference between humanizing and Taylorizing turns on being able to distinguish between useful and unavoidable variation/complexity (things that can be crafted) and useless and wasteful variation (things that really can, and should, be simplified or automated).

Indeed: the more of the latter, wasteful sorts of variation we eliminate, the more freedom we buy for ourselves to really shine in those areas where complexity and variation must be crafted.

So what are the kinds of variation that are unavoidable in software development, and what are the kinds of wasteful variation we always want to consider eliminating? Here are my initial lists. I am trying to launch a discussion here, not finish it, so please comment. As usual, I reserve the right to revise this blog post to include smart observations made by commenters and pals.

Necessary Variation Types

When people like Pete McBreen write books like Software Craftsmanship, and people like Eric Evans write books like Domain Driven Design, much of what they are talking about are the varieties of variation and complexity in software development that are unavoidable. The kinds of things that must be crafted, that unavoidably require skill, discipline,  experience, and even trust-community, passion, autodidactic reflex, and courage. These kinds of inherent variation include (but are not limited to):

Requirements Variation

Yes, from a Portfolio Manager or Product Manager’s perspective, it often makes sense to master the art of saying No to requirements. Joel Spolsky, for example, has written wisely about the nature of the “Consultingware” that results from essentially never saying No.

Nevertheless, we have no real control over the kinds of requirements pressure the market will bring to us, and to which our Product Manager might sensibly say Yes. You cannot, in 2002, anticipate rich-client web app behavior that your hand-crafted Web 1.0 application might be “required” by the market to suddenly include. A great deal of requirements variation is not only unavoidable, but wonderfully necessary: this is how, as an industry, we innovate new value flows.

Object Model Complexity

Yes, we want to keep our designs as simple as possible. Yet, over the course of 10 releases and thousands of lines of code, even in the best factored, best test-driven systems, there will be plenty of unanticipated variation and complexity in the object model. This is the problem to which we apply as much Software Craftsmanship as we can. We try to keep the order of complexity of our solutions relatively in line with the order of complexity of the problems they model, release after release. It is very complex to learn to write very simple code. And there is no avoiding it, if the goal is lowest TCO codebase assets, highest-velocity value flow, and good clean fun.

“Given” Technology Stack Variety

I’m not talking about technology/framework selection, but about instead how, for a given set of such selections, some amount of variation comes along. If your team is required to keep working in Java 5 /Spring MVC /Hibernate /Oracle 10g or whatever, then bang, you have a wide variety of syntactic and semantic variations that you cannot avoid, and which, to some extent, you must master.

Team Dynamics

People get hired and get ramped up. People go on vacation and get sick. People quit and get promoted and get fired. Many team membership changes are unavoidable and inherent. Some variation in team membership, experience levels, etc, are unavoidable. (But see below on the importance of optimizing that variation.)

Wasteful Variation Types

OK, here is where I start to enjoy myself. I get to list the kinds of problems I keep trying to solve, better and better. The ways we can improve. Again, this is a draft, partial list of main categories. Feel free to help me flesh it out with your comments.

Non-Deterministic, Imprecise Scope

Just because we cannot control how much requirements variation might arrive at our doorstep does not mean we cannot be precise about describing, planning, and testing a given new requirement’s completeness and robustness. Much of the storytest-driving, acceptance test-driving that agilists push for is about exactly this: given any wacky new requirement, we ought to be able click on a test and have it turn green when we are done-done-done with it. No, this is not easy or cheap. But it is cheaper than continuous scope-related creep, misunderstanding, and kindergarten blamestorming (“It’s a bug!”  “Hell no, it’s a feature!”).  So Yes, we can squeeze out lots of useless variation here, and we do.

Manual Build and Deployment Steps

My pal Mike Hill has blogged hilariously about the courage to automate away manual build voodoo, what he calls Jiggling Toilet Handles. The manual voodoo is classic useless variation. Yes all the little manual steps might be difficult to completely automate away, end-to-end. Doing so might require courage, discipline, and after-the-fact forgiveness. Nevertheless, every time we automate away some dumb, non-repeatable, expensive manual handle jiggle, we start to reap immediate, repeatable rewards. Others who have been fighting this good fight for years include the Pragmatic guys, with books like the original Pragmatic Programmer, and Mike Clark’s lovely Pragmatic Project Automation. Indeed, the entire practice of Continuous Integration attacks this category of wasteful variation.

Manual Regression Testing

Manual regression testing is a classic Sysiphian struggle, wrapped in a Faustian bargain. Your manual test-recorders or manual test plan executors are cheap per hour?  An indescribably false economy! No manual QA team can ever really hope to catch up or keep up. The untested items and defects pile up. Trust erodes. This, truly, is the canonical thing we can automate away, with one or more tiers of automated testing, most critically programmer-level TDD. Squeezing out this variation yields gigantic benefits. Skillful automated testing may well be the most critical contribution to software development of the last 15 years.

BigBallofMud Code

Every un-test-driven, unrefactored codebase eventually gets muddy, then really annoying, then unmanageable, then completely toxic. As a code asset deteriorates, the cost per new feature goes up dramatically in that codebase.  Yes, as we mentioned above, some complexity in a codebase is inevitable. But most complexity in most codebases is entirely useless and wasteful, and about as avoidable as the lung cancer deaths traced to smoking. Courage and discipline and skill and several specific practices are required to produce Clean Code.

Conveyance Muda

As a team, we knock down our cubie walls, we sit in the same open space, we pair, we have standups, and we write story tests (and programmer tests, for that matter). As a team, we continuously estimate, we continuously plan, we continuously retrospect. We do these things for several reasons, but they all have the partial effect of reducing the cost of getting some knowledge or work product from one person to another, from one state to another. These practices make all manner of conveyance vastly cheaper.

The alternative mass-creation of Word, Visio, Excel, and similar artifacts, then firing them in volcanoe-ash volleys over walls between our silos (using emails with dozens of cc’s!), then discussing them vaguely at large meetings, must surely be orders of magnitude more wasteful. It has repeatedly seemed so to me and my colleagues.

Wasteful Skill-Level Variation

Another silo thing. The lottery number: if you were the Nuxeo/Liferay/Maven guy, and none of the rest of us knew that stuff, and you get “hit by the lottery” and quit, how much money is wasted recovering/reconsistuting/reverse-engineering your knowledge?  Ouch!  We pair, and sit together, and collaborate closer and closer on a story’s definition of done, to squeeze out the useless variation in who knows what and who can do what.

Context-Switching Thrash

If you move programmers too frequently from one team to another, where they must struggle to learn whole new (to them) problem domains, solution domains, technology stacks, local customs, it is much like moving all the cab drivers from Miami to New York, and from New York to Miami. How long will it take before anyone gets to the airport on time?

No, you should probably not keep anyone on any agile team forever. But teams should mostly persist. If you let them persist, self-organize, and continuously improve, and if you mostly bring work to teams, instead of people to work, then you save all kinds of thrash cost, and you gain hard-to-describe production economies from new levels of passion, craft, collective courage, and overall quality and throughput.

OK. Enough for one session. Below are categories I have not yet fleshed out. I can and will. Meanwhile, what other kinds of main categories have I missed?  Where can I/we collapse categories together?  How useful is this principle of wasteful vs necessary variation useful to you, as a crude pattern language?  Lemme know, peeps and tweeps.

Poor Value Flows

Administrivia and Bureaucracy

Over-Engineering

Unnecessary Framework Complexity/Waste

Unnecessary Optimization

Simple “Clean Code” Metrics for C-Level Execs

A recent Twitter thread I was involved in goes something like this. Someone claimed that  software managers and executives should not have to care whether their developers are test-driving Clean Code.  They should be able to presume that their developers are always courageous, disciplined, and skillful enough to produce that level of quality. Ultimately that quality turns into least Total Cost of Ownership (TCO) for any codebase asset.

Sigh. Well, would that that were true. To my mind, it’s like saying that all consumers should presume that all cars are built as well as Toyotas. Sadly, they are not.

So, yes, of course, developers should take full ownership of the extent to which they create  Clean Code. Developers should own their own levels of skill, discipline, knowledge, passion, and courage. Absolutely so. Developers should refuse to cave to pressure to hack junk out the door to meet deadlines. That’s not what I am debating. I am debating whether or not the industry has accountability systems and quality monitoring systems to ensure that developers are in fact doing all of that.  My premise is that something like the opposite of that is going on.

If managers and developers are still being rewarded for hacking junk out the door, and executives and managers cannot and do not measure the TCO consequences, well then no wonder our culture of Software Craftsmanship is not spreading. We have a crappy incentive structure.

Would Toyota have surpassed GM in sales if no-one had detected how much lower the TCO is for the average Toyota?  Of course not. The car industry does, in fact, have accountability systems in place to measure asset quality, duty cycles, TCO. Too much money is at stake.

As a responsible car buyer, I inform myself with exactly the least info necessary and sufficient to determine whether I am about to buy a great car or a lemon. I have data to help me predict the outcome.

Managers and executives in software cannot expect that every codebase is as good as a Chevy, nor even a Yugo. Most enterprise software these days, 10+ years into an agile movement and Software Craftsmanship movement, is still gunk that continuously deteriorates.

And managers and executives cannot see that. They are buying lemon after lemon, not knowing any better.

We want managers of developers to insist on Clean Code, so we want them to be able to tell the difference between Clean and Mud, and to hire programmers who code Clean. And we want executives to hire managers like that. These inevitably will be executives who can distinguish between managers who know Clean Code and those who do not. I posit that these executives will in turn need to know how, at a very basic level, to distinguish between Clean and Mud. Only then can they preserve their asset value, and hire delegates who can.

Two Metrics You Can Use to Measure Code Asset Deterioration

At my current client, each team self-governs several kinds of objective and subjective Clean Code quality measures, including test coverage, cyclomatic complexity per module, average module size, coupling, etc. There are all kinds of details here around issues like automated deployment, test quality and semantics, etc. They don’t publish it all, they use most of it tactically, within the team boundaries, to hold themselves and each other accountable for continuous code improvement. The teams can and should own that stuff, and they do.

But you know what?  Each of these teams is also publishing at least two metrics to other teams and straight up the management chain for their codebase assets: test coverage and cyclomatic complexity per method.  The Continuous Integration plugins publish these metrics for all to see. And all any team is held accountable for is this: do not let these numbers slip between iterations. Anyone can see historical trend graphs for these numbers for any of the projects/codebases currently covered (there are several so far, and more each month).

Yes, these two measures are imperfect and can be gamed. Yes, test coverage is a one-way metric. But let’s presume for a moment that we are not hacking the coverage config to exclude huge whacks of yucky code, and we have good-faith participation on developers’ part. If average complexity per method goes from 4 to 6 over a two-week iteration, and if test coverage slips from 80% to 60%, does that not often mean that the codebase, as an asset, probably deteriorated?  My experience has been that it does.  As an owner of such an asset for which numbers had slipped like that, would you not care, and would you not want some answers?  I would, and I counsel others to care and dig in. I hereby counsel you, if you own such assets, to care if those two numbers are slipping from week to week. If they are, I bet you dollars to donuts your software asset is getting toxic.

A Culture of Accountability

So at this client, if those two metrics slip, teams hold each other accountable, and execs are learning to hold dev team managers accountable. Why not? Every car buyer understands MPG these days. Why not educate executives a little bit about how to monitor their code asset health?

Could there be a better 2 or 3 metrics to publish upstairs?  You guys are smart; you tell me. So far, these 2 are working pretty well for me. The published metrics are not sufficient to protect against asset deterioration, but so far they sure seem necessary.

So guess how this is turning out?  We are growing a top-to-bottom, side-to-side culture of Clean Code accountability in what once a garden-variety, badly-outsource-eviscerated, procedural-hacking sort of culture.  Partly by hiring lots of Been-There-Done-That agile coders, and partly with these metrics. Suddenly, managers who were only measuring cost per coding-hour (and slashing FTE jobs to get to LOW $/hour numbers) are measuring more meaningful things. Could we do better? Doubtless. Stop by, see what we are doing, and help improve it.

What metrics would you publish up the reporting chain, and between teams?  How would you help executives detect when their code assets are slipping into that horrible BigBallofMud state?

Speak up, all you shy people.

Background: Scrummerfall

Brad Wilson and others have blogged about a persistent “scrummerfall” pattern in agile transformations wherein, despite good-faith efforts on the part of teams to blast away cube walls, silos, and artificial role divisions, we still have programmers and testers working in phases. I’ve seen this repeatedly myself. Each little iteration is (at best) a two-phase waterfall project: programmers test-drive story implementations in isolation, then toss the stories over a tiny wall to testers in the same room, who write automated tests and do manual testing. And the testers find bugs not caught by programmers, in all the little nooks and crannies between systems, projects, frameworks, architectural layers, web service end points, yadda. This is not the only way in which crappy conveyance muda and role preservation shows up in iterations, but it is typical, and fresh in my mind.

Note that I am not arguing against the official role of tester on agile teams. To the contrary, I think the agile community at large tends to undervalue and misunderstand the value testers bring to agile teams and software development generally.

And on a side note, we could argue here about how useful non-isolation-testing is or is not, when we are supposed to be test-driving all of our code. We could argue about whether the need for automated integration tests are a measure of inadequate test-driving skill (which I do not actually believe).

But let’s not argue. Let’s instead agree that regardless of the story, its definition of done should include testing on which programmers and testers do some pairing. The story is not done unless programmers and testers have worked together on all the non-isolation testing that needs done. In a web app, this might include SeleniumRC or Watir testing. In any sort of app, it might be Story Testing of the kind we could do with Fit or Slim or RobotFramework or JBehave or EasyB or whatever. Let’s then also agree that our programmers should be learning more about testing generally, and we should be hiring testers who can program.

Most Good Agile Testers are Good Agile Programmers

Let me be more plain. Too few software testing professionals can program well. Agile Testing requires that most of its agile testers be programmers and, in my experience, pretty darned good ones. So, decent object modelers, pretty creative manual-step-automaters, pretty good build system massagers. Agility demands that testers evolve in the direction of solid, real-world development skill.

Hire Testers Who Are Good Programmers

So, please, stop hiring testers who can only do record/playback testing, manual Word doc test plan creation, etc, and have no programming ability. We have plenty of data now that whether your testers are in-house or off-shore, whether they have 2 years or 20 years of experience, most of them are going to need to be able to actually code once they join your agile teams.

This is not to say that programming is the most important or central skill a tester possesses. As Elisabeth Hendrickson recently commented, “Seems to me that professional testers do 4 things well: observe, notice what can be varied, predict risks, & explore methodically.” I agree. She and I and others wish to grow those skills as well.

But again, programming is one of the things good testers do well. And it is a means toward the above ends of noticing what can be varied, of predicting risks, of observing and exploring. And when not a means toward those things, it is an adjunct skill.

And in order to determine whether or not an agile testing candidate can actually code, you will need to test that candidate’s level of skill, using some kind of programming exercise.

And, presuming you are working in Java, I can help you with that.

More Background: an Agile Developer Coding Exercise

A bit of context: At Pillar Technology, I created a TicTacToe coding exercise (available on google code) for candidate agile Java programmers a couple of years ago. It has evolved and evolved, and we have seen several dozen programmers thrown through it. Those of you who know it, know, as I repeat constantly, that it has its flaws. Further, this is not the only hiring tool or technique we use, though we do try to use it for every programmer.

And I tell you what. Our ability to hire programmers with actual skill and experience in TDD, Refactoring, Simple Design, Clean Code, Design Patterns, and Software Craftsmanship generally has skyrocketed simply by throwing every candidate at this little exercise, and by having the right little community of Pillar programmers evaluate the code that the candidates write. And our cost per programmer hired has plummeted.

Where we had very little determinism and precision before, we now have some.

Finally: a Draft Agile Tester Coding Exercise

It turns out we also needed the ability to test how well agile testing candidates can code, at least in Java, in a particular context (Selenium RC web app GUI testing). I’ve now created another little google code project, this one for testing how well agile testers can write Java code for Selenium RC tests. It’s a very first draft (as of this writing), and it will need plenty of love and evolution and crusty criticism of the kind the other codebase got. I hereby solicit any and all, friendly or not. I just don’t want to get into conversations about whether this kind of automated testing is useful in the first place, since my operating premise is that it is.

The exercise currently uses as its System Under Test a fledgling Triangle Testing website under development by Elisabeth Hendrickson, to whom I owe thanks. Thanks, Elisabeth! As her triangle site SUT evolves, so may my codebase.

Anyone who you want to put through the exercise will need to be instructed to download and install both Eclipse (for example) and a Subversion client (like the Subclipse plugin for Eclipse). You’ll then need to instruct them to check out the coding exercise, and then finally to open and read the README file.  My intention is that from there on out, the exercise instructions are clear, starting with instructions for downloading and installing the TestNG plugin for Eclipse.

If you would like some instruction in how to interpret and assess the code your candidates write, please contact me separately.