What we Want: Expressive, Succinct Tests

Let’s say I want to test that I can log into an app. I want my Selenium RC Java test code to look something like this, because I want it to read, aloud, the way I would describe actually logging in and checking that everything went mostly OK:

@Test
public void canLoginToFatFreeCRM() throws Exception {
  LoginPage loginPage = new LoginPage();
  DashBoard homePage = loginPage.login(BrowserDriver.DEFAULT_USERNAME,
                          BrowserDriver.DEFAULT_PASSWORD);
 
  assertTrue(homePage.isLoaded());
}

Several techniques make this possible, but the one we will focus on in this blog post is the notion of a self-verifying PageObject.

Note that this test knows nothing about Selenium. Indeed, neither does the LoginPage class (below). More on that later. Note that this test does not include any waitUntilPageIsFrickinLoaded() calls, nor (worse) Thread.sleep() code, and very little direct instantiation of PageObjects. The page flow mechanics is encapsulated elsewhere. Note also that as I go from page to page, I don’t explicitly assert that I got there. I shouldn’t have to. My test should be able to take that for granted. It turns out that assertions are being made under the covers about having arrived successfully on intermediate pages, but that code is abstracted away. That’s what this post covers.

The only useful assertion in the test proper is that, once I have logged in with proper credentials, I arrive safely at the homepage (and even this assertion in the test is redundant, and included here only to reveal intention; the DashBoard page is also self-verifying).

Glance-Readability

A test as short and expressive as the one above passes the “glance readability” test. Anyone familiar with automated testing semantics, and the biz language of this CRM domain, ought to be able to grasp what this test does at a single glance.

Suffice to say, we want our Selenium RC tests, as much as they can, to be this succinct. And, again, that requires that they stick to expressing page traversal and manipulation in the local biz-specific language of the web app’s business domain. So, finally, let’s see the under-the-covers page self-verification mechanics.

PageObjects as I Use Them

The selenium-rc-patterns sample Java codebase that illustrates all of the patterns in this series of posts talks to a localhost copy of a Rails project called Fat Free CRM. If you want to learn a bit about this app, you can play with a hosted sample version here (you have to sign up for an account first, which is easy).

This CRM system has discrete web pages like a Dashboard (home page, essentially), and pages for Tasks, Leads, Campaigns, Contacts, Accounts, and Opportunities. For each of these actual, production pages, my selenium-rc-patterns Eclipse project contains a matching PageObject with methods and fields that provide access to the services on that page.

Above, when you ask an instance of a LoginPage to login(username, password), then the TextFields and PageLink type know how to return us a PageObject that should then be cast to a DashboardPage and returned. (More on that in another post.)

Selenium RC developers have been using a PageObject pattern for awhile. The pattern dates back, at least, to the original HTMLUnit, whose API rather strongly encourages you to represent your pages under test as extensions of what it calls an HtmlPage. I don’t use Se 2 yet (might someday, might not), so I don’t use its PageFactory pattern.

Instead, I use my own PageObjects that extend a BasePage (or BasePane), partly so that I can control how and when PageObjects are instantiated, and verify automatically and ambiently on instantiation that Selenium is indeed on that page. Again, that’s much of what keeps the above test code so simple: I’m not explicitly waiting for a page to load, and I’m not asserting all over the place that I’ve successfully arrived on a given page.

So, my LoginPage class has a login() method that accepts a username and password (we’ll cover the return types of these fields and methods in another post that describes how page-flow works):

package bizdomain.pages;
 
import util.BasePane;
import util.elements.PageLink;
import util.elements.TextField;
 
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS
                                           = "input[id=authentication_username]";
  private TextField userNameField;
  private TextField passwordField;
  private PageLink loginButton;
 
  public LoginPage() {
    super();
    userNameField = new TextField("input[id=authentication_username]");
    passwordField = new TextField("input[id=authentication_password]");
    loginButton = new PageLink("input[id=authentication_submit]",
                                                                                DashBoard.class);
  }
 
  @Override
  public String getPageLoadedCssLocator() {
    return PAGE_IS_LOADED_CSS;
  }
 
  public DashBoard login(String userName, String password) {
    userNameField.enter(userName);
    passwordField.enter(password);
    return (DashBoard) loginButton.clickToNewPage();
 
  }
}

Verifying That the Production Page has Been Loaded

The real point of my flavor of PageObject pattern is the self-verifying bit. Let’s dive into that — not later, but now.

Note that each of these PageObjects extends BasePane, and has a PAGE_IS_LOADED_CSS constant:

package bizdomain.pages;
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS = "input[id=authentication_username]";

In the LoginPage constructor way above up there, you can see we first explicitly call super() on BasePane. Here is BasePane:

public abstract class BasePane {
  public BasePane() {
    waitUntilLoaded();
  }
 
  public final boolean isLoaded() {
    return BrowserDriver.isElementPresent(getPageLoadedCssLocator());
  }
 
  public final boolean isVisible() {
    return BrowserDriver.isElementVisible(getPageLoadedCssLocator());
  }
 
  public final void waitUntilLoaded()  {
    BrowserDriver.waitForElementVisible(getPageLoadedCssLocator());
  }
 
  public abstract String getPageLoadedCssLocator();
}

You can see that this constructor calls waitUntilLoaded(), which makes a static call to a method on our BrowserDriver (the Facade / Decorator that handles all of the actual Selenium calls) in order to loop until our LoginPage (in this case) actually is loaded. The argument supplied is the result of calling getPageLoadedCssLocator().

But wait! That’s an abstract method! So Yes, we have something very like a template method pattern here in waitUntilLoaded(): the concrete implementation of getPageLoadedCssLocator() on LoginPage returns that PAGE_IS_LOADED_CSS constant String.

Deep, deep under the covers, the BrowserDriver.waitForElementIsVisible() method looks like this:

public static void waitForElementVisible(String cssLocator)  {
  for (int second = 0;; second++) {
    if (second >=
 Integer.valueOf(BrowserDriver.STANDARD_DHTML_LOAD_WAIT_  TIME) / MS_PER_SECOND)
 
  fail("Timeout waiting for element " + cssLocator + " to become visible.");
    if (isElementVisible(cssLocator)) break;
    sleepForASecond();
  }
}
 
public static boolean isElementVisible(String cssLocator) {
  injectJqueryIfAbsent();
  return executeJavascript(JqueryCodeFactory.getVisibilityCode(cssLocator)).equals(TRUE);
}

We’ll discuss the BrowerDriver class at length elsewhere. And we’ll also discuss the injectJqueryIfAbsent() method, which we hope an upcoming Selenium RC release will obviate.

The upshot of the waitForElementVisible() method and the LoginPage and BasePane code above is that the LoginPage object will not successfully finish instantiating until Selenium can successfully verify that a unique element on that page has been loaded. In other words, once a LoginPage instance is loaded in memory, we actually are on the LoginPage, by definition, as long as that CSS element selector syntax is correct. Voila, automatic, ambient page flow assertion.

This is much of how we get our test methods so succinct. The PageObjects take care of verifying for us, at instantiation, that we have safely arrived on their corresponding production app web pages.

Caveat Lector: there is a flaw in my code I have yet to squeeze out:  duplication between a BasePane, used above, which I usually use for dynamic changes in the HTML, and a BasePage, which presumes that a real HTTP Request/Response cycle has occurred. I will collapse those two together shortly.

Next: we’ll talk about reusable HTML Element objects, and how the linkish ones know, in my code, how to return the PageObject we wish to traverse to next.

I’ve blogged in the past about low-TCO (Total Cost of Ownership) Selenium RC testing in general, and about  how much of your automated testing budget should be allocated to different kinds of testing at different points in an agile transformation. Of late, I have been helping teams learn to use Selenium RC, in Java, to test web apps through their GUIs. Once the above Caveat Tester is spoken, I help them do the Selenium RC testing that they feel they MUST do as well as possible.

I have been diving more deeply into it for a client recently, and finding and refining some useful patterns I had not needed or found before (with the help of several colleagues). And, admittedly, some of what I am doing is refining my old patterns in response to things like Refactoring pressure. In general, I am devising increasingly DRY Object Oriented Selenium RC Java patterns. I recently presented on them at Agile 2010.  Using your subversion client, you  can checkout the code that illustrates these patterns in a Google Code repository here.  You can find the subversion check-out instructions here. You can find my slide deck from Agile 2010 here.

I intend to share each of these patterns with you, each in its own blog post, in coming weeks.

So watch this space for blog posts (with code, and links to a sample project on Google Code), covering the following topics:

• Clean, clear test methods
• Separating DRY test-framework code from “wetter” tests, and separating reusable, generic test-framework code from biz-domain-specific framework code
• Wrapping Selenium and other frameworks in Facade/Decorator that provides static Singleton test framework access.
• Pushing all framework static Singleton references down into base classes and html element classes.
• Biz-Domain-specific Page Objects that are “lazy loaded” and self-verifying, and that encourage Dry OO test code
• Always using CSS as a DOM element locator strategy; never using XPath.
• Injecting the entire jQuery framework into pages under test, in order to verify visibility of elements that appear after dynamic html or Ajax calls.
• Generic HTML Element objects (ala HtmlUnit’s API) that encourage Dry OO test code
• DhtmlLink and PageLink Factory classes that are instantiated with the Page Object classes they should “go to,” and that automatically return those objects when “clicked”.
• Pushing out access to common elements across several pages to static Singletons on base page classes.

The code I link to above illustrates all of these patterns, expecting (for now) to run against a localhost instance of a Rails app called FatFreeCRM, which is straightforward to download and install. I intend to publish a different example soon that points to, for example, Wikipedia.

The recent claims made by a well-known agile coaching thoughtleader notwithstanding, I work hard to get clients to adopt real Storytesting practices, with real Storytesting tools (FitNesse is still my tool of choice; I work mostly with Java teams). I will continue to do so. I find no false economy or Faustian bargain with FitNesse tests, and I suspect it is because I am coaching the use of them differently than James Shore is.

Manual Regression Testing = Really Bad Thing; Agreed

Regression testing of any kind is classically about proving that we are Building the Thing Right. For true regression protection purposes, I want manual regression tests to be replaced with a requisite blend of automated tests (using the testing triangle pattern for allocating test automation resources) plus a requisite amount of manual exploratory testing.

Whoa Nelly: Storytests Are Not About Bugs

But Storytesting / Automated Acceptance testing is really an entirely different kind of testing. It is indeed an unaffordable way to attempt to prove that we are Building the Thing Right, but in my experience, the perfect way to prove that we are Building the Right Thing. I want these kinds of tests to simply demonstrate that we mostly got the scope and Definition of Done right for a given story. This is a far cry from all of the edge cases and unhappy paths that make up complete regression protection for a story.

If, as James claims, clients are trying to use Storytests for what they are not good at, I stop it. I suggest other testing avenues for regression protection.

This difference really is critical. Storytests merely tend to prove, better than anything else, that we got the story done.

Granted, a story is not Done Done Done until we have squeezed all the defects out of it. I hope to heck the bottom of my testing triangle, where the giant, high-speed suites of tiny little xUnit isolation tests / microtests live, does the lion’s share of the regression protection for me. Yes, TDD/BDD are design practices. AND, only from my cold dead hands will you pry the regression protection those tests/specs provide me. Please, please, don’t try to use FitNesse for that work. Wrong tool, man.

The Benefits of a Precise, Deterministic Definition of Done

So if I do have awesome xUnit test suites (and a bit of other regression protection) to prove we have Built the Thing Right, my Storytests need only prove, to some customer-acceptable extent, that we have Built the Right Thing. What benefits do I give up if I give up this use of Storytesting?  Well, I have a list, but here is the number one item on it:

1. My best tool for story breakdown. You want me to prove that a story in a web application without browser-resident behavior got done as estimated in this Sprint? Some small increment of useful service layer code or biz logic or whatever?  Storytesting is the first thing I reach for.

   Without that practice, I have teams (especially new ones) presuming that stories can only be as small as the smallest bit of browser resident behavior they evidence. That is a truly horrible thing, because then my stories can clandestinely grow to ginormous size. This leads, in turn, to late cycle iteration surprises (“Uh, it turns out that we just found out that this 6 foot-pound story is really gonna be something like 67 foot-pounds. It won’t be ready for the verification meeting tomorrow.”)

   Heck, one recent team I coached had an app with no GUI-evident behavior anywhere. FitNesse was the perfect way for them to show progress. Indeed, to them, it now seems in retrospect that Storytesting was the only way to fly. Without something like it, there would have been no way for product owners to verify anything at all.

Retiring Old Storytests

Large suites of automated functional tests, in any tool, are notoriously expensive to maintain, especially compared to xUnit microtests. FitNesse, being a web app without in-built refactoring support for things like multiple references across tables and pages, can make things worse. (People are slapping FitNesse front ends on top of Selenium suites these days, which strike me as truly  horrible for regression suites.)

Fine. Storytests are functional tests. They run slow and are very expensive to maintain  Therefore let’s only keep our Stortytests for as long as they are useful for verification, requirements scope, acceptance kinds of purposes.

Do I really need to prove, in Sprint n+10, that I got scope correct in Sprint n?  I suggest that I don’t. That’s old news. Deleting old Storytest suites also applies healthy pressure on the team to derive their regression protection from healthier tests and mechanisms.

Small Groups of Stakeholders Can Learn to Collaborate on Storytests

Don’t believe for a minute that this is impossible to do. I have frequently done it. I am happy to show you how to do it.

Yes it is difficult, but compared to what? Teaching teams OOD?  Teaching teams TDD? Configuring a Tomcat cluster? Please.

I’ve had several successes getting small sub-teams of developers, testers, and (critically) product owners to collaborate on Storytest design and development. No, I don’t want testers writing acceptance tests alone. No, I don’t think Product Owners can or should write such tests on their own either. And also, perhaps controversially, I am starting to think that good old fashioned Decision Table style permutation tables, as a Storytesting semantics, is the sweet spot for Java Storytesting. BDD step definitions, as developed so far in at least two ways for FitNesse, leave me cold: either I have several tables referring to each other in a way that makes refactoring cumbersome, or I have complex fixture code that uses regex and annotations. I will use these things if pressed by a savvy, committed product owner, but otherwise, give me Slim Decision Tables.

Honestly, I have on several occasions found ways to craft suites of Decision Tables (nee ColumnFixture tables) so that they are plenty expressive for all concerned. I’ve had several teams, including product owners, fall in love with the practice and the tool. I’ll keep rolling with that.

Summary: Be Careful What You Throw Away, and Why

Used as, I would claim, originally intended, Storytests / Automated Acceptance tests are a wonderful and essential tool for agile teams. More on this in later posts. I personally cannot control scope, carve stories small enough, or show deterministic enough definition of done without them.

Yes, client teams can learn to use the practice and tools improperly, in which case, it’s our job to step in and suggest alternatives.

Let’s try to come to agreement as a community on the ROI, the uses, and the best practices and patterns for Storytesting before we declare it dead.

As is often said and written, the history of programming, and programming language design, is about programs becoming more expressive, fluent, lingual. All of the Object Oriented programmers I know and trust most would say that when they program, they are creating what are, in effect, languages. So, as a community of software craftsmen, how intentional and explicit are we about programming — and teaching programming — in a “lingual way,” and what does that mean?

I care a lot about helping novice OO programmers learn OOD, because when I set out to learn it years ago, my education was flawed. And because most programmers in OO languages are very bad at OOD. And because when I try to teach OOD to others, these concepts are still not as easy to teach as I want them to be. And finally, because there is no better or more natural metaphor for expressiveness than the notion of a spoken language itself. Perhaps by definition.

I want to encourage novice OO programmers to think in an expressive, lingual, semantic way. And the terms of art in OOD are not helpful enough there.

I Dislike a Lot of OOD Terms

Don’t get me wrong. There is a rich vocabulary about being clear, clean, and expressive in programming, but none of the terms helps me in the way that I want. Here are some terms that are helpful, but not in the right way: programming by intention, meaningful names, expressive names, abstraction level, object modeling, domain-driven design, etc, etc. Each of these terms is either just plain hard to learn (“abstraction level”), or it is  focused on too small a lingual granule (a meaningfully-named method), or it is not especially lingual in focus at all (“object modeling”).

An object model can be healthy in the sense that the classes and their methods are about the right size, and focus on about the right scope of responsibility. And it can have OK, not great names, that are not semantically coherent. And the whole thing can feel like a non-German speaker driving through Germany: “What a beautiful little town!  Where the hell are we?”

An abstraction level can be decoupled nicely, and still not be expressive in a lingual way. “Wow, nice clean scoping!  What’s going on in here?”

A group of classes, methods, and variables can be pretty well named, in their own narrow, individual  contexts, and still not form a semantically consistent little vernacular.

In Charges the DSL

There is a sudden sexiness emerging around Domain Specific Languages (DSLs), and Martin Fowler’s book will increase the buzz. To Fowler’s mind, based on his research into the prior art around DSLs, the term should be reserved for a level of fluency that is “sentence like” in its grammatical richness. The chief application is to make problem domains, more than solution domains, very fluent and expressive, especially to non-technical stakeholders, rather in a Ubiqituous Language, Domain-Driven Design fashion. Fair enough. It’s a great idea, and frequently worth the effort. I am 110% in favor of it.

But FitNesse/Slim Given/When/Then DSLs (for example) don’t solve my problem, which is this: encouraging OO programmers to program in a lingually expressive way, within the limits of whatever programming language they are using. You can create real DSLs in Java using techniques like method chaining, and tools like Hamcrest matchers, but that ain’t exactly novice-level craft.

Fowler’s book draft defines DSL to explicitly exclude traditional command-query class APIs in languages like Java and C#. I want a term that encourages, describes, and defines what it means to make those command-query APIs as lingual as possible. I want novices to have guidelines for creating command-query APIs that form consistent, lingual semantics as collections of verbs, nouns, and other parts of speech.

That thing. That thing I just said. That’s what I want a term for.

Lingos and Lingual Design

So, unscared by the prospect of inventing a dumb pair of terms, I am inventing these two, with the help of my pals Dave LeBlanc and Mike Hill, who helped me think this through. Truthfully, there is not much to it:

A Lingo, in our definition, is a coherent, object-model subset, a command-query API, that is semantically consistent in its limited grammar, and as fluent as the programming language permits (without fancy stuff).

One typical scope for a Lingo is a package that adheres to all of the package coherence principles of common reuse and common closure. Those classes that tend to be released, reused, and changed together and therefore tend to cohere in their own package together, really ought to be crafted as a little language. That’s an example of what we mean by a Lingo.

And, a class can be a Lingo, and should be when it can: the semantics of the method names within the class should be grammatically consistent.

Now, Lingual Design is simply an orientation toward ensuring that our command-query APIs have clear, SRP-compliant boundaries (e.g., package boundaries or class boundaries), and tend to hang together as coherent, consistent Lingos.

[TODO: Insert code examples here!]

No, Java and C# and many strongly typed languages do not make this easy to do, and make you jump through fancy hoops to get all the way to Fowler’s definition of DSL fluency. So what!

You Are Always Creating a Language

Whatever general purpose programming language, or Domain Indifferent Language (as Mike Hill puts it) you are using, no matter what sort of API and object model you are crafting, you are always creating another language. More or less crude, more or less fluent, more or less semantically consistent, whatever you end up making will be read by other programmers in the high hopes that it reads like a consistent little bit of spoken language.

Try thinking, for a bit of your programming, in terms of Lingual Design. Try to see the boundaries between, and the hierarchical tiers, of your Lingos.

How does it feel, and how does it work, to be intentional about OOD in this particular way?  Can this be a useful way to teach and learn OOD?

Are these terms useful or just more pedagogical noise?

Am I, once again, smoking crack?

Time, and my friend Ron, will tell. ;>

This balance can be hard for Westerners to grasp, but it helps us to learn balance. An odd example of Yin/Yang balance can be found in healthy programming, balanced with healthy testing.

The Yang of Programming

The Yin of Testing

Balanced Yang/Yin Programming

Healthy programming is lots and lots of passionate, creative Yang, with just the right amount of Yin observation and Testing. It is creative engineering and artistic passion balanced with a bit of skeptical, empirical Science. It is the black teardrop with the small white circle at its heart.

Balanced Yin/Yang Testing

Healthy testing embraces the calm, dispassionate, empirical, deeply-concentrating power of witnessing, observing, and noting the truth of reality, without preconception. It delights in defects not merely because it is fun to break things, but also because it is fun to find precise, repeatable, deterministic discoveries of any kind. A bug is just another kind of delightful new discovery that happens not to be desirable by the customer (in most cases!).

Balanced Yin testing begins with the open, empirical, unscripted extemporizing act of manual Exploratory Testing. This demands that ET skills have already been deeply somaticized by the tester, rather like the kata of programming, or like the “forms” of Tai Chi. Before you can deeply explore the behaviors of an application, you must already have internalized the skills of that exploration. I dare say that masterful ET has a meditative, contemplative aspect to it. You allow the application, as it truly is, to you draw you in, without agenda, without story.

And, healthy Yin testing is then balanced by the passionate Yang activity of test automation. Automating regression happy paths and unhappy paths is such a Yang activity. Another balancing Yang activity is courageously sharing results with programmers, and with the team. The best testers, Agile Testers, are powerfully creative in their own right as Yang programmers. But their Yang programming tends to serve their Yin goals of observation, gathering empirical results, witnesssing. Balanced Yin testing: the white teardrop of observation with the small black dot of Yang creation at its heart.

Balanced Testing/Programming: Conversations and Careers

Also consider becoming, for awhile, an Agile Tester. You think you understand every aspect of software testing, and how important all of the kinds of testing are? Rest more fully in the Yin of Agile Testing for a few days, weeks, iterations, or years, and think again. (Also, do not make up stories that this is somehow beneath you, or insufficiently challenging or stimulating, or not worthwhile, unless and until you have tried it in good faith, and found that to be so.)

Also consider becoming, for awhile, an Agile Programmer, or a Software Craftsman. You think you understand all of the craft and technical depth and pure creative striving required to test-drive all manner of production implementations through all manner of technology stacks, with least defects and cleanest code? Strive powerfully in the Yang of Agile Programming and Software Craftsmanship for a few days, weeks, iterations, or years, and think again. (Also, do not make up stories that you cannot do that, unless you have already tried in good faith, and failed.)

It’s Already Happening

Balance is possible between programming and testing, and it may help us to remember and honor our natural inclinations as Programmers and Testers as we work toward that balance. As programmers we are Creators, Engineers, Artists. As testers we are Observers, Skeptics, Research Scientists.

Let it Happen More On Your Team, In Your Career

And one avenue of balance, for the very brave and perhaps very advanced, is to push ourselves past our old internal stories about our inclinations and limitations. Specializing Generalists can shift their specialties occasionally. Creators: I dare you to master Observation. Obvservers: I dare you to master passionate Creation.

In the future, as teams become increasingly agile and healthy, I hope we will see a natural flow back and forth between the communities of those who self-identify as Testers, and those who self-identify as Programmers.

Because balance is better than good. Balance is at the heart of true excellence.

Well done Jean, Liz, Eric, and others. Good stuff:

A Community of Thinkers

I am a member of a community of thinkers.

I believe that communities exist as homes for professionals to learn, teach, and reflect on their work.

I challenge each community in the software industry to:

• reflect and honor the practitioners who make its existence possible;
• provide an excellent experience for its members;
• support the excellent experience its members provide for their clients and colleagues in all aspects of their professional interactions;
• exemplify, as a body, the professional and humane behavior of its members;
• engage and collaborate within and across communities through respectful exploration of diverse and divergent insights;
• embrace newcomers to the community openly and to celebrate ongoing journeys; and
• thrive on the sustained health of the community and its members through continual reflection and improvement.

I believe that leaders in each community have a responsibility to exhibit these behaviors, and that people who exhibit these behaviors will become leaders.

I am a member of a community of thinkers. If I should happen to be a catalyst more than others, I consider that a tribute to those who have inspired me.

”A Community of Thinkers” by Liz Keogh, Jean Tabaka and Eric Willeke is licensed under aCreative Commons Attribution-Share Alike 3.0 License. Please attribute to the distributor of your copy or derivative.

By way of even more background, my original post on this topic is here.

Patterns are Emerging

In response to several comments and conversations with people in the industry, several of us, most recently Chris Beale, Gary Baker, John Huston, and Daryl Kulak, have been converging on a definition of this new role, this new variety of leadership that we want agile teams to have. Many others are also involved in the conversation by now; too many to list here. The more we talk about it, the more comments we receive, the more all of us can see what we want. We are not in complete unanimity, by any stretch. Some folks think we are smoking crack. Not the first time. But most of us in the conversation can see useful patterns emerging.

This is a Big Role

We can also see that this is quite  a lot of skill, experience, leadership, and sheer work to expect of one person. So part of my premise goes like this: the Agile Team Lead is the person that is ultimately accountable for ensuring that this work gets done. Depending on how the skillsets, preferences, team maturity, and other factors shake out for a leader and his/her team, the Agile Team Lead may well end up delegating much leadership to others (and other, subordinate leaders) within the team (the key phrase there being “within the team.”). So an Agile Team Lead might very well delegate to a Tech Lead, to an Agile Test Lead, to a Project Tracking Lead, to a Continuous Deployment Lead, etc., to cover off concerns. Whatever. The main point, of course, is that the buck stops with the Agile Team Lead. That person is, for whatever duration they are lead, the single wringable neck (for all responsibilities owned by the team itself).

This Role Must Emerge From Within the Team, and Must Satisfy All Stakeholders

So, while I am decidedly talking about DESIGNATING a person to act as the team lead, I am decidedly not talking about a traditional manager being imposed, permanently, from above, by fiat. There can be no Agile Team Lead designated FOR a team that does not know, admire, trust, respect, and like that person. Really, the final say in who leads the team should be the team’s.

This Role Should Likely Rotate

This may be something we want several team members to try out, to get good at. Team members could rotate through it, perhaps on a fixed schedule.

This Conversation, Um, Leads to a Few Others

I’d like to postpone to another series of blog posts, (already underway, however), the thorny issues around the enterprise structural implications that arise from this new role, and related observations. Suffice it to say, for now, that I personally don’t know how to make agile teams work, nor how to make this role of Agile Team Lead work, unless everyone in the team room reports directly to the Agile Team Lead, and no-one else. I mean that that one person is accountable for all of the incentive structure, performance evaluation, and (in large part) the hiring and firing for everyone on the team. This does not, again, mean that they own it entirely, nor that they do it permanently.

One other premise: we are deliberately attempting to erase old agile team management role and responsibility labels, like Scrum Master and Project Manager, which we increasingly believe to be fundamentally broken. We find the controversy around this topic to be perfectly matched to the stakes involved for the enterprise: our current patchwork of agile team management and project management and product ownership and process improvement management consists of more holes than cheese. Too many team-level concerns go unmet; too many stakeholder concerns go unmet; too many user and market concerns go unmet. We have been in denial about this shortfall. Let’s give up that denial.

Love it or hate it, we need a paradigm shift around agile team leadership. We need that leadership to be better consolidated, better focused, better defined, and better matched to the dialectic between internal team needs and external stakeholder needs. We need one person who is held accountable the extent to which an agile team’s authority and responsibility are, in fact, congruent.

And we need none of that to violate the principles of self-organizing teams, leadership arising within the team, and teams owning the team’s concerns. The team owns all of these things, but the team lead is accountable for the team actually delivering it. Like I said, thorny.

Summary: What we Want in Our Agile Team Lead

Most of the following is in summary, bullet-point form, with a bit of explanatory text. We’re still mainly starting this conversation; much is left to be done to think this through from the perspective of the team itself, from the perspective of “continuous improvement,” from the perspective of individual Releases/Projects being executed well, from the perspective of Product Owners and Portfolio Owners, from the perspective of all of the different natural roles, responsibilities, skills, learning styles, and personality types that the team needs. These is just another draft sketch of the responsibility categories of this specific leadership role.

I suspect there are still several holes in our thinking, several bits of overlap, and perhaps some false premises. No matter. It’s out here now for us all to discuss. Better in front of your eyes here in public, than lurking, not yet “perfect,” on someone’s laptop.

Continuous Leadership

• Understand the goals of the organization, and the exact ways in which the team furthers them
• Provide a single point of leadership for the team; continuously wins the team’s implicit and explicit trust and admiration. Builds a “safe container” for the team within which work can be done creatively, collaboratively, honestly, accountably, and joyfully.
• Helps continuously build trust and respect between the team and its stakeholders (in a way that buys more and more freedom to speak up, freedom to experiment, freedom to “fail,” freedom to learn).
• Provide a single point of team accountability to the team’s stakeholders (in a way that ensures a single, coherent message to stakeholders about any team commitment about what went well, and what did not go well, how it happened, and why).
• Helps continuously improve internal team-cohesion: this includes mutual respect, mutual trust, morale, rapport, celebration, accountability, fun, and affection (yes, we have to like each other).

Continuous Planning

• Ensures that the team is making more and more accurate and precise commitments.
• Ensures that the team is delivering more and more predictably on its commitments.
• “Gets the lights on,” in every area of process and practice, and ensures everything is “big and visible,” both tactically within the team, and for purposes of publishing to external stakeholders.
• Identifies, gathers, and acts tactically on meaningful performance/progress metrics
• “Makes the plan the bad guy“: ensures that Definitions of Done, short-term scope, long-term scope, deadlines, quality measures, and all other commitments are expressed in “plans” that remove moral judgment and blame from continuous improvement. Plans include code, tests, cards, and other artifacts. The goal is to create an environment in which mistakes are expected, any shortfall is cheap and easy to learn from, and everyone’s reflex is to refine the “plan,” not to point fingers of blame.
• Drives Commitment-making for and with the team. This includes ensuring that the Definition of Done for any small or large commitment is precise enough and deterministic enough to reduce risk of misunderstanding, scope creep, and under-delivery as much as possible.
• Ensures that the plan changes with demand/supply.

Continuous Execution

• Monitors/manages team velocity/throughput. This includes work breakdown practices, estimation practices, tracking practices.
• Reports meaningful metrics/data within the team, tactically, and likely publishes different metrics/data outside the team to external stakeholders.
• Gets the team the resources it needs to deliver on its commitments; removes roadblocks
• Escalates blockages effectively. When the Agile Team Lead cannot unblock something, he or she escalates the issue to someone with the authority to remove that blockage, and makes clear to all stakeholders what the risks are of leaving that issue unblocked.
• Keeps flow, momentum and focus in the team.

Continuous Risk Reduction

• Continuously identifies short-term risks (both Sprint-scope and long-term, Release-scope and Portfolio-scope) across the full spectrum of people, process, technology, structure, and culture.
• Makes risks and potential impacts big and visible to the right people. This is another example of “Making the Plan the Bad Guy.”
• Ensures that risk reduction is integrated into planning and execution
• Measures the effectiveness of risk avoidance and the impact of unmitigated risk.

Continuous Improvement (Agile Coaching)

• Drives the improvement the overall Definition of Done (at each level of commitment scope, from the Story through the Release, and perhaps beyond). This has the effect of improving the quality of commitments, building a habit of overdelivering, building stakeholder trust, smoothing the flow of work (with fewer and fewer scope misunderstandings, production defects, and other “unplanned work phases”).
• Senses where performance breakdowns are occurring
• Creates big and visible means for measuring breakdowns
• Interprets data to identify biggest opportunities for improvement.
• Works with team to identify and implement improvements. This obviously includes owning a retrospective practice, and also includes owning the effectiveness and accountability culture around a reliable pattern of Inspection and Adaptation. It includes things like helping the team learn the difference between ego-based whining and team-based servant leadership, and the difference between vague negative generalization and concrete improvement proposal.
• Investigates and learns emerging practices from other teams, organizations, and communities.

OK, so once more, some disclaimer. This is work in progress. Expect more posts, other venues of conversation. Meanwhile, we welcome, indeed we desperately need, your input.

I came up in a software industry in which the atomic unit of production capacity was The Programmer. Programmers were largely assigned, as needed, to ephemeral entities called Projects. Yes, it had a Taylorist mechanistic undertone, and often still does.

Cut to scene: The Project Manager whisks into the Development Manager’s office and declaims, rather like a Hollywood casting director, “Well, you need to get us Joe and Terri for the Melathobner Project. We can’t do it without them. It just won’t work without them.”  (Lots of action films have the word “Project” right in the title!  Coincidence?  I think not.)

As a film crew comes together once, against tremendous odds fought by various producers and directors, to birth a film production, so have I seen most projects run. As if after this project, the world might end. As if after this project, we could all retire. As if deep down in their hearts, stakeholders and managers secretly believe this: “The idea that this project might actually make it out the door on-time, under-budget, with few defects?  That’s hilarious.”

Then, at miraculous Project conclusion, all cast and crew return to their structural silos (their true homes). Then off to the next Project. Emergency permeates the whole mechanism. Whoa!  This next Project might be the one that finally kills us all!

Project: Wrong Atomic Unit of Production?

Even if a product or system might persist for years, each of its major releases (sequels, in fact) is typically still a Project. Most software projects still have this one-time, heroic, desperate, fire-and-forget feel to it. This very narrow focus, this “goal obsession” on the part of project managers was recently described well here. Certainly there are Project Managers who are better than that, and focused more broadly on notions of business value flow, portfolio, market differentiation, team productivity, context switching costs. They are the exception, partly for reasons I cover below.

One or the root cause issues here is this: as an atomic unit of production, Projects can be inherently bad. And, as I’ll discuss later, fundamentally different than an alternative granule: the Release. First, let’s go to a military metaphor, which is always galvanizing.

Corporate Silos: 14th Century Italian Civil War Between City States

Meanwhile, what, in the context of software Projects, is the corporate atomic unit of budget authority, responsibility, and power? The structural silo. Ouch. We have somehow evolved default software development structure and process that seem to quietly mimic 14th Century Italian feudal warfare.

Our structural silos, under their various VPs and Directors, are capable of behaving like city states. People who report up through Architecture, Development, Production Deployjment/Support, DBAs, QA, Facilities, etc — are typically most loyal to their role and responsibility silos. How could they not be? That is how they have been incentivized and empowered. They represent abstract notions of “quality” and “architecture” and “programming,” divorced from the very real, complex, organic, end-to-end needs of the enterprise (or worse; more below).

None of these silos owns the proper flow of business value through the organization. You could say that our corporate Italy has insufficient representation and incentive for its larger, nation-sized concerns. It’s OK with our corporate Firenza if Italia perishes, as nonsensical as that seems at first blush.

None of the city states owns whether we are reducing costs and increasing revenues for the entire enterprise. Indeed, I have frequently seen these city states incentivized to combat cost reduction and revenue increase, in order to gather more resources and power to themselves. QA is incentivized to find bugs, and find them they do; their weapon of choice: the bug report system. Architecture is incentivized to represent the bigger picture, the technology future; their weapons of choice: Visio and PowerPoint. And so on. It’s a fabulous buck-passing machine at best and, (segue alert! New metaphor) at worst, as if some corporate departments are like parasites feeding on the enterprise as host organism.

Note that I am not saying that the enterprise does not need to address the concerns and skills and worldviews traditionally represented by architects, testers, production deployment and configuration experts, project managers, etc. I am not against all specialization.

But meanwhile, again and again, the Project Managers assemble these temporary Waterfall initiatives, using the best go-to condottieri (think “Ronin”) they can find, and lead them through Waterfall-phase silo-gauntlets, one at a time. As the project winds down, the temporary allegiances dissolve. Back everyone goes to their little city-state.

So, as I’ll discuss, what I increasingly question is the idea of empowering structural silos that represent sets of skills, concerns, and worldviews, using budgets, hiring authority, etc. If that’s how the money flows, into these little city states, I think you should expect your own little replay of the 14th Century Italian civil wars.

Voila, Disruptive Change:
The Agile Team as New Atomic Unit of Production Capacity

So we have this existing corporate socio-economic fabric: Projects being pushed, against powerful odds, through this fractured system of city states. Real business value is not created as a direct result of this fabric, but despite it. People strike backroom deals. A barter economy of favors breaks out. Grudges arise, and are repaid. It is, to my mind, more ugly than pretty.

And then along comes this notion of a self-organizing, empowered, cross-functional, high-discipline, highly-skilled Agile Team, as the new fundamental unit of production capacity. We’re going to mow down all these cube walls, bring all of these people in from their various city states, several of whom quietly loathe each other, and ask them to form a cohesive team. We’re going to ask them to sit with each other, deliver measurable business value in a steady flow together. Form, Norm, and Storm. We’re going to ask them to treat this new team space, not like a brief Project-scope visit to a kind of summer camp, but as their permanent home. We’re asking them to break their powerful, traditional allegiance to their city state. Remove that tattoo; forget that secret handshake. Even though that’s how their performance gets evaluated. That’s how they get paid. No wonder it so frequently looks like a bunch of Hippie Kumbaya.

No wonder the corporate anti-bodies swarm (oops, yes, another metaphor; should have warned you). No wonder the first several retrospectives feel like the teacher asking the class who put the thumbtack on her chair.

Yes, I am Over-Simplifying

To make a point, yes I am. The truth is, I have seen the above scenario more than once. The larger the organization I am working with, the worse it is, and the likelier it is to fight agility, and fight smooth business value flow.

But let’s at least find a common pattern language or metaphor set for this fundamental issue of “cultural change” in the enterprise. In future blog posts, which will be about how all the Hippy Kumbaya agile team culture can actually help the entire enterprise evolve in a way that results in true competitive advantage at the enterprise level, I’ll propose new terms and newish structure (not entirely my invention).

Meanwhile, though, here is my premise:

Your entire corporate socio-economic fabric might be fundamentally busted, seen through the lense of software development “Projects.” You might be recreating civil war in Italy in the 14th century. This might make a great film, but it could be a terrible context within which to try to optimize how the collaborative creation of software systems (be they products out in the market wild, or internal web services) increases your revenue, or decreases your expenses.

I’ve had both. In both cases, I learned not to mess up in the following way: trying to grab the bird and toss it out of the house. Birds hate that, and are really skillful at resisting the attempt. You or the bird can also get badly injured that way.

As a Shu-level, novice coach, I did the classic wrong-headed stuff, and still work hard to resist temptations to do the same stuff: pushing, pulling, badgering the client from a bad situation to a good one. It really is like chasing a bird around the house, brandishing a tennis racket. (Note to self: No!)

Releasing Attachment to the Outcome

We cannot truly control whether, indeed, the bird leaves the house, much less how fast or how well. There are no shortcuts for us or the bird. We can, however, strongly influence how the whole thing unfolds.

It really does have to be the bird’s idea to leave the house, if it will ever in fact happen. This may take a few seconds, a few minutes, a few hours (a few weeks! a few months!  a few years!). Take your time.

Those of us who have (as I have) released birds, bats, and squirrels from the house have learned (A) how to open and close doors, and leave seed trails, in a way that invites the unhappy bird or squirrel out of the house; and (B) patience. With one bat, who flew along walls at 2″ from the wall, straight to each corner, then turned an abrupt 90-degree turn at the next corner, over and over, in perfect silence, it really took awhile. That bat took 30 minutes to notice that there was a door open along one of those walls (if the door had opened in, instead of out, it might have taken 1 minute).

When he finally flew through it, I pounced up and shut that door, preventing regression. I had been sitting there, drinking a beer, and tracking him as he flew these perfectly rectangular room circuits (not easy: bats indoors are flying about as fast as your neck muscles will allow your head to pivot).

Once he had found that first door, he seemed to sense that other open doors were good things, and flew straight through the remaining rooms out to freedom, bugs, and no doubt a well-earned nap. In classic coaching fashion, I had reached that tipping point where I had won enough trust to say goodbye.

Close Some Doors, Open Others

If you close doors to the inner part of the house (example: removing folks from their cubes and placing them in open workspaces, or not engaging in thermonuclear email exchanges about the New Agile Initiative),  you can invite clients to not worsen their situation.

If you open other doors and windows to the great agile outdoors, you invite clients to try inviting, healthy, addictively fun, cool new things (example: tweaking the card wall to reveal blockages or gaps in value flow; helping clients write their first few storytests).

Asking Gently Provocative Questions: Opening Windows in the Mind

By this, I don’t mean “Where the hell are all of your unit tests”?  (BTW, you know the question I REALLY don’t mean?  It’s this one:  ”Don’t you realize testability is vastly more crucial than encapsulation, you dolt?”)

I mean, instead, questions like “Has Rob paired with Lily recently?”  or “How did method-level cyclomatic complexity change this past iteration?”  or “How do you think that business verification went?” or “What do you think about the size of that test?” or “What do you think of that class name?” or “Did we see this retrospective improvement item in the last retrospective?” or “What do you really enjoy doing most on the team?”

If the Bird Blogs About How Cleverly I Let Him Out of the House,
I Probably Messed Up

The best of my personal coaches model for me, and counsel to me, the art of the client never discovering that the great idea was not originally theirs. Smart subordinates have been leading their managers from below in this fashion since the dawn of command and control, no doubt.

Not only is it OK, in a coaching position, to be a peer-to-peer influencer, as opposed to a top-down manager, it is typically preferable. People who I coach end up taking leaps of faith and trying scary new things, in good faith, not because anyone, including me, commands them to do them, or to try them. They do so because they trust me, like me, respect me, and happen to have had this cool idea pop spontaneously into their heads, unbidden.

The coaches I most respect, and most seek to work with, are those who help me as I refine my Ha and Ri in this coaching by question, coaching by seed trail, coaching by patient, engaged, gently provocative observation.

If you see me badgering someone to just shut up and do this Super Advanced Best Agile Practice Thing, please bust me on it. I will likely thank you (eventually, anyway).

Premise: It Ain’t Project Management

Meanwhile, coincidentally, I had been in conversations for the last week with stakeholders about how, in an agile transition, old patterns of Project Management are not a good fit. Traditional Project Managers are typically insufficiently team-focused and insufficiently portfolio-focused, while too top-down and imperative. Agile teams and systems have long lifecycles and broadening authority and responsibility that traditional Project Managers cannot typically see. With all due respect to Spann (with whom I think I agree nearly entirely), I would claim that, adaptive or not, neither the adjective “Project” nor the noun “Manager” helps the person trying to lead an agile team. The old culture and terminology are equally misplaced in an emerging agile world.

So, true, the agile community has not had great answers to questions about who leads and manages agile teams, much less how. So, last week, when I was asked by my best Pillar pal and mentor what the main responsibility, the main tools, and the guiding principle of agile team leadership might be, I realized, after some reflection, that I have been baking these in my head for awhile, semi-consciously. David Spann and Abby Fichtner helped me bake them a bit further.

Premise: It Ain’t Tech Leadership

The Agile Team Lead (for lack of a better role term) is not the tech lead. Many teams need formal technical leadership; many teams may grow out of this, into a pattern of entirely emergent tech leadership and architecture. In any case, the tech lead role is not the right role (nor is it typically, as Abby implies, the right skills match or personality-type match) for team leadership overall.

Premise: It Ain’t Scrum Master

As Abby claims at the start of her post, focus on methodology alone, even while there is frequently a temporary place for a certain style of such coaching, is again not the right focus for an Agile Team Lead. This is another focus that is too narrow, and too short. The team should outlive its Scrum Master by years; that may be a measure of a Scrum Master’s skill: the ability to leave the team in a cohesive and self-improving state. No, process coaching is not exactly agile team leadership either.

Look!  A Man with Nine Legs! Oh Bloody Hell, He Ran Away

We need something “completely different,” however rare, in our Agile Team Lead. Here are the things I now want:

Overarching Responsibility: Optimizing Flow of Business Value

In whatever ways a team improves: technical, interpersonal, process, practice, attitude, morale — we can measure and express them in how business value is delivered more and more efficiently, easily, steadily, skillfully, and (I’ve seen it) joyfully. The Agile Team Lead can trace all improvements back to how well the team delivers ROI and reduces code asset TCO, and back to how delighted the customer and team both are. The Agile Team Lead is in the business of knocking the customer’s socks off, over and over again.

Four Main Tools

The Agile Team Lead optimizes business value flow mainly with four varieties of tactic:

1. Continuous Team Building — this includes all of the soft, fuzzy means Spann lists, whose ends are team cohesion, high ambient trust, high ambient respect, high morale, good humor, equanimity, close interpersonal connection. In Jazz, musicians talk about a band having so much groove that everyone can read everyone else’s mind. Reinforcing that is the Agile Team Lead’s link inward toward the team (bear with my link metaphor here for a few more paragraphs).
2. Continuous Planning — for current purposes, I mean by this all of the agile practices and patterns around eliciting, analyzing, scoping, dividing, prioritizing, iterating, planning, testing, tracking, publishing, and delivering. The Agile Team Lead does not perform all of these practices solo, but leads their performance. This is how the Agile Team Lead is held to account by all managers and stakeholders outside the team. This is the outward link from the team to the world.
3. Continuous Unblocking — teams continuously need outside help, resources, more team members, fewer team members, and other emergent support that they, as a team, lack the authority to achieve. The Agile Team Lead does the heavy corporate lifting to get these deals swung, to keep the team in steady flow. This is the link from the outside world back to the team.
4. Continuous Improvement — The Agile Team Lead guides, without pushing, the zillion varieties of retrospective improvements that the team does own the authority to make. This does include holding the team accountable for owning the ever-changing lists of things to be improved, and concrete improvement proposals and plans. Note that this is not the same as coaching, which may need to happen separately. Teams had better outgrow coaching, while they are never likely to outgrow a need for leadership. These are the links from the team back to itself.

Guiding Principle: Servant Leadership

The leaders most admired in the agile community and software community, as in the world at large, are those who lead by giving and supporting. This is classically the root cause of a graduation from “good” to “great.”

And we are learning that it is simply the most effective variety of leadership and management: collaborative, listening, deeply present, deeply respectful, enthusiastic, inspiring, supportive, celebratory. And certainly trusted and adaptive, as Spann claims.

The Servant Leader attaches identity to the team’s abilities and accomplishments, not just to his or her own. Spann has captured several of the beliefs and concrete behaviors that we can only get from these leaders who lead to serve, not to rule. Indeed, an agile team’s Agile Team Lead must earn the team’s consent to be lead. Mature self-organizing teams eventually outgrow structural, imperative command and control, imposed by fiat.

Doesn’t it seem sometimes that the whole technical world is trying to go collaborative, trust-community, peer-to-peer? Aren’t these same patterns emerging everywhere, in different bits and bobs? Certainly it feels that way to me. Flat is the new hierarchy; village is the new city; conversation is the new document; trust is the new currency; small is the new big.

Thoughts?

That’s my sketch so far: the Agile Team Lead, through the lense of Servant Leadership, uses Continuous Team Building, Continuous Planning, Continuous Unblocking, and Continuous Improvement to optimize how business value flows through the team.

I am still baking this one; your feedback is appreciated.