Automated Acceptance Tests: Hold on Just a Second Here

Long Live Storytests, Dang Blast It

The recent claims made by a well-known agile coaching thoughtleader notwithstanding, I work hard to get clients to adopt real Storytesting practices, with real Storytesting tools (FitNesse is still my tool of choice; I work mostly with Java teams). I will continue to do so. I find no false economy or Faustian bargain with FitNesse tests, and I suspect it is because I am coaching the use of them differently than James Shore is.

Manual Regression Testing = Really Bad Thing; Agreed

Regression testing of any kind is classically about proving that we are Building the Thing Right. For true regression protection purposes, I want manual regression tests to be replaced with a requisite blend of automated tests (using the testing triangle pattern for allocating test automation resources) plus a requisite amount of manual exploratory testing.

Whoa Nelly: Storytests Are Not About Bugs

But Storytesting / Automated Acceptance testing is really an entirely different kind of testing. It is indeed an unaffordable way to attempt to prove that we are Building the Thing Right, but in my experience, the perfect way to prove that we are Building the Right Thing. I want these kinds of tests to simply demonstrate that we mostly got the scope and Definition of Done right for a given story. This is a far cry from all of the edge cases and unhappy paths that make up complete regression protection for a story.

If, as James claims, clients are trying to use Storytests for what they are not good at, I stop it. I suggest other testing avenues for regression protection.

This difference really is critical. Storytests merely tend to prove, better than anything else, that we got the story done.

Granted, a story is not Done Done Done until we have squeezed all the defects out of it. I hope to heck the bottom of my testing triangle, where the giant, high-speed suites of tiny little xUnit isolation tests / microtests live, does the lion’s share of the regression protection for me. Yes, TDD/BDD are design practices. AND, only from my cold dead hands will you pry the regression protection those tests/specs provide me. Please, please, don’t try to use FitNesse for that work. Wrong tool, man.

The Benefits of a Precise, Deterministic Definition of Done

So if I do have awesome xUnit test suites (and a bit of other regression protection) to prove we have Built the Thing Right, my Storytests need only prove, to some customer-acceptable extent, that we have Built the Right Thing. What benefits do I give up if I give up this use of Storytesting?  Well, I have a list, but here is the number one item on it:

  1. My best tool for story breakdown. You want me to prove that a story in a web application without browser-resident behavior got done as estimated in this Sprint? Some small increment of useful service layer code or biz logic or whatever?  Storytesting is the first thing I reach for.

    Without that practice, I have teams (especially new ones) presuming that stories can only be as small as the smallest bit of browser resident behavior they evidence. That is a truly horrible thing, because then my stories can clandestinely grow to ginormous size. This leads, in turn, to late cycle iteration surprises (“Uh, it turns out that we just found out that this 6 foot-pound story is really gonna be something like 67 foot-pounds. It won’t be ready for the verification meeting tomorrow.”)

    Heck, one recent team I coached had an app with no GUI-evident behavior anywhere. FitNesse was the perfect way for them to show progress. Indeed, to them, it now seems in retrospect that Storytesting was the only way to fly. Without something like it, there would have been no way for product owners to verify anything at all.

Retiring Old Storytests

Large suites of automated functional tests, in any tool, are notoriously expensive to maintain, especially compared to xUnit microtests. FitNesse, being a web app without in-built refactoring support for things like multiple references across tables and pages, can make things worse. (People are slapping FitNesse front ends on top of Selenium suites these days, which strike me as truly  horrible for regression suites.)

Fine. Storytests are functional tests. They run slow and are very expensive to maintain  Therefore let’s only keep our Stortytests for as long as they are useful for verification, requirements scope, acceptance kinds of purposes.

Do I really need to prove, in Sprint n+10, that I got scope correct in Sprint n?  I suggest that I don’t. That’s old news. Deleting old Storytest suites also applies healthy pressure on the team to derive their regression protection from healthier tests and mechanisms.

Small Groups of Stakeholders Can Learn to Collaborate on Storytests

Don’t believe for a minute that this is impossible to do. I have frequently done it. I am happy to show you how to do it.

Yes it is difficult, but compared to what? Teaching teams OOD?  Teaching teams TDD? Configuring a Tomcat cluster? Please.

I’ve had several successes getting small sub-teams of developers, testers, and (critically) product owners to collaborate on Storytest design and development. No, I don’t want testers writing acceptance tests alone. No, I don’t think Product Owners can or should write such tests on their own either. And also, perhaps controversially, I am starting to think that good old fashioned Decision Table style permutation tables, as a Storytesting semantics, is the sweet spot for Java Storytesting. BDD step definitions, as developed so far in at least two ways for FitNesse, leave me cold: either I have several tables referring to each other in a way that makes refactoring cumbersome, or I have complex fixture code that uses regex and annotations. I will use these things if pressed by a savvy, committed product owner, but otherwise, give me Slim Decision Tables.

Honestly, I have on several occasions found ways to craft suites of Decision Tables (nee ColumnFixture tables) so that they are plenty expressive for all concerned. I’ve had several teams, including product owners, fall in love with the practice and the tool. I’ll keep rolling with that.

Summary: Be Careful What You Throw Away, and Why

Used as, I would claim, originally intended, Storytests / Automated Acceptance tests are a wonderful and essential tool for agile teams. More on this in later posts. I personally cannot control scope, carve stories small enough, or show deterministic enough definition of done without them.

Yes, client teams can learn to use the practice and tools improperly, in which case, it’s our job to step in and suggest alternatives.

Let’s try to come to agreement as a community on the ROI, the uses, and the best practices and patterns for Storytesting before we declare it dead.