Flipping the Automated Testing Triangle: the Upshot

Posted on by

This afternoon at Agile 2008 in Toronto, which has been a smashing good time, I led (with lots of help from Lisa Crispin, and then serendipitously from Brian Marick, J.B. Rainsberger, Dave LeBlanc, Matt VanVleet, and Declan Whelan) a presentation/workshop/micro-Open-Space session. I began with a premise from Mike Cohn: the idea of an ideal testing pyramid, or triangle, with three tiers:

automated testing triangle

What makes the triangle ideal is that you are spending most of your resources, and relying most, on the bottom-most tier of xUnit tests to do good things like protect you from defects and drive your design. That suite does the lion’s share of your regression protection, for example. And conversely, you are relying least, and expending least resources, on very slow, brittle, large, black-box, through-the-GUI, recorded kinds of tests as created by tools like Selenium IDE.

My notion is that this minimizes the Total Cost of Ownership (TCO) of your automated tests. Eventually.

It’s a long (3-hour!) conversation, but the gist, as presented, is that nearly no teams get to start with so well-formed a triangle, and in fact the triangle is most typically inverted for teams just starting down the agile path:

bad_automated_testing_triangle

Given such a challenging starting point, you need three automated testing initiatives, one for each of the three kinds of tests (which I refer to metaphorically as brick, stick, and straw), to ensure that you end up with the triangle above. And this is hard, in large part because it is so hard to learn to produce really, really effective suites of unit/isolation/programmer/micro tests (as defined by folks like Michael Feathers, J.B. Rainsberger, Mike Hill, Bob Martin, etc). And the triangle-flipping is hard also because we are talking about culture change, and resistance to it. Well, of course, there are lots of other reasons why it’s hard. It’s hard.

In the presentation, we explored, together, lots of subcategories of this automated-testing-triangle-flipping challenge. To my mind, there were lots of aha moments and unexpected conclusions and good ideas.

So, I’ll blog more on the details later. In the meantime, as promised to several people,  here is a link to a PDF version of the presentation. I have had trouble uploading the Keynote version, and so abandoned that. PDF is likely best in any case.

Thanks again to all who attended, and all who helped me prepare and present. It was a blast.

My next goal for this material is to extend it into an experiential, inspirational, 2-day training course that allows cross-disciplinary teams to experience the nasty cruddy inverted testing triangle on day one, and the lovely, right-side-up triangle on day two. More later.  Now, at 2:00 AM, off to bed with this agilista.