Speculation = Bad
Most of the work I do will — at some point — be reviewed by someone for completeness, accuracy, and quality. I will need that feedback. On a team, we often need to incorporate everyone’s great ideas. Again, feedback. Whether work is individual or collaborative, the less work I personally do before I get the feedback I need (customer review, team-member review, compilation, running test, etc), the less speculative work I have done. The less work I am likely to have to redo. The fewer defects I am likely to have introduced. The less work, as Pillar Technology VP Chris Beale says, I “put at risk.” The less likely I am to have completely wasted my time barking up the wrong tree, chasing a wild goose, you get the idea.
The agile literature is not exactly rife with formal references to the principle of end-to-end early. Yet all good agile teams practice it, more or less formally. At Pillar Technology, we strive to be explicit about this interesting agile principle, and to apply it ubiquitously and rigorously.
In this and future blogs on this topic I’ll explore a few specific applications of e2e, and also what it “feels like” to have this principle woven into your DNA (I admit to some Chutzpah there, but I’ve been at this awhile). So, first, a couple of concepts at the Principle level, before we dive into example e2e in practice. How about starting with an attempted definition? Open to your feedback, of course:
End-to-End Early, applied to any practice, gives us faster, cheaper feedback. Keeping feedback loops small tends to drive out speculation, which reduces rework, waste, and confusion. This tends to keep costs lower, keep options open, and increase ROI.
Tiny little feedback loops are a rare commodity in software development, where traditional waterfall development defers feedback like end-to-end integration, end-to-end testing, and customer feedback until very late indeed. Many agile practices are designed to give us earlier, better feedback, but it is worthwhile understanding the e2e principle in general, and its universal power to save time and money by reducing waste and rework.
So enough theory. How does this work? How do we use this principle to make practice-level decisions?
Dividing a Release into Value Stories and User Stories
As part of a so-called Pillar Plan & Define phase (where we practice what we call Agile Project Scoping, in order to commit to a pretty hard dollar figure and date up front, where our clients typically insist on getting them), Pillar software consultants divide valuable work into successively smaller, iteratively-deliverable chunks. They first elicit and prioritize the big chunks – Value Stories – and plan for those to be delivered in early releases. (In other agile methods these Value Stories might be called epics or themes, but would not likely be tied back to concrete expressions of business value, while our Value Stories typically are.)
Value Stories are then divided into (at least) User Stories, which in turn are prioritized, and divided into tasks (using a technique called Construction Modeling). Finally these tasks are estimated and role-assigned. When we set about to divide a Value Story into these smaller parts, we will use one of a handful of these and/or other techniques that focus us all on thin, “horizontal” end-to-end paths. (So Yes, to speak briefly to agile purists, we go through this exercise of boiling all of this out up-front, for estimating purposes, whether or not these specific User Stories and tasks end up surviving as iterations unfold. More on this in a future blog.)
First, a definition of “thin,” since that is much of what gives us the “early” part of “end-to-end early.” The thinnest, smallest bit of business value our User Story can capture, the likelier we are to deliver it quickly and without defects. The quicker and better we deliver it, the quicker we can obtain real feedback from numerous stakeholders, at different steps along the delivery path (BAs, Product Managers, users, testing staff, etc). The sooner we get that feedback, the more cheaply and easily we can make course corrections And again: the sooner and cheaper we can make any kind of course correction, the more speculative work, rework, and waste we are “driving out of the system.”
By “horizontal” end-to-end, I mean a series of chronological steps. In particular, I mean the entire trip that a chunk of work must traverse in order to deliver its freight of business value, so that ROI can begin to be earned. So if XP calls the running, tested, delivered, and accepted state “Done Done,” then perhaps I mean “Done Done Done.”
Specifically, this includes (at least) all steps from the inception of a User Story, through requirements elicitation, acceptance test writing, design, development, user acceptance and verification, and production deployment. Even that set of steps is not enough, for most clients, to truly capture end-to-end. If a User Story is part of an application that must integrate with other applications (in a SOA architecture, for example) before stakeholders consider it truly “done,” then that integration step should be part of the end-to-end path.
If we cannot begin to earn ROI on a User Story until, for example, some documentation is written, some sales have occurred, and some user training is complete, then perhaps those steps too should be in our definition of “end-to-end.” If we are consulting at a level that includes mentoring both software development process and business process, then we might very well go that far in our e2e definition. We might not tick something off on the backlog until people who have nothing to do with software development have verified that our feature is in the field, saving or making money.
Designing User Stories to Traverse Architectural Layers
Technical uncertainty is project risk, pure and simple. Such risk might pounce on us at any moment from behind any piece of technology through which our data must pass (a framework, a data repository, a UI technology, an external service, you name it). To mitigate such risk, we want some of our early work on a project to deliver running, tested code that actually uses all of the risky touch points as early as possible. We want our e2e paths to give us the soonest, best possible feedback on how workable they actually are.
Martin Fowler has a great article on software architecture that wanders toward a definition of architecture that sounds like this: “architecture
is the decisions that you wish you could get right early in a project, but that you are not necessarily more likely to get right than any other.” He goes on to discuss how the least reversible things we commit to in the design, the really big important things (according to us) are what we might agree constitute the architecture. If we accept this definition, then we want to keep the cost of such irreversibility as low as possible, and e2e can help us quite a bit with that. Big, irreversible architectural decisions need not be committed to all at once, in one go. We can and should experiment, or at least explore, using e2e. If we can and must back out of such big decisions early, then let’s find a way to do it.
When we at Pillar create User Stories early in a release, part of our purpose is to have the data traverse “vertical” round trips through all of the anticipated architectural layers or touch points in a system. This is especially important when our architecture includes layers, frameworks, services, repositories, APIs, GUIs, or external dependencies about which we don’t yet know enough. Of course, such slices are not always truly topologically vertical, say, in an SOA context. The paths might explode all over the enterprise. But within an application, we’ll mostly have horizontal layers that we want to traverse vertically, as completely as possible.
If a candidate open-source Java workflow engine or CMS system or OR/M mapping framework proves unworkable for a given system, project team, or technology stack, my architectural e2e User Stories or spikes will let me know quickly enough that I can change course, making an alternate selection while that change is still possible or perhaps even cheap.
Again, I want this work to be as thin as possible, while still delivering measurable business value. Thin, small stories are also nearly always easier to estimate, plan, track, and verify. Small and thin are great good things in agility.
With these thin vertical slices, I mitigate the “irreversibility” of my architecture, and I drive more speculation out of the system, which again reduces rework, waste, and ultimately expense. Can’t use that particular workflow engine? Whew! At least we are choosing to swap it out in week 2, as opposed to week 20.
Ubiquitous e2e Example: Simple Unit Tests
E2E can apply to nearly any activity, once you are experienced at spotting speculative work (which basically means planning to deliver too big a chunk of anything, which in turn delays your feedback, and potentially incurs more rework expense for you).
A teeny-tiny example: how big should a jUnit test be? How much behavior should it specify and test? Answer: one discrete path through the “System Under Test.” This is known in Gerard Meszaros’ book, xUnit Test Patterns: Refactoring Test Code, as the Simple Test pattern, and as the “Verify One Condition per Test” principle.
But lurking beneath this testing principle is the End-to-End Early principle. By test-driving, keeping my unit test simple, and getting to green quickly, I learn quickly about (A) how workable my existing design is, (B) how difficult it is to actually do the work, (C) how long it takes me to do that work, and perhaps also things like (D) unhappy-path edge cases and exception cases I had not considered earlier. If I am pairing or conducting code reviews, then I can also get faster reaction and feedback from another pair of programmer eyes, which will doubtless catch something I missed.
This is, of course, the original test-red-green-refactor cycle from classic XP. A thin end-to-end slice is nearly always just a tiny feedback loop.
All of this has the effects that e2e always has: faster feedback, faster and cheaper course correction, lower rework cost, time and money saved.
And in this particular example, I get additional, vital benefits: my Simple Tests tend to “localize defects” really well, which makes debugging trivial; they tend to be robust (hard to break by design or requirements changes); and they tend to be easier to read and understand, for programmers new to the codebase.
Next time on this topic: some more examples, less technical examples, and a qualitative sense of how e2e “feels.”