In past posts, I have shared my experiences with you concerning how best to use Selenium 1 RC (Remote Control) to test web applications, based on my experiences helping teams. I even wrote a bit of framework code that I shared with you that had helped me and others minimize total-cost-of-ownership of Se test suites. Back when Se RC was one of the very few games in town for page-traversal and DOM-traversal, I found it tricky but workable. I had to write a lot of framework code to keep Se 1 RC Java code as clean as I needed it to be. But it all worked, and I could keep costs contained.

Soooo, a couple of years ago, when I tried using Se 2, I ran into several problems. And I was skeptical about how in the world three different web testing frameworks (Selenium, HTMLUnit, and WebDriver) could be successfully married. At that time I could not, in good concience, recommend to my clients that they use Se 2 and WebDriver. I was afraid that Se had become bloatware. I was afraid that the entire tool was nearing end-of-life.

SE PULLS OUT OF A NOSE DIVE

Well, I’ve been using Se 2 and WebDriver (NOT Se 2 RC) recently (in Java), as I work with Josh Kerievsky and others at Industrial Logic to develop a Selenium 2 album and workshop, and I am here to report to you that as far as I can tell, Se 2 / WebDriver rocks. End-to-end testing with tools like Se are still, in my opinion, testing options of last resort, for reasons I covered in another old post (in short, NEVER use Se to test ANYTHING you can test more affordably with another automated testing tool). But Se 2 / WebDriver in its current incarnation (I’m using selenium-server-standalone jar 2.32 as of this writing) is indeed a whole new ballgame, and deserves more attention, more traction, and more respect in the Web App testing world. Let me summarize.

In With The New

Se 2 is a marriage of Selenium, HTMLUnit (which years and years ago invented the notion of PageObjects, very quietly, simply as a matter of good OO practice), and WebDriver (which is very strong at automating browsers natively, from the operating system, in whatever way a particular browser wants to be controlled by a particular OS version). And in my recent experience, many things that used to burn me about Se RC and Se testing generally are now either way easier, or completely fixed in my current version of Se 2.

The OLD WAY: UGLY SE 1 CODE

Let’s see an example. Selenium 1 Remote Control (RC), when I asked it questions, returned me primitives: integers, strings, booleans. These primitives are all about “how to get lots of stuff done, no matter how confusing that is to try to figure out later.” Here is a classic, old-fashioned, crappy Selenium 1 RC test, with Se calls in-line in the test code (Java in this case):

public class ClassicFuglySeRCTest
    private static SeleniumServer seleniumServer;
    private static DefaultSelenium selenium;

    protected static final String SELENIUM_SERVER_HOST = "localhost";
    protected static final int SELENIUM_SERVER_PORT = 4444;	

    @Before
    public void setup() {
    	launchSeleniumBrowser();
    }

    @Test
    public void navigatePages() {
    	selenium.open("http://demo.fatfreecrm.com/login");
    	selenium.waitForPageToLoad("60000");

    	selenium.type("css=input[id=authentication_username]", "seleniumpatterns");
    	selenium.type("css=input[id=authentication_password]", "seleniumpatterns");
    	selenium.click("css=input[id=authentication_submit]");
    	selenium.waitForPageToLoad("60000");

    	assertTrue(selenium.isElementPresent("css=div[id=welcome]"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] span[id='welcome_username']"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] a[id=jumper]"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] a[href='/profile']"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] a[href='/logout']"));

    	selenium.click("css=div[id=tabs]  a:contains('Tasks')");
    	selenium.waitForPageToLoad("60000");

    	assertTrue(selenium.isElementPresent("css=div[id=welcome]"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] span[id='welcome_username']"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] a[id=jumper]"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] a[href='/profile']"));
    	assertTrue(selenium.isElementPresent("css=div[id=welcome] a[href='/logout']"));

...
   }
}

Few people ever recommended writing Se RC code like this, but too many programmers, testers, and teams wrote (and still write) Selenium RC code exactly this. There are many problems with this code. You cannot easily tell what the code is doing (in terms of the app being tested, much less its problem domain). And most of us in the Se world have seen much worse code than this, sad to say. But the main thing here, for our discussion, is that you cannot tell easily WHAT pages are being traversed, and WHY they are being traversed (for functional testing purposes). You cannot tell WHAT the test is doing. Also, the mechanics of waiting for pages to be available is embedded in the tests. As I’ve covered elsewhere, you can do a lot of things to clean up code like this. And you shouldn’t have to do so much work.

AN OBJECT ORIENTED API

Let’s hear it for a good old-fashioned Object Oriented API, in which, when I ask a framework object a question, I get returned to me a true object that skillfully encapsulates just the behavior and state I require.

These days, I’m mostly writing Se 2 / WebDriver code that looks like this:

public class WhenAUserCreatesAnAccount extends BaseFFCRMTest {
	private static final String ACCOUNT_NAME = "Bob's Bearings and BassOmatics";
	private WebElement accountsTab;
	private WebElement createAccountPanel;

	@Test
	public void theyCanVerifyThatTheyCreatedIt() throws Exception {
		accountsTab = goToAccountsTab();

		createAnAccount(ACCOUNT_NAME);
		verifyAccountCreated(ACCOUNT_NAME);

		deleteAccount(ACCOUNT_NAME);
		verifyAccountDeleted(ACCOUNT_NAME);
	}

	private void createAnAccount(String accountName) throws Exception {
		createAccountPanel = openAccountCreationPane();
		fillOutCreateAccountPanel(accountName);
	}

	private WebElement openAccountCreationPane() {
		WebElement createAccountPaneOpeningToggle = accountsTab.findElement(By
				.id("create_account_arrow"));
		createAccountPaneOpeningToggle.click();
		xdriver.assertElementPresent(By.id("create_account_title"));
		return accountsTab.findElement(By.id("create_account"));
	}

	private void fillOutCreateAccountPanel(String accountName) throws Exception {
		enterNewAccountName(accountName);
		submitForm();
	}

	private void enterNewAccountName(String accountName) {
		WebElement accountNameInputField = createAccountPanel.findElement(By
				.id("account_name"));
		accountNameInputField.sendKeys(accountName);
	}

	private void submitForm() throws Exception {
		WebElement submit = createAccountPanel.findElement(By
				.className("buttonbar"));
		submit.submit();
	}

	private void verifyAccountCreated(String accountName) {
		searchForAccount();
		WebElement accountLink = xdriver.getElementOnceNotStale(By.partialLinkText("BassOmatics"));

		accountLink.getTagName();
		accountLink.click();
		driver.findElement(By.id("edit_account_title"));
	}

	private void searchForAccount() {
		WebElement searchBox = driver.findElement(By.id("query"));
		searchBox.sendKeys(ACCOUNT_NAME);
	}

	private void deleteAccount(String accountName) {
		WebElement deleteAccountLink = driver.findElement(By.linkText("Delete?"));
		deleteAccountLink.click();
		WebElement confirmation = driver.findElement(By.linkText("Yes"));
		confirmation.click();
	}

	private void verifyAccountDeleted(String accountName) {
		searchForAccount();
		WebElement noResultsFound = driver.findElement(By.id("empty"));
		xdriver.assertElementVisible(noResultsFound);
		assertEquals("Couldn't find any accounts matching " + ACCOUNT_NAME
				+ "; please try another query.", noResultsFound.getText());
	}
}

No, I am not employing any PageObjects here, because I have not needed to. I did have to home-grow a bit of frameworkie-code to work around occasional StaleElementReferenceExceptions that too-frequently crop up when testing Ajax-rich apps in which jQuery does a lot of attaching, detaching, and re-attaching elements to the DOM. Yes, WebDriver should provide a driver-configuration API for, perhaps, returning only elements that are NOT STALE, since they are useless. This would impose an ambient load on all element-locating findElement(By …) code, so a nice complement would be an explicit as-needed call to something like findElementOnceFresh(By …). Meanwhile, I am using this workaround, adapted from various ideas on the interwebz:

public WebElement getElementOnceNotStale(By locator) {
	WebElement element = null;
	long maxTimeInMillis = timeoutInSeconds/1000; 
	long loopWaitTimeInMillis = 500; 
	long elapsedTimeInMillis = 0;

	do {
		try {
			element = driver.findElement(locator);
			flowContinuesIfNotStale(element);

			break; 
		} catch (StaleElementReferenceException sere) {
			// Element is stale; need to wait briefly and retry findElement()
		}

		try {
				Thread.sleep(loopWaitTimeInMillis);
		} catch (InterruptedException ie) {
			ie.printStackTrace();
		}

	} while (weHaventTimedOut(maxTimeInMillis, loopWaitTimeInMillis, elapsedTimeInMillis));

	return element;
}

private void flowContinuesIfNotStale(WebElement element) {
	element.getTagName();
}

private boolean weHaventTimedOut(long maxTime, long waitTime, long elapsedTimeInMillis) {
	return (elapsedTimeInMillis += waitTime) < maxTime;
}

This is illustrative. My point is that in the arms race between web-app-testing frameworks and internet application richness (great gobs of good, bad, and ugly legacy JS and jQuery, much of it un-microtested), workarounds are still inevitable.

But with Se 2, I am writing cleaner code, faster and easier. And so far, I am rolling my own PageObjects much less frequently, since WebElement objects are very handy right out of the box.

More importantly, Same source origin policy workarounds (proxy servers), and other issues resulting from Se 1 driving the browser from within the JS sandbox, are overcome entirely by Se 2 / WebDriver, which drives the browser natively from outside, in whichever way a browser/OS permutation prefers the browser be driven. This is huge.

So I am doing more with method extractions, so far, and less with entire framework classes. Since test code is not production code, and functional test code can be exceptionally expensive to maintain, less code is a very good thing.

I’ll keep you updated. But I’m going to get to know Se 2 / WebDriver really, really well.

In a recent twitter thread amongst several long-time programmers, including me, Tim Ottinger, Corey Haines, Dave Astels, and Jason Gorman, we agreed that we wished we had a collective series of pitches (blog posts, interviews in YouTube videos, a domain), about why and how we got started as programmers. And why we are still so passionate about it after years and years. And related questions. So we are launching this thing, inspired by Corey’s original “How I Got Started in Programming” interviews , and we will refine it as agilists do: via collaboration, iteration and incremental improvement.

My Personal Programming Mission

I love helping people, and programmers in particular, learn to be great. I mostly program, these days, with this mission in my mind and heart. Here’s my story.

My Dad is a Physicist. He used to do lots and lots of expensive banging of particles into each other at high speed, at various particle accelerators. Giant round electromagnets, like some James Bond special effect. (When my Dad worked at CERN, we would bike and scooter everywhere, just like the guy in the picture, back before Segways.) Lead bricks, liquid nitrogen, and wires everywhere. Radiation detection badges on every shirt.

So this relates back to programming, I swear. My Dad and his grad students were hacking around on expensive DEC boxes in the 60s to do data analysis, using Fortran and Unix shell scripts, and Nixie Tube digital readouts. I was in awe of all of this, truly. He got into the books by measuring the decay rate of the Sigma Hyperon. Back when that was totally HARD to do.

One of the many things my Dad was, and is, good at, is pulling community, inspiration, money, practical hacking, and great research ideas together. And he was especially good at helping people learn Science. My Dad is good at making Physics, and learning Physics in particular, really fun and accessible. We would go to the Physics building on a weekend, and hang out with one of Dad’s heros, Bill Vulcan, a guy who could hack together electronics to do anything. The researchers would ask Bill to cook up some odd, special, one-off device for a cyclotron run, and (this is in like 1966, mind you) Bill would just make it. My Dad loved that, and it was infectious.

Learning By Blowing Stuff Up

You want to know about gas pressure as it relates to temperature? My Dad would buy a brand new metal gas can, and put it (without the cap) into our kitchen oven preheated to 600 degrees. My Mom comes in and says “Bob! Why is the oven at 600 degrees!”  Dad says “Because I can’t get it to go any higher.”

When the new, empty gas can is glowing a nice red color, my Dad gets his welders’ gloves (important accessory for research Physicists), pulls out the can, screws on the cap, and plunges the can into a tub of water spiked with ice and dry ice (also important, as was liquid nitrogen, though we didn’t usually keep that at the house).

The gas can makes a terrifying sound, and scrunches up to about the size of a baseball. As I remember it anyway. I should try it again. Hmmmm. [Oh, safety tip: don't have your hand in the gas can handle when you plunge the hot can. It doesn't stay a handle for long.]

In a class on the Physics of electronics, Dad would walk into a class with a fully-charged-up  10,000-microfarad capacitor with the terminals running parallel to each other. It was the size of a can of coffee beans back then. And he would be wearing his safety goggles, and those welders gloves again, and the biggest flat-head screwdriver Sears makes. He would walk up to the front row of a stadium-seating classful of students, and say, “As those of you who did the reading for last week know, this is a capacitor, and capacitors hold charge.”

And then he would do the obvious thing: discharge that sucker. Sounded like 5 12-gauge shotguns going off at once. It got one’s attention.

Years later, my brothers and my Dad would experiment with heavily modified spud cannons, and Dad would determine how high we were launching spuds by timing launch to landing.

Anyway, stuff like that. All through my youth. That’s how I got addicted to helping people learn, and making learning fun.

Why I Got Into Programming

I was in college in 1978 at the University of California Santa Barbara, studying English, playing a lot of music and writing  a lot of poetry, and, ah, other stuff. Some buddies of mine and I decided to take the new programming class, because they had finally gotten green-screen terminals (of a crude sort), and a real-time OS (W00t!) for their IBM 360 clone, and the new programming intro class was taught by a well-known UCSB CS hippie, Dr Ken Klingenstein, who wore bell bottoms, and had hair like someone out of a Peter Max poster. It was Dr Ken who told me, when programming still looked scary to me, like, way too much Math, “Patrick, it turns out that poets and artists come up with great algorithms and data structures. It’s not all just Math. You should stick this out.” He was so right.

And first class, Dr Ken threw down a case of good beer on the podium up front, and said “Whoever among you can come up with a three-dimensional tic-tac-toe game that beats mine, within two weeks, gets this case of beer.” It was good German beer.

We knew right then that Ken knew how to make learning fun. We were working in WatFiv, a Fortan variant (thanks, guys). We worked our tails off, going to the CS lab between midnight and dawn to get the most terminal time. A friend of mine, Mark Gordon, devised a game that actually beat Ken’s. Ken was, unexpectedly, chagrinned. But he handed over the beer. These days Ken is an Internet bigwig. I’m unsurprised.

And the class got more interesting after that. I ended up studying CS at UC Santa Cruz (WAY more hippie professors – great stories there: the grads and undergrads and profs all just hung out together and came up with stuff to learn and teach: I was immersed in boolean math, turing machines, and on and on), and UC Berkeley (where I was taught DEC PDP-11 assembler from Dr Arthur Gill, from Scotland, who as I remember it, had written the definitive text on DEC PDP-11 assembler, and disliked humans). I worked in assembly language, Fortran, PL/1, and Pascal. I wish to goodness I had bumped into Alan Kay, but that didn’t happen. I bumped into fugly COBOL shops instead, in Walnut Creek, CA. Talk about bad luck.

Unlike many of my contemporaries, I did not program for all of my software career. I was daunted by COBOL, and drawn equally to programming and writing. So I spent years as a tech writer, then years programming in the small (hacking) as a UI/UX designer, and then a “multimedia programmer” (hacker). All this before I got serious, not that many years ago, about OO and agile programming and software craftsmanship.

Why I Still Love Programming

Well-crafted code that is demonstrably valuable, correct, defect free, and — best of all — in some way new and delightful in ways customers didn’t expect! Well. It makes one feel like a wizard. It is a kind of wizardry.

And it demands all the things I love most: creativity, collaboration, inspiration, improvisation, discipline, and continuous improvement and refinement. Above all, it demands continuous learning.

Programming really, really challenges me. I find talking and writing prose much easier, frankly. But when I truly grok key concepts in programming, I get an awesome thrill.

And when I can help someone else grok some key programming concept, I get a much bigger thrill.

Why Programming Matters

This world we have made is increasingly made of programs, most of them absolutely terrifyingly badly made. (For a wonderful summary of the history of this, and the stakes involved, read Jason Gorman’s awesome blogpost on the subject.) For a broader treatment of how declining software quality (“normalization of deviance”) relates to quality issues in other industries, see Dave Rooney’s excellent summary here.

Meanwhile the Internet, as it evolves like a vast organism, matters as much as anything humans have done, period. We will rely more and more on it for connection, for learning, for helping each other, and for evolving consciousness. Truly, it matters a great deal how well the fabric of the Internet, and all of the programs on it, are made. (I link all the time to Wikipedia, increasingly an international treasure. Do you know what that thing is made of, and how? Ouch.)

The software that is showing up in our cars sometimes crashes. I rented a Ford Edge last year whose dashboard display kept rebooting. It was ominous. Now my car can crash before it crashes.

That’s not OK. That’s pathetic. We can and must and will do better. It will take many more passionate, productive, pragmatic, disciplined, dedicated software craftsmen. It will take more creativity, collaboration, inspiration, improvisation, discipline, and continuous improvement and refinement.

And above all, continuous learning.

Why Men and Women Should Consider Programming as a Career

Programming done really well, by the best programmers I know, is awesomely fun, inspiring, and cool. The culture of the best 10,000 programmers in the world is populated by people who are perhaps 1000 times more skillful, passionate, and productive than the average programmer. (Oddly, most of them are also very good musicians. Most of them, in fact, are great guitarists. Go figure, Dr Ken.)

Programming as a craft (pursued with creativity, collaboration, inspiration, improvisation, discipline, and continuous improvement and refinement and learning) is deeply rewarding. It’s really, really fun. It makes and saves people time and money. It connects people. It enables people to have fun.

Small groups of programmers have about them the same magic as the very best teams in professional sports, the very best Special Forces military teams, the very best college faculties, the very best teams of any kind who routinely challenge each other and themselves. (Heck, you can do it throwing fish.) Groups of the best programmers are very boisterous, sometimes contentious, inspiring. Very alive. It’s almost like they are throwing fish. But ideas instead. I think you follow me.

I created this animoto video and this other one a couple of years ago from pix I took at the very first CodeRetreat. (The music for both vids, by the way, is mine, from like 16 years ago, back when I did sound tracks for CD-ROM games.)

I think about moments like that when I think about awesome programmers.

Someday, as in the novel Diamond Age, perhaps software can help make learning 1000 times better than our current “education systems,” which are paternalistic, conformist, hierarchical, performance-based, dehumanizing, mechanizing, demoralizing, and otherwise just plain bad. I really do look to programmers to help change that.

The Kind of Programming You Don’t Want to Do

You don’t want to end up with your head down in a cubicle, engaged in Microsoft Artifact Wars all day. The more time you are spending in ClearCase, Word, Excel, and (God save us all) Outlook, the more you are likely to be a software wage-slave.

That was never the only option, and it is decreasingly the only option. There is a small but powerful, growing army out there of programmers who will not give up on the true promise of programming. Programming and learning to program, like Physics and learning Physics, can be an absolute gas.

Trust me, you want to be part of something that cool, something that makes you feel that alive.

Other Blogs and Resources In this Blog Chain

As they emerge, I will link to YouTube vids, other blogs, books, and other resources that answer these (and related questions).

In the long run, as Corey points out, what we really want is to bust out of our own community into a mainstream, funded source of continuous inspiration for those who might flourish in software. Something like this might be related.

Start with this: Corey’s vids, and the Passionate Programmer by Chad Fowler.

Also, Jason Gorman’s passionate call to action along these lines: reaching out to folks and pitching programming to them as a career.

Also, check out This Developer’s Life, and this thread on old-skool programming inspiration.

[More soon...]

[Agile Programming: Lesson One, Part Four]

I’ve been pitching to the world this notion of a gradual ascent into programming excellence, starting with just a bit of refactoring the innards of classes in a small problem space, and just a bit of test-driving in the same problem space.

I ask people to do the exercises, keeping methods really, really small, and really simple. But up until now, if someone asked, Got a Test for That?, my answer, sadly, was No, Sorry, Use Static Analysis Tools – There Are Tons of Them Out There. Lame answer.

Inspired (again) at a recent CodeMash by another whack at the excellent Ruby Koans, and having heard great things about similar Scala Koans, I wanted a koans-ish learning tool for refactoring. You know what’s cool and fun about these koans? Everything, that’s what. Learning should be fun. Yes, that’s right. You heard it here first. These test-based koans are super fun.

When I asked people to keep methods under some number of lines of code, and under some cyclomatic complexity threshold, I Wanted a Test for That. And now, indeed, I do. I have two Whack-A-Method exercises; one for refactoring a sub-optimal implementation of BankOCRKata, and another for continuing to test-drive a pretty-good-so-far partial implementation of the BankOCRKata. The first exercise has a couple of really Eager end-to-end tests, but Hey! They give you 100% Code Coverage! Yay for anything!

So, these are Eclipse projects. If you have Eclemma installed, then you can pull down Coverage As > JUnit Test, and behold: you see your code coverage, of course. And in either Whack-A-Method exercise, you see the results of the tests for current production code behavior, AND YOU ALSO see results of tests that automagically recurse through your output folders for .class files, and ask a little jarred-up, embedded static analysis tool called CyVis (Tx, Guys!) to instantiate a test case for each of your classes, and instantiate a different test case for each of your methods.

Because I used the Junit 4 built-in parameterized test runner extension, you don’t get much info from the tests that run green, and you cannot (at least as yet) click through ugly classes or methods in the failures (though I’ve provided meaningful failure semantics).

But Hey!  It’s totally version 0.1, and it works OK.

Caveat Lector: Yes, We are Refactoring on Red

So, in the real world, you never want to refactor on a red bar, because you cannot tell whether you are screwing up existing behavior.

And, on the other hand, in the world of koans and learning tools, tests are an awesomely addicting, engaging learning mechanism. So, friends, consider this current Whack-a-Method a proof of concept. It’s easy to try, requires no installation, works right outa the box, and is fun.

But you must indeed distinguish between red bars in the tests that cover the production code, vs whackAmethod red bars. In this pair of exercises, it’s OK if your whackAmethod tests are red while you refactor. The other tests are saving your bacon with respect to code behavior. It’s not OK if those non-whackAmethod tests are red.

Of course, you are heading toward a green bar. And you want to get there as fast as you can, and stay there as much of the time as you can.

And ultimately this tool is likely to become plugins and extensions for editors like emacs and eclipse. Not because the world needs scads of additional source analysis plugins, but because most of them are as simple as a Saturn V rocket, and this one is dead simple. And for now, as I’ve said before, you really only need a couple of metrics.

Whack-Extract Till Green

In the refactoring Whack-A-Method exercise, you just keep extracting methods and classes until everything runs green. Easy? Try it and find out!

In the TDD Whack-A-Method exercise, you just keep test-driving your code till you have fulfilled all of the requirements in the Requirements.txt file, and you keep all the tests green as much of the time as you can.

Tweak the Thresholds, Once You Know What You’re Doing

You can tweak the constants in CleanTestBase.java to tighten or loosen the method/class size and complexity thresholds. But please don’t at first. The defaults help learning to happen.

Does it Eat its Own Dogfood?

Yes, it does. All the classes in the whackAmethod package are themselves fully Whack-A-Method compliant, given the default thresholds. Let’s see a plugin do THAT.

A Bit of 0.1 Framework Fer Ya

So, if you want to try this experiment on any Java codebase of your own, you totally can, with minimal copy, paste, config (well, as far as I know today — again, version 0.1).

1. Copy the entire whackAmethod package from my test source-folder to your own test source folder. (If you want to refactor my source, go for it, lemme know, and I’ll likely give you commit rights to the project).
2. Copy the cyvis.jar and the asm-all-2.1.jar to your project and build path.
3. Change the constant STARTING_DIRECTORY_NAME in  RecursiveClassFileFinder.java to point to the root of your output folder (it currently defaults to “bin”).
4. Run the tests. The Whack-A-Method tests should all run.
5. Contact me with any problems.

At the start of this series of posts I suggested that those who really, really want to become Agile Programmers start with a couple of refactoring exercises. In another post I explained why I suggest this as a starting point. I’ll reiterate here that I am suggesting that you perform those exercises several times, several ways, before you attempt the exercises coming up next.

All done?  OK. Cool. Now, three posts in, I’ll introduce the ideas of microtests and Test-Driven Development (referring you to other sources for deeper dives into these large topics).

If you have been following along in code, as well as in your reading, you may by now have discovered and learned several things. Here are a few, any one or two of which may have occurred to you:

• You would not really be safe trying to do those original refactoring exercises without the handful of tests that cover them.
• It’s not hard to break existing tests if you keep running them after each little refactoring step
• The refactoring steps in the exercises can be pretty hard to do, even in those really small amounts of code.
• As you extract behavior into lots of new methods, and perhaps into a few new classes, it would be really cool to have new, separate JUnit tests that cover those specific little behaviors.
• It can be tricky trying to determine how to keep test code itself clean and concise.

I suspect that after enough repetitions of those first exercises I suggested, one or more of these does occur to you. Let’s dig into testing in the context of these kinds of observations.

Definition: Microtests

By Microtests, which term was coined by Mike (@GeePawHill) Hill, I mean the same thing that people like Mike Feathers and J.B. Rainsberger (and many other TDD thoughtleaders) mean by unit tests or isolation tests or programmer tests. So in Java, a JUnit test is a Microtest if it:

• Tests a single discrete behavior/responsibility (“in isolation”)
• Runs completely in memory: does not touch databases, GUI frameworks, file systems, or networks. Those other tests are frequently useful, but they are not microtests.
• Runs really, really fast. Fast enough means somewhere between 10 and 100 per second, or faster.

Vital Ends: Enough Microtests for a Production Codebase

Every production codebase desperately needs to have enough microtest coverage.

Your codebase has enough microtests if your code coverage (as previously discussed) is at least 85%, and your median test-method length and production method-length is really small (as previously discussed), and each test class and test method is itself SRP-compliant, and very experienced microtesters have looked over the entire test suite and pronounced it healthy.

There are leaps of faith for you to take here, admittedly. And that last clause is really fuzzy and subjective. I get it.

Nevertheless, I’ll emphasize this:

Aside from the production code itself, there is no more important artifact for any codebase than its suite of microtests. A codebase without enough microtests, as Feathers has pointed out, is by definition Legacy Code. Agile Programmers are careful about microtests the way veteran skydivers are careful about packing parachutes.

Again, an exhaustive suite of microtests are ends, worthy in their own right, for protecting us from introducing defects when we change code.

But as is very frequently discussed in agile circles, Test-Driven Development (TDD), as a means to those ends, is important in other ways.

Definition: Test-Driven Development (TDD)

Test-Driven Development is the practice and craft of writing tiny little microtests for production code before you write the production code itself. The practice has been covered exhaustively in other sources, so I’ll only sketch it here, starting with its benefits. Again, prepare to make leaps of faith:

• TDD is the most efficient, least-expensive way to produce high-ROI, low-TCO suites of exhaustive microtests.
• TDD is the most efficient, least-expensive way to drive production code in the direction of high-ROI, low-TCO Clean Code.
• TDD and microtesting have layers of sophistication, auxiliary practices and techniques, and otherwise a world of richness of their own.
• Skilled TDD (and refactoring) completely change how you feel about programming, for the better.
• Only after you have written a few thousand microtests using TDD (and other practices) do those first four points above become experientially clear.
• For many programmers, TDD seems at first (perhaps for months) to be a maddening, fanatical waste of time. It feels (to some) like walking backward up a staircase, blindfolded, carrying a tray of expensive flute glasses full of expensive champagne. Senseless, in other words. But that does pass.

Yes, you can write tests last, after you have written production code. But in exercises coming up soon here, I hope to provide you an opportunity to learn that this slows you down more than you can tolerate. I’ll also explain why that seems to be true.

One more thing: it can be very, very difficult to do real TDD, or to add or change code at all, in existing codebases with very low code coverage. Before continuing with this post/lesson, if you doubt what I just said, please consider attempting this exercise I created a few years ago. Then come back here and continue.

The TDD References

First, the references. TDD basics are covered on-line pretty darned well. The more advanced, sneaky design mechanisms behind it are covered (for the advanced reader) here and here and here and here.

Aside: My fave, fave intro to TDD online is @JamesShore ‘s “Let’s Play” series. James models to us all expertise, poise, fun, and best of all, supreme openness and vulnerability. Also the desire for a pair! You can follow these videos and learn enormous amounts (the pace of learning is different from what I am attempting here — it’s OK if you find yourself lost by him pretty fast; if you revisit these next year you likely won’t be). You can also hear his cat whining about TDD in the background, if you listen carefully.

In all seriousness, in the opinion of some, James dives quickly into “TDD by the book,” which is to say with very, very little up-front design. Note that, as James proceeds, given his level of skill, this is not a serious problem for him. Master that he is, he can refactor himself out of any design misstep. This is not TDD at a novice level. Also notice in this series of videos how deeply James dives into mastering the problem domain at hand. It is his deepening mastery of that domain, coupled with his TDD mastery, that allows his design to emerge in increments that might at first seem ungainly, but are in fact elegant.

TDD is also covered in a slew of great books. Good ones in Java include (at least)

• Agile Java,
• TDD: a Practical Guide,
• TDD: By Example,
• and my all-time fave: Test Driven.

No, you need not purchase and read all of these books. Any one of them is likely sufficient to challenge you with Java TDD exercises and thinking for some time. And you need not purchase any of them this minute. But you will need at least one of them soon, both for tutorial and for reference. There is a completely different slew of books for  test-driving in other languages, BTW. [More later.]

The Classic Red-Green-Refactor TDD Cycle

The classic TDD cycle is covered well, in all seriousness, in the wikipedia TDD write-up. When you are learning TDD, taking this grand leap of faith, stick to this cycle like glue. Follow it blindly for a good long while. Internalize it.

If you need a little humor to keep you on track, you can use this handy on-line “pair partner” to remind you where you are in the cycle.

A New Rule to Follow in Our Challenges

We’ve been working in the BankOCR Kata for awhile now. We’ve been refactoring existing code. As we go, we’ve been adhering to certain rules (paraphrased here, for our next challenges):

• Keep all tests running green as much as possible. Also try to purposefully break them sometimes, to see what kinds of coverage they give you in the code (more on that later, too).
• Rename each entity (project, package, class, method, variable) at least twice, as you learn more and more about what it should be doing
• By the time you are done, no method should contain more than 8 lines of code (this also applies to test methods), and most methods should be in the neighborhood of 4 or 5 lines of code, including declarations and return statements. This includes test methods.
• Don’t let code coverage fall below 85%.

As we switch to test-driving brand-new code, the rules above still apply. To this I’ll add a new rule, for the test-driven code:

• Test-drive classes that contain only a single public method, no more than 3 fields, and no more than 4 private helper methods. Try to avoid accessors (getters and setters). Can you avoid them? If not, why not?

Your Next Challenges

Start by ignoring the design you have been working with in the first exercises. Empty your mind of design ideas for the BankOCR Kata. Start with this new, nearly fresh version of an implementation, which can now help you with the special love provided by the Whack-A-Method testing mechanism.

Try test-driving just one part of the BankOCR Kata at a time, and in this order (for now):

• Test-drive code that parses  a bunch of raw OCR-like String data (represented however you like) into separate OCR digits. Remember the input data looks like this:

_     _  _     _  _  _  _  _
| |  | _| _||_||_ |_   ||_||_|
|_|  ||_  _|  | _||_|  ||_| _|

• How many classes does this need?  I mean, if they are really, really small, SRP-compliant classes?
• Next, test-drive code that converts those individual OCR digits into integers. Again, how many classes do you need? How do you know?
• Finally, test-drive code that converts groups of those OCR digits into proper account numbers.

Now, after having done that the above way, throw away all the code, and do it in reverse. Is that harder? Easier?

Remember to follow all of the above rules. And stay tuned, next, for some more deep dive into emergent design: the magic thing that happens as we begin to master TDD, Clean Method Hierarchies, and Clean Code generally.

Two posts ago, I suggested a first pair of exercises for programmers who want to learn some Agile Programming off-line, which is to say, by themselves. The exercises in that first post ask you to focus on learning to refactor existing code to what I call a Clean Method Hierarchy. It asked you to make a leap of faith, trying the exercises, without first reading several more paragraphs about why I think that makes such a great starting point for learning Agile Programming. In this post, I’m trying to backfill that explanation and justification.

The Original Goals

In that first post, I suggested that the target level of cleanliness you should shoot for in the first two exercises should include these characteristics:

• Keep all existing tests, such as they are, running green as much as possible. Also try to purposefully break them sometimes, to see what kinds of coverage they give you in the code (more on that later, too).
• Rename each entity (project, package, class, method, variable) at least twice, as you learn more and more about what it should be doing
• By the time you are done, no method should contain more than 8 lines of code (this also applies to test methods), and most methods should be in the neighborhood of 4 or 5 lines of code, including declarations and return statements.

Now: Why Start with Just These Exercises?

Below, I explain why it matters so much that you learn to accomplish the above ends instinctively as an agile programmer, as a software craftsman.

A Clean Class is Much Easier to Learn than Clean Everything Else

Below we’ll dive into the handful of things you need to understand deeply in order to get really, really good at keeping one class clean. Compared to what is necessary to keep an entire object model clean, a multi-tiered architecture clean, a codebase build clean, web application framework implementations clean, continuous integration systems clean, and continuous delivery systems clean (not to mention thorny subjects like exception handling and concurrency), the short list of things required to keep one class clean is pleasantly tractable.

It’s Easier Partly Because There are Only a Few Things Involved

The specific list of practices, principles, and patterns, as mentioned in the first post, includes:

• Code Coverage
• Naming and Renaming
• Basic Refactoring
• Command/Query Separation
• The Extract Method refactoring
• The Introduce variable/field refactorings (and scope change refactorings generally)
• The Extract Class refactoring
• The Move Method refactoring
• The Single-Responsibility Principle (SRP)
• Keeping Modules Really, Really, Insanely Small
• Squeeze Class

In this post, let’s cover just the few of these that are most vital:

• Code Coverage
• Naming and Renaming
• Basic Refactoring
• The Extract Method refactoring
• Keeping Modules Small
• The SRP (Single-Responsibility Principle)
• Cyclomatic  Complexity
• Squeeze Class

Code Coverage

To the above list of original goals, let me now suggest that you add this item (you are going to keep working on the exercises over and over again, right?):

• Don’t let code coverage fall below 85%.

The definition of a Clean Class includes (contextually) something like 85% code coverage (the measured extend to which your code is protected by tests). (For some kinds of code, you’ll need higher coverage.) For Java code, you can measure code coverage  (and observe the detailed results, line by line of code) using any number of commercial and open source tools, including IDE plugins. One such useful, good-enough open-source pair of code-coverage tools for Java is Emma and Eclemma.

The reason good enough code coverage with tests is so important is that without those tests, you don’t know if you are introducing defects when you change the code.

Good enough code coverage does not just mean the coverage metric and other metrics. It mainly means that if you accidentally introduce a defect in existing code, some test somewhere that just ran green now runs red.

Does the code, as originally delivered to you, have 85% coverage?  When you first went through the exercise, did you decrease or increase the coverage, or did it stay the same? How did that happen?

The Problem with Code Coverage: It’s a One-Way Metric

If you measure that a codebase has 2% code coverage, it is guaranteed to be in danger. Increasingly, the industry-accepted definition of Legacy Code is simply “insufficient tests.”

On the other hand, if you have 90% code coverage, what does that mean? Unfortunately, in the absence of other assessment and measurement, it means nothing. Smart people have learned how to “game” the coverage metric so that they can achieve the magic, imposed high numbers with very few, very bad tests that have no real value.

Only if both code coverage numbers, median module size, and median cyclomatic complexity are reasonably good for a codebase can you begin to breathe easy about it. That’s because

If you have high enough code coverage, across a codebase that consists of very small classes and methods, you can almost always rescue it completely (if you need to).

Put another way:

If you have high code coverage and low median method-level cyclomatic complexity alone, your codebase is probably in pretty good shape.

Notice the word probably. It is still possible to test-drive codebases with 85% coverage and median method-level complexity of 2, and still have a very bad object model, very hard-to-understand names, and a slew of other non-trivial problems. But you can probably do something about those other problems, if truly you have 85% coverage and median method-level complexity of 2.

That’s much of why I emphasize mastering this small list of things first, in these first few exercises: it can make the difference between a codebase that can be rescued, and one that truly must be thrown away and rewritten.

So what about these few other things?

Naming and Renaming

The most vital, and most subjective and difficult, Clean Code practice is the art of naming and renaming. The very best writing on the subject is by Tim Ottinger (@tottinge), who covered it succinctly and beautifully in Chapter Two of Bob Martin’s Clean Code book (this is one of the first books you should buy, BTW). Tim self-describes as more of an artist than a scientist/logician, and I think that may be why he is such a virtuosic namer. Naming and renaming are art, not science. I won’t recap Tim’s material here; like I said, you’ll need to own the book anyway in order to make sense of this series of blog posts.

I’ll add my own naming practice, FWIW: Attempt to Rename Each Entity Three Times, at Each Contextual Change

• When you first encounter an entity (class, field, constant, method, local variable) that you don’t completely understand at first glance, consider renaming it to what you suspect it actually means. Feel free, for especially pernicious names, to include modifiers like “perhaps,” “maybe,” or “WTF” to the name, in order to remind you that you know this name is not quite right.
• The next time you encounter that same entity in the context of code change, if you have now learned more about it’s responsibility, rename it again.
• Do this one more time (at least), the next time you find yourself working with the same entity, as the code evolves, and especially as your understanding of the underlying problem domain evolves.
• Anytime an entity’s relationship with its clients, its delegates, its home scope, or any of its context changes, it may indeed require a new name. Is some new client calling it? Did you change its access level or method signature? Did you extract behavior from it? Did you move it to another class? Did it’s class name change, or its class’s package name? These are reasons to consider renaming it.

Basic Refactoring

The original Refactoring book by Martin Fowler does not attempt to sort refactorings in order of most common use, or in order of easiest to learn. That’s OK. Since the book was first published, the list of known, useful refactorings has continued to evolve. But learning refactorings in alphabetical order is not necessarily the most pragmatic or friendly approach.

There are a handful of refactorings that are really all you need for the scope of this series of blog posts on refactoring code to Clean Method Hierarchies. We’ll cover about a half-dozen refactorings that I believe are most useful to learn first, in coming blog posts.

For today, please consider, primarily, the Extract Method refactoring.

The Extract Method Refactoring

Extract Method, which most IDE’s will let you attempt automatically these days, is simply about taking complex methods and extracting smaller bits of named responsibility out of them in the form of new methods. It is simply letting the pregnant cat give birth to her kittens, naturally. One of its best on-line write-ups is here.

Keeping Modules Really, Really, Insanely Small

Really, really small means, in Java, median method size of around 5 lines; and median class size of around 5 methods. Insanely small means median method size of 3 or 4 lines, and median class size of 3 methods. Again, the premise is that this code has good enough code coverage.

Although there is more detail involved here (see below), and it can be frightfully hard to keep methods this small when you are not yet skillful at it, it can be astonishing what a positive difference it can make.

When modules are really, really small, it begins to become magically easier to read, understand, and modify code — especially for those who have not seen that code before, or for awhile.

Yes, this is my judgment and experience talking. It requires a leap of faith from you. Please see if it’s  true for you, after you have worked enough code into this kind of shape. And please let me know what you find.

Now onto a few deeper items behind “small modules.”

The Single-Responsibility Principle (SRP)

The Single-Responsibility Principle states that every object “should have a single responsibility, and that responsibility should be entirely encapsulated by the class.”  But the SRP applies equally to methods and modules at other levels of granularity and abstraction.

And as I was taught the SRP, every module should have a single responsibility means that its components all operate at the same level of abstraction. What the hell is a “level of abstraction”?  Indeed. This is not always easily detected and policed.

Think back to the “building a garage” example in the first post. A Clean Method Hierarchy means that at each level of the “outline,” each method is SRP-compliant at that level. This doesn’t even really begin to make sense until you have used the Extract Method refactoring to make lots and lots of methods really, really, insanely small.

One rule of thumb, however: if a method has more than a single level of indentation, it is not likely SRP-compliant: it likely includes code from at least two different levels of abstraction. And it likely has a Cyclomatic Complexity that is unnecessarily high. Nested if statements within if statements within if statements are always asking for trouble.

Cyclomatic Complexity

Cyclomatic Complexity is simple the number of discrete flow-of-control paths a module has. In Java, for example, every conditional statement or clause, every loop, every continue statement, and every other return point or exception clause constitutes another level of complexity.

Every method with low enough Cyclomatic Complexity is by definition SRP-compliant and small enough. Wherever you can, shoot for a method-level complexity of 1 or 2 (e.g., a 5-line method with a single If statement).

The reason that Cyclomatic Complexity matters so much in Agile Programming is that complex code attracts more complexity.

Code complexity is like crime in a neighborhood: it attracts more crime, which attracts more crime.

Complex code is hard code to read, hard code to test, and hard code to modify safely. It contributes to, in Bob Martin’s pattern language, the “viscosity” of code: complexity in a module, alone, can make it substantially easier to “do the wrong thing” than to “do the right thing.”

As you can read at the bottom of this wikipedia write-up, there is increasing evidence that code complexity corresponds with high defect rates. This is like high crime and general urban decay correlating. And it makes intuitive sense to agile programmers. If I cannot easily understand, test, or modify complex code, of course it provides a breeding ground for defects. Defects will tend to lurk in places where they are hard to find, verify, and fix.

We need our object models, our lower level designs, our service layers, our architectures, our builds, and our frameworks to be as simple as possible. Unnecessary complexity is ever an enemy in code as in prose (“Omit needless words.”).

But first, we need you to become expert at keeping the small stuff, the easier stuff, insanely simple. Learn to excel at it the way you would learn, as a classical pianist, to play Chopin etudes at insanely high speeds with insanely smooth flow. Learn to excel at it as you would practice military techniques in a safe bootcamp before you are thrown before enemy fire.

Regardless of the level of complexity and code coverage your current or eventual jobs permit, learn to turn the dials to 11 in these exercises, so that when you need them in a pinch, they flow like a jazz solo.

Squeeze Class

Squeeze Class is the term invented by Mike (“@GeePaw”) Hill for the algorithm the exercises in the first post make you repeat, again and again:

• Try to extract small methods from larger ones, within a single class
• Run into variable scope problems and control flow problems and conditional logic problems that you have to fix first
• Try to extract methods again; this time you have more success
• Now extract those smaller methods into even smaller methods
• Fix, at a lower level, the same kinds of problems you encountered the first time
• Now finish extracting those smaller methods
• Keep going until you have extracted till you drop — you can go no further
• Now notice, from various clues, that your class full of small methods really “wants” to be two or three or four separate classes, each with its own responsibility.

Squeeze Class is one way to refactor to Clean Method Hierarchies, SRP-compliant methods, and SRP-compliant classes.

Tracking Test Coverage and Median Complexity Together

Commercial static analysis tools like Clover will give you handy mechanisms for plotting module size against code coverage throughout a codebase. With Clover, it’s also trivial to measure and plot complexity and coverage together.  This is a fabulous, actionable measure of how your codebase is deteriorating or improving as an asset as time goes by.

But you can measure code coverage with open source tools, and some IDEs will monitor complexity for you in real-time as well.

For the exercises from the first post, I’ve provided a fun, odd, useful mechanism for monitoring size and complexity called Whack-A-Method. For code coverage, I recommend tools like Emma and Eclemma.

With and without the help of tests and tools, you will need to learn to recognize when a module is violating the SRP. Learn to spot when it is conflating and combining responsibilities that belong in different levels of abstraction. Again, in our book outline metaphor: learn when a chapter really wants to be a section, or a section really wants to be a subsection, or when a subsection needs no title at all.

Summary: Keep Your Methods Really Small, and SRP Compliant, with Low Complexity

There is a roteness and mechanistic quality to keeping methods under 5 lines, and classes under 3 or 4 methods. Paint the fence; sand the floor. That’s OK. Practice like that for awhile anyway. A lot of new learning has this rote quality while principles and deeper meaning are internalized.

If you are truly committed to learning this Agile Programming magic deeply, you will make this leap of faith like many of us have. And after awhile, you will indeed get a feel for it. You will be able to spot SRP violation.

Eventually, you will go from keeping modules small in order to approximate SRP-compliance, to mastering SRP-compliance in a way that automatically results in small modules.

And then, magical things will begin to happen in your code. Magical things that you have difficulty expressing to people sometimes, much less justifying.

And if you are like me, Extract Method and Renaming will become a way of life: I use them all the time to learn new code, to learn new problem domains, to spike problems. I use them continually in production code, test code, and toy code.

Extract Method and Renaming are two refactorings that are especially useful for learning to keep software truly soft.

Next Up: We’ll dive more deeply into microtests, and test-driving our KataBankOCR problem domain.

The Myth of Affordable Technical Debt

Along the way, I’ve re-discovered that too, too many software professionals still misunderstand, fundamentally, what Agile Programming and Software Craftsmanship are about, and why we bother with them. So I’m summarizing my view on that here. I’m not recapitulating the history now (later, perhaps). I’m just trying to gently make the case that sufficient long-term quality is something we can never not-afford.

In the heat of the production pressure to get a User Story out the door by hook or by crook, the discussion about “how much craftsmanship can we afford here” does keep arising (with the best intentions on all sides, no doubt). That’s my context for how and why programmers should learn deep craft. I suggest that almost no-one in software is ready to ask that question.

By Agile Programming, I Do Mean Computer Programming

My definition of Agile Programming is computer programming done at a high level of craft. (Note that in this series of posts, I am not talking about the entire superset of Agile Software Development practices and principles, some of which are technical, and some of which are not.)

Another definition of Agile Programming is Clean Code. These are ways to think about what it is. But why do we do it? It really is all about the smoothest flow of value through the code and the team. It is about highest ROI, and lowest TCO. Put another way:

Any programming practice or choice that does not make or save money for someone, somewhere, someday is not agile, and is not software craftsmanship.

Truly: Why Do we Bother? Are we Being Quality Fanatics?

We are not being fanatics. We are avoiding lots and lots of pain.

To a new regular-army combatant fresh to Guadalcanal in 1942, it may appear that the combat-veteran marines are completely obsessive-compulsive about cleaning their weapons. That’s because those fresh arrivals have not had their weapons jam frequently enough in heated enough battle in a nasty jungle. Field stripping a weapon in a way that appears obsessive and fanatic is just pain prevention. Pure and simple.

The idea of the high software craft inherent in true Agile Programming is to provide exactly, and only, the highest short-term ROI and long-term ROI, and the lowest short-term and long-term TCO for a codebase as simply another asset to be managed (around which, for example,  CAPEX and OPEX expenditures must be carefully predicted and managed these days).

Too frequently, we are asked to build straw houses when only brick houses will do, and we have no way of making it clear how false an economy this is. In a comment thread on a great blog post of a friend of mine’s, the debate rages about, essentially, the economics of Clean Code (“CC”) versus Quick and Dirty (“QD”).

Two things: I have frequently found that the advocates of QD have not bothered to learn CC deeply. They are arguing from at least partial ignorance and fear. (That is not true, BTW, of Naresh Jain, who is a very advanced Agile Programmer.) If you really are as good at Agile Programming as Josh Kerievsky, or Ron Jeffries, or Mike Hill, or Naresh Jain, then yeah, I’m willing to have User-Story-specific discussions about specific short-term “speed vs quality” tradeoffs. Otherwise, I’m not willing to have that discussion with you. All these guys have spent more than 10,000 hours test-driving. Once you’ve done that, let’s talk about CC vs QD again.

Only if you are deeply, deeply expert at test-driving clean code does it make sense for you to talk about breaking the rules. Only then do you have a feel, and deep knowledge about, the rules for breaking the rules.

The other thing is that QD experts (the Duct-Tape Programmer) are not taking long-term ROI and TCO into account fully enough. Yes, I want to ship product as regularly and quickly as I possibly can, with least thumb-twiddling. But no, I don’t want an asset in my portfolio made of duct tape and WD40. What’s the duty cycle on that thing? One release? Two?

The QD experts are arguing that QD works for a User Story, or an Iteration, or a Release. Well, yeah, you can have a roof over your head faster if you build with straw. So what? Is that a way of saying “I don’t believe in storms.” ? In the specific context of getting good-enough releases out the door for startup ventures, are we making it clear enough when we are distinguishing between prototypes designed to win investment capital, and production code designed to live for 25 releases?

The questions “How much craftsmanship can we afford for this Iteration? or this Release? or this User Story?”,  when asked about anything but spikes or prototypes, are nearly always premised in false economy.

As my friend Dave LeBlanc says: Later == Never. You will never be able to “Refactor” your straw house into a brick house. It will need to be rebuilt from scratch, eventually. Factor this into your ROI and TCO estimates. Oh, again, in the context of CC vs QD for startups, there is another sad corollary I’ve seen borne out time after time:

Every prototype ships, and becomes long-term production code. Especially for boot-strapped startups, whose investors will not agree to purchase or fund prototypes.

High software craft, Agile Programming, are not knob polishing, or gold plating. Agile Programming is about getting it right the first time, and getting it right over and over again. It is about optimizing the long-term return on an investment.

How come our codebases cannot last as long as our best cars? Why do most codebases that have not exhausted their business value end up going through expensive, dangerous rewrites?  Aren’t these rewrites fundamentally avoidable? You bet they are.

The fundamental unit of waste in software development is also the fundamental unit of denial: the gigantic codebase rewrite.

We can do much, much better than that. That’s what Agile Programming and software craft is about.

This level of dedication to quality is very much like Honda dedication to quality. It’s not about burlwood on the dashboard, or spoilers on the trunk, or pinstripes down the sides (though if you want those in the GUI, we can do that too, as in an Infiniti). It’s about a 300,000-mile-duty-cycle drivetrain, come hell or high water. It’s about the industry growing out of its addiction to rewriting business-valuable codebases after a few releases, because we can’t work them anymore.

How many 1993 Chevies does anyone in Michigan see out there on the roads, still purring along with few issues? Few. But you know what? There are still lots of 1993 Toyota Camries out there on these snowy, badly-maintained roads. Until recently I drove one every day (and I still own it as a spare-loaner-beater). It’s a little dinged up. And the AC just gave out (after 18 years!). But it also still starts in 6 degree weather, and runs smooth at 80mph, and gets 30mpg. Because those Camries were made with highest long-term ROI and lowest long-term TCO in mind.

So, throughout the series of “How To” posts, please keep this definition of the what and why of Agile Programming in mind.

Great questions: I’ve tried to address them in this other post. Meanwhile, for programmers who wish to learn how to become more skillful, valuable, and excellent, the paragraphs and exercises below are for you.

The Bad News: Lots of Learning, Poorly Arranged

Agile Programming with object-oriented programming languages takes a good long while to learn to do “well enough.”

One chunk of practices at a time, or in a single chunk of total mastery, the books ask you to pick up technical practices, principles, and patterns:  TDD. Refactoring. Clean Code. Etc, etc.

It’s not entirely unfair to say this:

As currently described by the various books and training available, learning Agile Programming is itself a waterfall project.

Your journey will be incremental and iterative, but the books themselves are, without intending it, phase-like. Even when they provide good tutorial, they provide it in the context of an isolated practice or set of practices. Too many books presume you have read the other books already. And some of them are truly only reference, with no real tutorial at all.

Even if the learning path were better (and a better learning path is exactly what I hope to provide here), and even if you are an experienced programmer, it takes thousands of hours to master Agile Programming well enough that you can be reliably productive on a mature, healthy agile team in the real world.

“Good enough” Agile Programming skill, by that standard, requires an amount of learning equivalent to an engineering Masters Degree, coupled with many more hours of supervised residency. You’ll need to learn the material using “breakable toy” codebases, and then (in a sense), all over again in real enterprise codebases. And it’s a moving target: the craft keeps advancing in the direction of saving us and our stakeholders time and money, which is what all software craft is intended to accomplish.

By the way, please don’t call it “TDD (Test-Driven Development),” which is one critical practice in Agile Programming, but only one practice:

Calling Agile Programming TDD is like calling a car a transmission. Trust me: you’re gonna need the whole car.

The Good News: You Can Start Small

What you want is to learn a thin slice of Refactoring, along with a thin slice of TDD, and a thin slice of Clean Code, etc. One iteration at a time, one increment at a time. I intend to help you with a learning path that looks just like that.

So, as a programmer committed to becoming skillful in Agile Programming, where do you start? I have a suggestion that I’ve been marinating in for some time now. It might not surprise you, at this point, that I do not recommend starting by learning how to Test-Drive. Instead, I recommend starting with just a bit of Refactoring. Learn how to clean up small messes. Learn how to turn fairly small Java messes into spotlessly Clean Code.

Programmers should begin by learning how to make a few methods in a couple of classes spotlessly clean, exactly because only when they are truly expert and speedy at that, will they be able to make sensible choices when faced with horrific codebases, tight deadlines, mediocre colleagues, or any two of those three.

Once you’re good at cleaning small messes, and at test-driving small chunks of code, you’ll be ready for larger messes and larger test-driving projects.

Lesson One: Refactoring to a Clean Method Hierarchy

If life throws at you a decently test-protected, somewhat ugly Java class (which won’t happen all that frequently, I admit), learn how to get it into a decent Clean Hierarchy of Composed Methods — a Clean Method Hierarchy. This is my term for a class whose hierarchy is outlined and organized into short, clear, descriptive methods at each hierarchical level.

Get good at cleaning up small messes in this specific way, THEN learn how to test-drive a replacement for the refactored code. (In a separate post, I’ll try to justify why I think this makes a good starting point; meanwhile, I’m going to ask you to make that leap of faith.)

Hierarchy: Outline Within Outline

Complex systems are naturally hierarchical, and we use outlining as a tool to help the right hierarchy emerge. Books, symphonies, organizations — all tend to have natural hierarchies. (They also tend to be more network-like than hierarchical, but that’s a concern for later.) If you were writing a book called How to Build Your Own Garage, would it’s Table of Contents start like this?

• Buy more two-stroke fuel
• Fuel up the big chainsaw
• Take down any big Willow trees out back
• Rent a stump grinder
• Remove the tree stumps
• Cut the trunks into logs
• Have logs and stump debris hauled away…

Well, I hope it would not. Like any thoughtful book author, you would likely outline your book, then re-outline, then outline again.

One outline draft, translated into Java, might something like this:

public void buildYourGarage() {
  chooseYourSite()
  removeAnyNearbyTreesWithPotentiallyInvasiveRoots()
  clearTheSite()
  digTheFooterTrenches()
  pourFooterAndFoundation()
  frameTheWalls()
  roofTheGarage()
  installServices()
  installTheDoor()
  finishingTouches()
}

But of course, few books, like few Java classes, can get away with hierarchies this shallow. Again, being the thoughtful book author you are, you would sooner or later see that another hierarchical level is trying to emerge here. It seems that buildYourGarage() really wants to be just a few methods at a larger hierarchical level than the ones above:

public void buildYourGarage() {
  prepareTheSite()
  pourTheFoundation()
  buildTheStructure()
  finishingTouches()
}

The rest of our hierarchy might then look like this:

private void prepareTheSite() {
  chooseYourSite()
  removeAnyNeaarbyTreesWithPotentiallyInvasiveRoots()
  clearTheSite()
}

private void pourTheFoundation() {
  digTheFooterTrenches()
  pourFooterAndFoundation()
}

private void buildTheStructure() {
  frameTheWalls()
  roofTheGarage()
  installServices()
  installTheDoor()
}

Believe it or not, and like it or not, the consensus of Agile Programmers is that this is the sort of hierarchy into which we need to organize our code.

This is your ends, the way you want your individual classes to read when you are done test-driving or refactoring them. A class that is a Clean Method Hierarchy reads like a really well-written book outline, with clear and sensible hierarchical levels: chapters, sections, sub-sections, and sub-sub-sections. You can see how a good author has thought through a book’s natural hierarchy in a really well-organized Table of Contents.

Most Java code I read does not read this way. Indeed, much of it reads not like a Table of Contents, but like an Index that someone has resorted in page-number order. Like an exploded view of a car’s parts. Like the inventory list for a large hardware store after a tornado. You get the idea.

So Again, Don’t Test-Drive Clean Hierarchies First

Again, for reasons I’ll provide elsewhere, based in my coding, coaching, and training experience, I really think you just start by refactoring code that could have been test-driven to a Clean Method Hierarchy, but WAS NOT. This drives home certain principles, patterns, practices, and techniques in a way that will make your test-driving more meaningful and easier to learn later. I predict. We’ll revisit the whole idea of test-driving Clean Method Hierarchies in another post.

Some Exercises to Start With

I’ll give you the following two little code laboratories in which to practice learning how to get to a Clean Method Hierarchy state. These are inspired by Uncle Bob Martin’s Extract Till You Drop blog post. The rules for the Challenges are fairly simple:

• Keep all existing tests, such as they are, running green as much as possible. Also try to purposefully break them sometimes, to see what kinds of coverage they give you in the code (more on that later, too).
• Rename each entity (project, package, class, method, variable) at least twice, as you learn more and more about what it should be doing
• By the time you are done, no method should contain more than 8 lines of code (this also applies to test methods), and most methods should be in the neighborhood of 4 or 5 lines of code, including declarations and return statements.

The “Somewhat Ugly” Challenge

Learn to start with a somewhat-ugly class (an implementation of part of a kata called the BankOCRKata), and refactor the code till it’s gorgeous. Do that lots of times (dozens), several ways. Here are three ways, right off the bat:

1. Refactor all of the methods in the AccountNumber class, without extracting any of them to new classes. You can change variable scope however you like.
2. Same as first way, except try not to turn method-scope variables into fields on the class. Try instead to find some other way to accomplish your method extractions.
3. Same as first way, except extract groups of related methods to new classes however you see fit. Try to keep each class to no more than 3 public methods, and no more than 5 private helper methods. As you extract behavior to new classes, write new tests for the public methods in new test classes.

You can find the Java code for this first Challenge here, in the Pillar Technology github repository. It’s set up to work out of the box as an Eclipse project. (BTW, as I describe here, it is now wrapped up in a little testing framework called Whack-A-Method, which can help you police your compliance with the above goals as you go.)

EM-Pathy: The “Way Uglier” Challenge

Then tackle a much uglier class, and refactor it till it’s gorgeous. Again, repeat this lots of times. You can find the Java code for this second Challenge here, in the Pillar Technology github repository. It too is set up to work out of the box as an Eclipse project.

Clean Method Hierarchy: Practices and Principles Under the Hood

In other posts, there are a few things I’ll introduce you to (should you not already know them) as you learn to master this technique of refactoring classes into well-organized hierarchies. For the exercises in this post, I believe the list includes the following items (I reserve the right to revise this list as smart people give me feedback):

• Code Coverage
• Naming and Renaming
• Basic Refactoring
• Command/Query Separation
• The Extract Method refactoring
• The Introduce variable/field refactorings (and scope change refactorings generally)
• The Extract Class refactoring
• The Move Method refactoring
• The Single-Responsibility Principle (SRP)
• Keeping Modules Really, Really, Insanely Small
• Squeeze Class

You can read about several of these in more detail in this following post.

If you practice understanding and using these and only these principles, patterns, and practices, you can get to a point where, indeed, your methods and classes are very clean, and you have great test coverage.

And maybe, for a little while, that’s not just OK, but awesome. At that point, you will be ready for different learning.

What Not To Learn Yet

Eventually you will have to learn agile patterns, principles, and practices that touch on dependency injection, test doubles, storytesting/acceptance-test-driven development, complex object models (Gof Design Patterns, small and large refactorings), databases and data modeling, continuous integration, version control strategies, continuous deployment, heavyweight frameworks, convention-based frameworks, concurrency, pair-programming, domain-driven design, dynamically typed vs statically typed languages, etc, etc.

But not today. In fact, not this month, nor in the coming few months. You may very well want to explore some of that material; good for you if you do.

In the meantime, though, if you really are committed to becoming expert at Agile Programming, and if you really are puzzled about where to start, why not start by mastering the art of keeping one class gorgeously clean (which may involve breaking it up into a few classes), and gorgeously tested?

You might say that with this first bit of knowledge to master, what you are really learning to conquer is modular programming. That’s fine. We’ll have more for you to learn later, but truth be told, most programmers in the software industry have not yet mastered modular programming as initially described well more than 3o years ago — and Agile Programming gives us just the tools to measure how modular our code truly is.

I guarantee you this: Agile Programming Lesson One a bite-sized bit of learning to start with, and it cannot possibly hurt you. If you get stuck in the above exercises, try reading ahead to this next bit of explanation, to see if it helps.

In the last post, we looked at self-verifying PageObjects: Java classes that are “testable representations” of a web app page, and that automatically verify for us, on instantiation, that we have indeed arrived on the actual web page we intended to.

So how do we handle page flow? I’ve handled it numerous ways in the past. I don’t like any of them anymore. Ultimately what I wanted, and now have, is something that mimics the way an actual <a href></a> takes you to a new page: classes that represent html links that, when clicked by Selenium, take you (predictably) to new PageObjects or pane objects. So the Selenium RC Java mechanics for page flow are no longer cluttering the test code, nor, in fact, the PageObjects.

Instead, I use what I call ElementObject classes. PageLink and DhtmlLink classes, in particular, behave like little Factories. When instantiated, these objects are told which PageObject to instantiate when Selenium “clicks” on the actual html links those classes represent. Whew!  Confused yet? Let’s see some code, starting with a package structure, and let’s explain ElementObjects more generally.

Background: ElementObjects

util.elements package structure

Above are the ElementObject classes I use, in HTMLUnit style, to represent discrete  html element varieties and their behaviors.

(Note: The introductory post in this  series tells you how to get all of this code. If you have the code, explore this little object tree, starting with the abstract BaseElement class. Explore how different pages and panes use these element classes.)

Let’s tour a slice of the BaseElement tree briefly. Here is the abstract BaseElement class:

package util.elements;
package util.elements;

import util.browserdriver.BrowserDriver;

public abstract class BaseElement {
  public String locator;

  public BaseElement(String locator) {
    this.locator = locator;
  }

  public final boolean isPresent() {
    return BrowserDriver.isElementPresent(locator);
  }

  public final boolean isVisible() {
    return BrowserDriver.isElementVisible(locator);
  }
}

So all elements, since they extend BaseElement, have locators (which are all CSS selectors, as discussed in the previous post). The isPresent() method (using the conventional Selenium isElementPresent() method), uses vanilla Selenium to reveal whether any element is indeed present in the browser’s representation of the HTML, rendered (usually) by a real HTTP Request/Response cycle. Similarly, isVisible() reveals whether  an element is “visible” after any dynamic call that does NOT involve a real HTTP Request/Reponse cycle. Under the covers, isVisible() uses jQuery to check visibility in the in-memory DOM. The short answer to “Why have both?” is that isVisible() is heavier-weight and slower, but usually works when isPresent() does not. The even shorter answer is that browser manufacturers hate each other, and sometimes ignore the w3c, but they all standardize on jQuery.

So, for example,  ClickableElement extends BaseElement, and adds just a predictable smidgeon of clickable-ness:

package util.elements;
package util.elements;

import util.browserdriver.BrowserDriver;

public class ClickableElement extends BaseElement {

  public ClickableElement(String locator) {
    super(locator);
  }

  public void click() {
    BrowserDriver.click(locator);
  }
}

And further down the tree, a CheckBox gives us the ability to check a checkbox, or if it is one of those fancy graphical checkbox simulacra, we can just click() on it using the inherited method in ClickableElement. Either way, we have checkbox-looking things covered.

package util.elements;

import util.browserdriver.BrowserDriver;

public class CheckBox extends ClickableElement {

  public CheckBox(String locator) {
    super(locator);
  }

  public void check() {
    BrowserDriver.check(locator);
  }
}

Notice all of the static method calls to BrowserDriver.java. We’ll cover him in deep detail later, but he is our Decorator/Facade for Selenium and jQuery calls. It provides singleton access to DefaultSelenium and several of its useful methods, as well as several convenience methods we have found useful.

The point here is that only down at the ElementObject level does our framework “know” about BrowserDriver, and the nuts-and-bolts mechanics of getting Selenium (or whatever frameworks we want) to do stuff to real web page elements.

And of course the most essential point is that in our PageObjects, we can represent every HTML element as one of these types, keeping the test framework code nice and DRY.

In general, this is how the ElementObjects work. Caveat Lector: I don’t have a full, complete set of these in my sample code; I have the ones I have been using all the time. Feel free to extend this package as necessary (and to send me your contributions).

PageLinks and DhtmlLinks

Now the meat of this post: how do we get from page to page?

First let’s look at how a PageLink client uses it for a field on a PageObject. Let’s look at a test for creating a Task in FatFreeCRM:

package task;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

import java.util.Random;

import org.junit.Before;
import org.junit.Test;

import bizdomain.common.BaseWebTest;
import bizdomain.common.Common;
import bizdomain.pages.TasksPage;
import bizdomain.panes.CreateTaskPane;
import bizdomain.panes.DueTodayTaskStrip;

public class CreateTaskTest extends BaseWebTest {
  private TasksPage tasksPage;
  private String taskName;

  @Before
  public void setup() {
    tasksPage = Common.mainNavTabsSet.tasks.clickToNewPage();
    taskName = createRandomTaskName();
  }

  @Test
  public void canCreateAndCompleteLunchTodayTask() throws Exception {
    assertTrue(tasksPage.noPendingTasksLabel.isVisible());
    CreateTaskPane createTaskPane = tasksPage.createTaskLink.clickToNewContainer();

    DueTodayTaskStrip taskStrip = createTaskPane.createTask("Today", "Lunch", taskName);
    assertFalse(tasksPage.noPendingTasksLabel.isVisible());

    String expectedCategory = "Lunch";
    assertTrue(taskStrip.taskNameLabel.getText().contains(taskName));
    assertEquals(taskStrip.whenLabel.getText(), expectedCategory);

    taskStrip.clickCompleteBox();
    assertFalse(taskStrip.isVisible());
  }

  private String createRandomTaskName() {
    Random intGenerator = new Random(System.currentTimeMillis());
    int randomInt = intGenerator.nextInt();
    return "testTask" + randomInt;
  }

}

Note in the setUp() method that we get to a TasksPage by clicking clickToNewPage() on the tasks field on a MainNavigationTabsSet class, which looks like this:

package bizdomain.panes;

import util.BasePane;
import util.elements.PageLink;
import bizdomain.pages.AccountsPage;
import bizdomain.pages.CampaignsPage;
import bizdomain.pages.ContactsPage;
import bizdomain.pages.DashBoard;
import bizdomain.pages.LeadsPage;
import bizdomain.pages.OpportunitiesPage;
import bizdomain.pages.TasksPage;

public class MainNavigationTabsSet extends BasePane {
  public static final String PANE_IS_LOADED_CSS = "div[id=tabs]";

  public PageLink dashBoard;
  public PageLink tasks;
  ...

  public MainNavigationTabsSet() {
    dashBoard = PageLink.create(
        PANE_IS_LOADED_CSS + " a:contains('Dashboard')", DashBoard.class);
    tasks = PageLink.create(PANE_IS_LOADED_CSS + " a:contains('Tasks')",
        TasksPage.class);
    ... 
 }

  @Override

  public String getPageLoadedCssLocator() {
    return PANE_IS_LOADED_CSS;
  }

}

Note that tasks field is of type PageLink, parameterized with a class. And notice that when you instantiate a PageLink, you call a PageLink.create() method. Between these two things, we ensure that when we ask Selenium to click on that tasks link, we end up instantiating an actual TasksPage PageObject.

It’s a smidge confusing, perhaps. Bear with me. Here is the PageLink class:

package util.elements;

import util.browserdriver.BrowserDriver;

public class PageLink<T> extends ClickableElement {
	private Class<T> clazz;

	public PageLink(String locator, Class<T> clazz) {
		super(locator);
		this.clazz = clazz;
	}

	public T clickToNewPage() {
		try {
			click();
                        BrowserDriver.waitForPageToLoad( 
                              BrowserDriver.STANDARD_PAGE_LOAD_WAIT_TIME);
			return clazz.getConstructor().newInstance();
		} catch(Exception e) { 
			throw new RuntimeException(e);
		}
	}
}

Notice that PageLink extends ClickableElement, so whatever element a PageLink object represents, we can click on it. Yay! Notice also that when we instantiate PageLink, it expects a css locator.

Most critically and cryptically, notice all the Java generics magic, which can be confusing Java syntax to many programmers. The upshot is noisy framework code, but it keeps explicit casting out of the test code. It keeps the test code very simple indeed.

Now notice what happens when a test method calls the clickToNewPage() method: first, we click on ourselves (which will, if you look at click() in ClickableElement, get BrowserDriver to click on the locator we were instantiated with. Then the tricky bits. We call clickToNewPage(), which asks BrowserDriver.waitForPageToLoad() to let us know when the new page is done loading. Because Selenium can tell when a real HTTP Request/Response cycle is done, this will work for real pages with actual, different urls.

Then, aha! Our little Factory method uses a little lightweight Java Reflection to construct and return us an instance of that PageObject whose Class we passed in on instantiation. In MainNavigationTabsSet above, you can see that the tasks PageLink is declared and instantiated with TasksPage.class. So indeed, a self-verifying TasksPage PageObject gets constructed and handed back to test code when that test calls Common.mainNavTabsSet.tasks.clickToNewPage().

So a bit of Java design forethought gives us this ability to click on a tasks link, and proceed without further ceremony to a TasksPage. Our page flow does, in fact, much mirror the HTML and pages in the production code.

Our test code (above) is, again, succinct and expressive, and only a link to a page need know how to get there. Just like in real life.

Similar to PageLink, DhtmlLink provides us a little factory for dynamic html behaviors (clickToNewContainer()), whether they have Ajax-like server conversations, or just change the html in the browser. In this case, because we don’t have a real HTTP Request/Response cycle, we don’t call waitForPageToLoad() on Selenium. Selenium has no idea whether the “page” or “pane” we need is now visible. Our PageObjects take care of that. We just click(). Like PageLink, DhtmlLink uses much of the same Java generics gobbletygook to keep track of concrete Class types to be instantiated at our request. In this case, the base class is BasePane, not BasePage:

package util.elements;

import util.BasePane;

public class DhtmlLink<T> extends ClickableElement {
	private Class<T> clazz;

	public DhtmlLink(String locator, Class<T> clazz) {
		super(locator);
		this.clazz = clazz;
	}

	public T clickToNewContainer() {
		try {
			click();
			return clazz.getConstructor().newInstance();
		} catch (Exception e) {
			throw new RuntimeException(e);
		}
	}
}

Next up: what the heck is this BrowserDriver thing, and how might we use it?

What we Want: Expressive, Succinct Tests

Let’s say I want to test that I can log into an app. I want my Selenium RC Java test code to look something like this, because I want it to read, aloud, the way I would describe actually logging in and checking that everything went mostly OK:

@Test
public void canLoginToFatFreeCRM() throws Exception {
  LoginPage loginPage = new LoginPage();
  DashBoard homePage = loginPage.login(BrowserDriver.DEFAULT_USERNAME,
                          BrowserDriver.DEFAULT_PASSWORD);

  assertTrue(homePage.isLoaded());
}

Several techniques make this possible, but the one we will focus on in this blog post is the notion of a self-verifying PageObject, which is essentially a “testable representation” of an actual web app page we wish to traverse, manipulate, and test. So my LoginPage class, above, allows me succinct manipulation of the corresponding login page in the system under test.

Note that this test knows nothing about Selenium. Indeed, neither does the LoginPage class (below). More on that later. Note that this test does not include any waitUntilPageIsFrickinLoaded() calls, nor (worse) Thread.sleep() code, and very little direct instantiation of PageObjects. The page flow mechanics is encapsulated elsewhere. Note also that as I go from page to page, I don’t explicitly assert that I got there. I shouldn’t have to. My test should be able to take that for granted. It turns out that assertions are being made under the covers about having arrived successfully on intermediate pages, but that code is abstracted away. That’s what this post covers.

The only useful assertion in the test proper is that, once I have logged in with proper credentials, I arrive safely at the homepage (and even this assertion in the test is redundant, and included here only to reveal intention; the DashBoard page is also self-verifying).

Glance-Readability

A test as short and expressive as the one above passes the “glance readability” test. Anyone familiar with automated testing semantics, and the biz language of this CRM domain, ought to be able to grasp what this test does at a single glance.

Suffice to say, we want our Selenium RC tests, as much as they can, to be this succinct. And, again, that requires that they stick to expressing page traversal and manipulation in the local biz-specific language of the web app’s business domain. So, finally, let’s see the under-the-covers page self-verification mechanics.

PageObjects as I Use Them

The selenium-rc-patterns sample Java codebase that illustrates all of the patterns in this series of posts talks to a localhost copy of a Rails project called Fat Free CRM. If you want to learn a bit about this app, you can play with a hosted sample version here (you have to sign up for an account first, which is easy).

This CRM system has discrete web pages like a Dashboard (home page, essentially), and pages for Tasks, Leads, Campaigns, Contacts, Accounts, and Opportunities. For each of these actual, production pages, my selenium-rc-patterns Eclipse project contains a matching PageObject with methods and fields that provide access to the services on that page.

Above, when you ask an instance of a LoginPage to login(username, password), then the TextFields and PageLink type know how to return us a PageObject that should then be cast to a DashboardPage and returned. (More on that in another post.)

Selenium RC developers have been using a PageObject pattern for awhile. The pattern dates back, at least, to the original HTMLUnit, whose API rather strongly encourages you to represent your pages under test as extensions of what it calls an HtmlPage. I don’t use Se 2 yet (might someday, might not), so I don’t use its PageFactory pattern.

Instead, I use my own PageObjects that extend a BasePage (or BasePane), partly so that I can control how and when PageObjects are instantiated, and verify automatically and ambiently on instantiation that Selenium is indeed on that page. Again, that’s much of what keeps the above test code so simple: I’m not explicitly waiting for a page to load, and I’m not asserting all over the place that I’ve successfully arrived on a given page.

So, my LoginPage class has a login() method that accepts a username and password (we’ll cover the return types of these fields and methods in another post that describes how page-flow works):

package bizdomain.pages;

import util.BasePane;
import util.elements.PageLink;
import util.elements.TextField;

public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS = "input[id=authentication_username]";

  private TextField userNameField;
  private TextField passwordField;
  private PageLink<DashBoard> loginButton;

  public LoginPage() {
    super();
    userNameField = new TextField("input[id=authentication_username]");
    passwordField = new TextField("input[id=authentication_password]");
    loginButton = PageLink.create("input[id=authentication_submit]",
        DashBoard.class);
  }

  @Override
  public String getPageLoadedCssLocator() {
    return PAGE_IS_LOADED_CSS;
  }

  public DashBoard login(String userName, String password) {
    userNameField.enter(userName);
    passwordField.enter(password);
    return loginButton.clickToNewPage();

  }
}

Verifying That the Production Page has Been Loaded

The real point of my flavor of PageObject pattern is the self-verifying bit. Let’s dive into that — not later, but now.

Note that each of these PageObjects extends BasePane, and has a PAGE_IS_LOADED_CSS constant:

package bizdomain.pages;
public class LoginPage extends BasePane {
  public static final String PAGE_IS_LOADED_CSS = "input[id=authentication_username]";

In the LoginPage constructor way above up there, you can see we first explicitly call super() on BasePane. Here is BasePane:

public abstract class BasePane {
  public BasePane() {
    waitUntilLoaded();
  }

  public final boolean isLoaded() {
    return BrowserDriver.isElementPresent(getPageLoadedCssLocator());
  }

  public final boolean isVisible() {
    return BrowserDriver.isElementVisible(getPageLoadedCssLocator());
  }

  public final void waitUntilLoaded()  {
    BrowserDriver.waitForElementVisible(getPageLoadedCssLocator());
  }

  public abstract String getPageLoadedCssLocator();
}

You can see that this constructor calls waitUntilLoaded(), which makes a static call to a method on our BrowserDriver (the Facade / Decorator that handles all of the actual Selenium calls) in order to loop until our LoginPage (in this case) actually is loaded. The argument supplied is the result of calling getPageLoadedCssLocator().

But wait! That’s an abstract method! So Yes, we have something very like a template method pattern here in waitUntilLoaded(): the concrete implementation of getPageLoadedCssLocator() on LoginPage returns that PAGE_IS_LOADED_CSS constant String.

Deep, deep under the covers, the BrowserDriver.waitForElementIsVisible() method looks like this:

public static void waitForElementVisible(String cssLocator)  {
  for (int second = 0;; second++) {
    if (second >=
 Integer.valueOf(BrowserDriver.STANDARD_DHTML_LOAD_WAIT_  TIME) / MS_PER_SECOND)

  fail("Timeout waiting for element " + cssLocator + " to become visible.");
    if (isElementVisible(cssLocator)) break;
    sleepForASecond();
  }
}

public static boolean isElementVisible(String cssLocator) {
  injectJqueryIfAbsent();
  return executeJavascript(JqueryCodeFactory.getVisibilityCode(cssLocator)).equals(TRUE);
}

We’ll discuss the BrowerDriver class at length elsewhere. And we’ll also discuss the injectJqueryIfAbsent() method, which we hope an upcoming Selenium RC release will obviate.

The upshot of the waitForElementVisible() method and the LoginPage and BasePane code above is that the LoginPage object will not successfully finish instantiating until Selenium can successfully verify that a unique element on that page has been loaded. In other words, once a LoginPage instance is loaded in memory, we actually are on the LoginPage, by definition, as long as that CSS element selector syntax is correct. Voila, automatic, ambient page flow assertion.

This is much of how we get our test methods so succinct. The PageObjects take care of verifying for us, at instantiation, that we have safely arrived on their corresponding production app web pages.

Caveat Lector: there is a flaw in my code I have yet to squeeze out:  duplication between a BasePane, used above, which I usually use for dynamic changes in the HTML, and a BasePage, which presumes that a real HTTP Request/Response cycle has occurred. I will collapse those two together shortly.

Next: we’ll talk about reusable HTML Element objects, and how the linkish ones know, in my code, how to return the PageObject we wish to traverse to next.

I’ve blogged in the past about low-TCO (Total Cost of Ownership) Selenium RC testing in general, and about  how much of your automated testing budget should be allocated to different kinds of testing at different points in an agile transformation. Of late, I have been helping teams learn to use Selenium RC, in Java, to test web apps through their GUIs. Once the above Caveat Tester is spoken, I help them do the Selenium RC testing that they feel they MUST do as well as possible.

I have been diving more deeply into it for several clients recently, and finding and refining some useful patterns I had not needed or found before (with the help of several colleagues). And, admittedly, some of what I am doing is refining my old patterns in response to things like Refactoring pressure. In general, I am devising increasingly DRY Object Oriented Selenium RC Java patterns. I recently presented on them at Agile 2010.  Using git, you  can checkout the code that illustrates these patterns from github here.

I intend to share each of these patterns with you, each in its own blog post, in coming weeks.

So watch this space for blog posts covering the following topics:

• Clean, clear test methods
• Separating DRY test-framework code from “wetter” tests, and separating reusable, generic test-framework code from biz-domain-specific framework code
• Wrapping Selenium and other frameworks in Facade/Decorator that provides static Singleton test framework access.
• Pushing all framework static Singleton references down into base classes and html element classes.
• Biz-Domain-specific Page Objects that are “lazy loaded” and self-verifying, and that encourage Dry OO test code
• Always using CSS as a DOM element locator strategy; never using XPath.
• Injecting the entire jQuery framework into pages under test, in order to verify visibility of elements that appear after dynamic html or Ajax calls.
• Generic HTML Element objects (ala HtmlUnit’s API) that encourage Dry OO test code
• DhtmlLink and PageLink Factory classes that are instantiated with the Page Object classes they should “go to,” and that automatically return those objects when “clicked”.
• Pushing out access to common elements across several pages to static Singletons on base page classes.

The code I link to above illustrates all of these patterns, expecting (for now) to run against a demo instance of a Rails app called FatFreeCRM, which is available for you to play with at http://demo.fatfreecrm.com/login.