OK, Now Let’s Discuss Microtests & TDD

Agile Programming: Lesson One (Part Three)

At the start of this series of posts I suggested that those who really, really want to become Agile Programmers start with a couple of refactoring exercises. In another post I explained why I suggest this as a starting point. I’ll reiterate here that I am suggesting that you perform those exercises several times, several ways, before you attempt the exercises coming up next.

All done?  OK. Cool. Now, three posts in, I’ll introduce the ideas of microtests and Test-Driven Development (referring you to other sources for deeper dives into these large topics).

If you have been following along in code, as well as in your reading, you may by now have discovered and learned several things. Here are a few, any one or two of which may have occurred to you:

  • You would not really be safe trying to do those original refactoring exercises without the handful of tests that cover them.
  • It’s not hard to break existing tests if you keep running them after each little refactoring step
  • The refactoring steps in the exercises can be pretty hard to do, even in those really small amounts of code.
  • As you extract behavior into lots of new methods, and perhaps into a few new classes, it would be really cool to have new, separate JUnit tests that cover those specific little behaviors.
  • It can be tricky trying to determine how to keep test code itself clean and concise.

I suspect that after enough repetitions of those first exercises I suggested, one or more of these does occur to you. Let’s dig into testing in the context of these kinds of observations.

Definition: Microtests

By Microtests, which term was coined by Mike (@GeePawHill) Hill, I mean the same thing that people like Mike Feathers and J.B. Rainsberger (and many other TDD thoughtleaders) mean by unit tests or isolation tests or programmer tests. So in Java, a JUnit test is a Microtest if it:

  • Tests a single discrete behavior/responsibility (“in isolation”)
  • Runs completely in memory: does not touch databases, GUI frameworks, file systems, or networks. Those other tests are frequently useful, but they are not microtests.
  • Runs really, really fast. Fast enough means somewhere between 10 and 100 per second, or faster.

Vital Ends: Enough Microtests for a Production Codebase

Every production codebase desperately needs to have enough microtest coverage.

Your codebase has enough microtests if your code coverage (as previously discussed) is at least 85%, and your median test-method length and production method-length is really small (as previously discussed), and each test class and test method is itself SRP-compliant, and very experienced microtesters have looked over the entire test suite and pronounced it healthy.

There are leaps of faith for you to take here, admittedly. And that last clause is really fuzzy and subjective. I get it.

Nevertheless, I’ll emphasize this:

Aside from the production code itself, there is no more important artifact for any codebase than its suite of microtests. A codebase without enough microtests, as Feathers has pointed out, is by definition Legacy Code. Agile Programmers are careful about microtests the way veteran skydivers are careful about packing parachutes.

Again, an exhaustive suite of microtests are ends, worthy in their own right, for protecting us from introducing defects when we change code.

But as is very frequently discussed in agile circles, Test-Driven Development (TDD), as a means to those ends, is important in other ways.

Definition: Test-Driven Development (TDD)

Test-Driven Development is the practice and craft of writing tiny little microtests for production code before you write the production code itself. The practice has been covered exhaustively in other sources, so I’ll only sketch it here, starting with its benefits. Again, prepare to make leaps of faith:

  • TDD is the most efficient, least-expensive way to produce high-ROI, low-TCO suites of exhaustive microtests.
  • TDD is the most efficient, least-expensive way to drive production code in the direction of high-ROI, low-TCO Clean Code.
  • TDD and microtesting have layers of sophistication, auxiliary practices and techniques, and otherwise a world of richness of their own.
  • Skilled TDD (and refactoring) completely change how you feel about programming, for the better.
  • Only after you have written a few thousand microtests using TDD (and other practices) do those first four points above become experientially clear.
  • For many programmers, TDD seems at first (perhaps for months) to be a maddening, fanatical waste of time. It feels (to some) like walking backward up a staircase, blindfolded, carrying a tray of expensive flute glasses full of expensive champagne. Senseless, in other words. But that does pass.

Yes, you can write tests last, after you have written production code. But in exercises coming up soon here, I hope to provide you an opportunity to learn that this slows you down more than you can tolerate. I’ll also explain why that seems to be true.

One more thing: it can be very, very difficult to do real TDD, or to add or change code at all, in existing codebases with very low code coverage. Before continuing with this post/lesson, if you doubt what I just said, please consider attempting this exercise I created a few years ago. Then come back here and continue.

The TDD References

First, the references. TDD basics are covered on-line pretty darned well. The more advanced, sneaky design mechanisms behind it are covered (for the advanced reader) here and here and here and here.

Aside: My fave, fave intro to TDD online is @JamesShore ‘s “Let’s Play” series. James models to us all expertise, poise, fun, and best of all, supreme openness and vulnerability. Also the desire for a pair! You can follow these videos and learn enormous amounts (the pace of learning is different from what I am attempting here — it’s OK if you find yourself lost by him pretty fast; if you revisit these next year you likely won’t be). You can also hear his cat whining about TDD in the background, if you listen carefully.

In all seriousness, in the opinion of some, James dives quickly into “TDD by the book,” which is to say with very, very little up-front design. Note that, as James proceeds, given his level of skill, this is not a serious problem for him. Master that he is, he can refactor himself out of any design misstep. This is not TDD at a novice level. Also notice in this series of videos how deeply James dives into mastering the problem domain at hand. It is his deepening mastery of that domain, coupled with his TDD mastery, that allows his design to emerge in increments that might at first seem ungainly, but are in fact elegant.

TDD is also covered in a slew of great books. Good ones in Java include (at least)

No, you need not purchase and read all of these books. Any one of them is likely sufficient to challenge you with Java TDD exercises and thinking for some time. And you need not purchase any of them this minute. But you will need at least one of them soon, both for tutorial and for reference. There is a completely different slew of books for  test-driving in other languages, BTW. [More later.]

The Classic Red-Green-Refactor TDD Cycle

The classic TDD cycle is covered well, in all seriousness, in the wikipedia TDD write-up. When you are learning TDD, taking this grand leap of faith, stick to this cycle like glue. Follow it blindly for a good long while. Internalize it.

If you need a little humor to keep you on track, you can use this handy on-line “pair partner” to remind you where you are in the cycle.

A New Rule to Follow in Our Challenges

We’ve been working in the BankOCR Kata for awhile now. We’ve been refactoring existing code. As we go, we’ve been adhering to certain rules (paraphrased here, for our next challenges):

  • Keep all tests running green as much as possible. Also try to purposefully break them sometimes, to see what kinds of coverage they give you in the code (more on that later, too).
  • Rename each entity (project, package, class, method, variable) at least twice, as you learn more and more about what it should be doing
  • By the time you are done, no method should contain more than 8 lines of code (this also applies to test methods), and most methods should be in the neighborhood of 4 or 5 lines of code, including declarations and return statements. This includes test methods.
  • Don’t let code coverage fall below 85%.

As we switch to test-driving brand-new code, the rules above still apply. To this I’ll add a new rule, for the test-driven code:

  • Test-drive classes that contain only a single public method, no more than 3 fields, and no more than 4 private helper methods. Try to avoid accessors (getters and setters). Can you avoid them? If not, why not?

Your Next Challenges

Start by ignoring the design you have been working with in the first exercises. Empty your mind of design ideas for the BankOCR Kata. Start with this new, nearly fresh version of an implementation, which can now help you with the special love provided by the Whack-A-Method testing mechanism.

Try test-driving just one part of the BankOCR Kata at a time, and in this order (for now):

  • Test-drive code that parses  a bunch of raw OCR-like String data (represented however you like) into separate OCR digits. Remember the input data looks like this:

  • How many classes does this need?  I mean, if they are really, really small, SRP-compliant classes?
  • Next, test-drive code that converts those individual OCR digits into integers. Again, how many classes do you need? How do you know?
  • Finally, test-drive code that converts groups of those OCR digits into proper account numbers.

Now, after having done that the above way, throw away all the code, and do it in reverse. Is that harder? Easier?

Remember to follow all of the above rules. And stay tuned, next, for some more deep dive into emergent design: the magic thing that happens as we begin to master TDD, Clean Method Hierarchies, and Clean Code generally.

Clean Method Hierarchies: Why Start Here?

Agile Programming: Lesson One (Part Two)

Two posts ago, I suggested a first pair of exercises for programmers who want to learn some Agile Programming off-line, which is to say, by themselves. The exercises in that first post ask you to focus on learning to refactor existing code to what I call a Clean Method Hierarchy. It asked you to make a leap of faith, trying the exercises, without first reading several more paragraphs about why I think that makes such a great starting point for learning Agile Programming. In this post, I’m trying to backfill that explanation and justification.

The Original Goals

In that first post, I suggested that the target level of cleanliness you should shoot for in the first two exercises should include these characteristics:

  • Keep all existing tests, such as they are, running green as much as possible. Also try to purposefully break them sometimes, to see what kinds of coverage they give you in the code (more on that later, too).
  • Rename each entity (project, package, class, method, variable) at least twice, as you learn more and more about what it should be doing
  • By the time you are done, no method should contain more than 8 lines of code (this also applies to test methods), and most methods should be in the neighborhood of 4 or 5 lines of code, including declarations and return statements.

Now: Why Start with Just These Exercises?

Below, I explain why it matters so much that you learn to accomplish the above ends instinctively as an agile programmer, as a software craftsman.

A Clean Class is Much Easier to Learn than Clean Everything Else

Below we’ll dive into the handful of things you need to understand deeply in order to get really, really good at keeping one class clean. Compared to what is necessary to keep an entire object model clean, a multi-tiered architecture clean, a codebase build clean, web application framework implementations clean, continuous integration systems clean, and continuous delivery systems clean (not to mention thorny subjects like exception handling and concurrency), the short list of things required to keep one class clean is pleasantly tractable.

It’s Easier Partly Because There are Only a Few Things Involved

The specific list of practices, principles, and patterns, as mentioned in the first post, includes:

In this post, let’s cover just the few of these that are most vital:

  • Code Coverage
  • Naming and Renaming
  • Basic Refactoring
  • The Extract Method refactoring
  • Keeping Modules Small
  • The SRP (Single-Responsibility Principle)
  • Cyclomatic  Complexity
  • Squeeze Class

Code Coverage

To the above list of original goals, let me now suggest that you add this item (you are going to keep working on the exercises over and over again, right?):

The definition of a Clean Class includes (contextually) something like 85% code coverage (the measured extend to which your code is protected by tests). (For some kinds of code, you’ll need higher coverage.) For Java code, you can measure code coverage  (and observe the detailed results, line by line of code) using any number of commercial and open source tools, including IDE plugins. One such useful, good-enough open-source pair of code-coverage tools for Java is Emma and Eclemma.

The reason good enough code coverage with tests is so important is that without those tests, you don’t know if you are introducing defects when you change the code.

Good enough code coverage does not just mean the coverage metric and other metrics. It mainly means that if you accidentally introduce a defect in existing code, some test somewhere that just ran green now runs red.

Does the code, as originally delivered to you, have 85% coverage?  When you first went through the exercise, did you decrease or increase the coverage, or did it stay the same? How did that happen?

The Problem with Code Coverage: It’s a One-Way Metric

If you measure that a codebase has 2% code coverage, it is guaranteed to be in danger. Increasingly, the industry-accepted definition of Legacy Code is simply “insufficient tests.”

On the other hand, if you have 90% code coverage, what does that mean? Unfortunately, in the absence of other assessment and measurement, it means nothing. Smart people have learned how to “game” the coverage metric so that they can achieve the magic, imposed high numbers with very few, very bad tests that have no real value.

Only if both code coverage numbers, median module size, and median cyclomatic complexity are reasonably good for a codebase can you begin to breathe easy about it. That’s because

If you have high enough code coverage, across a codebase that consists of very small classes and methods, you can almost always rescue it completely (if you need to).

Put another way:

If you have high code coverage and low median method-level cyclomatic complexity alone, your codebase is probably in pretty good shape.

Notice the word probably. It is still possible to test-drive codebases with 85% coverage and median method-level complexity of 2, and still have a very bad object model, very hard-to-understand names, and a slew of other non-trivial problems. But you can probably do something about those other problems, if truly you have 85% coverage and median method-level complexity of 2.

That’s much of why I emphasize mastering this small list of things first, in these first few exercises: it can make the difference between a codebase that can be rescued, and one that truly must be thrown away and rewritten.

So what about these few other things?

Naming and Renaming

The most vital, and most subjective and difficult, Clean Code practice is the art of naming and renaming. The very best writing on the subject is by Tim Ottinger (@tottinge), who covered it succinctly and beautifully in Chapter Two of Bob Martin’s Clean Code book (this is one of the first books you should buy, BTW). Tim self-describes as more of an artist than a scientist/logician, and I think that may be why he is such a virtuosic namer. Naming and renaming are art, not science. I won’t recap Tim’s material here; like I said, you’ll need to own the book anyway in order to make sense of this series of blog posts.

I’ll add my own naming practice, FWIW: Attempt to Rename Each Entity Three Times, at Each Contextual Change

  • When you first encounter an entity (class, field, constant, method, local variable) that you don’t completely understand at first glance, consider renaming it to what you suspect it actually means. Feel free, for especially pernicious names, to include modifiers like “perhaps,” “maybe,” or “WTF” to the name, in order to remind you that you know this name is not quite right.
  • The next time you encounter that same entity in the context of code change, if you have now learned more about it’s responsibility, rename it again.
  • Do this one more time (at least), the next time you find yourself working with the same entity, as the code evolves, and especially as your understanding of the underlying problem domain evolves.
  • Anytime an entity’s relationship with its clients, its delegates, its home scope, or any of its context changes, it may indeed require a new name. Is some new client calling it? Did you change its access level or method signature? Did you extract behavior from it? Did you move it to another class? Did it’s class name change, or its class’s package name? These are reasons to consider renaming it.

Basic Refactoring

The original Refactoring book by Martin Fowler does not attempt to sort refactorings in order of most common use, or in order of easiest to learn. That’s OK. Since the book was first published, the list of known, useful refactorings has continued to evolve. But learning refactorings in alphabetical order is not necessarily the most pragmatic or friendly approach.

There are a handful of refactorings that are really all you need for the scope of this series of blog posts on refactoring code to Clean Method Hierarchies. We’ll cover about a half-dozen refactorings that I believe are most useful to learn first, in coming blog posts.

For today, please consider, primarily, the Extract Method refactoring.

The Extract Method Refactoring

Extract Method, which most IDE’s will let you attempt automatically these days, is simply about taking complex methods and extracting smaller bits of named responsibility out of them in the form of new methods. It is simply letting the pregnant cat give birth to her kittens, naturally. One of its best on-line write-ups is here.

Keeping Modules Really, Really, Insanely Small

Really, really small means, in Java, median method size of around 5 lines; and median class size of around 5 methods. Insanely small means median method size of 3 or 4 lines, and median class size of 3 methods. Again, the premise is that this code has good enough code coverage.

Although there is more detail involved here (see below), and it can be frightfully hard to keep methods this small when you are not yet skillful at it, it can be astonishing what a positive difference it can make.

When modules are really, really small, it begins to become magically easier to read, understand, and modify code — especially for those who have not seen that code before, or for awhile.

Yes, this is my judgment and experience talking. It requires a leap of faith from you. Please see if it’s  true for you, after you have worked enough code into this kind of shape. And please let me know what you find.

Now onto a few deeper items behind “small modules.”

The Single-Responsibility Principle (SRP)

The Single-Responsibility Principle states that every object “should have a single responsibility, and that responsibility should be entirely encapsulated by the class.”  But the SRP applies equally to methods and modules at other levels of granularity and abstraction.

And as I was taught the SRP, every module should have a single responsibility means that its components all operate at the same level of abstraction. What the hell is a “level of abstraction”?  Indeed. This is not always easily detected and policed.

Think back to the “building a garage” example in the first post. A Clean Method Hierarchy means that at each level of the “outline,” each method is SRP-compliant at that level. This doesn’t even really begin to make sense until you have used the Extract Method refactoring to make lots and lots of methods really, really, insanely small.

One rule of thumb, however: if a method has more than a single level of indentation, it is not likely SRP-compliant: it likely includes code from at least two different levels of abstraction. And it likely has a Cyclomatic Complexity that is unnecessarily high. Nested if statements within if statements within if statements are always asking for trouble.

Cyclomatic Complexity

Cyclomatic Complexity is simple the number of discrete flow-of-control paths a module has. In Java, for example, every conditional statement or clause, every loop, every continue statement, and every other return point or exception clause constitutes another level of complexity.

Every method with low enough Cyclomatic Complexity is by definition SRP-compliant and small enough. Wherever you can, shoot for a method-level complexity of 1 or 2 (e.g., a 5-line method with a single If statement).

The reason that Cyclomatic Complexity matters so much in Agile Programming is that complex code attracts more complexity.

Code complexity is like crime in a neighborhood: it attracts more crime, which attracts more crime.

Complex code is hard code to read, hard code to test, and hard code to modify safely. It contributes to, in Bob Martin’s pattern language, the “viscosity” of code: complexity in a module, alone, can make it substantially easier to “do the wrong thing” than to “do the right thing.”

As you can read at the bottom of this wikipedia write-up, there is increasing evidence that code complexity corresponds with high defect rates. This is like high crime and general urban decay correlating. And it makes intuitive sense to agile programmers. If I cannot easily understand, test, or modify complex code, of course it provides a breeding ground for defects. Defects will tend to lurk in places where they are hard to find, verify, and fix.

We need our object models, our lower level designs, our service layers, our architectures, our builds, and our frameworks to be as simple as possible. Unnecessary complexity is ever an enemy in code as in prose (“Omit needless words.”).

But first, we need you to become expert at keeping the small stuff, the easier stuff, insanely simple. Learn to excel at it the way you would learn, as a classical pianist, to play Chopin etudes at insanely high speeds with insanely smooth flow. Learn to excel at it as you would practice military techniques in a safe bootcamp before you are thrown before enemy fire.

Regardless of the level of complexity and code coverage your current or eventual jobs permit, learn to turn the dials to 11 in these exercises, so that when you need them in a pinch, they flow like a jazz solo.

Squeeze Class

Squeeze Class is the term invented by Mike (“@GeePaw”) Hill for the algorithm the exercises in the first post make you repeat, again and again:

  • Try to extract small methods from larger ones, within a single class
  • Run into variable scope problems and control flow problems and conditional logic problems that you have to fix first
  • Try to extract methods again; this time you have more success
  • Now extract those smaller methods into even smaller methods
  • Fix, at a lower level, the same kinds of problems you encountered the first time
  • Now finish extracting those smaller methods
  • Keep going until you have extracted till you drop — you can go no further
  • Now notice, from various clues, that your class full of small methods really “wants” to be two or three or four separate classes, each with its own responsibility.

Squeeze Class is one way to refactor to Clean Method Hierarchies, SRP-compliant methods, and SRP-compliant classes.

Tracking Test Coverage and Median Complexity Together

Commercial static analysis tools like Clover will give you handy mechanisms for plotting module size against code coverage throughout a codebase. With Clover, it’s also trivial to measure and plot complexity and coverage together.  This is a fabulous, actionable measure of how your codebase is deteriorating or improving as an asset as time goes by.

But you can measure code coverage with open source tools, and some IDEs will monitor complexity for you in real-time as well.

For the exercises from the first post, I’ve provided a fun, odd, useful mechanism for monitoring size and complexity called Whack-A-Method. For code coverage, I recommend tools like Emma and Eclemma.

With and without the help of tests and tools, you will need to learn to recognize when a module is violating the SRP. Learn to spot when it is conflating and combining responsibilities that belong in different levels of abstraction. Again, in our book outline metaphor: learn when a chapter really wants to be a section, or a section really wants to be a subsection, or when a subsection needs no title at all.

Summary: Keep Your Methods Really Small, and SRP Compliant, with Low Complexity

There is a roteness and mechanistic quality to keeping methods under 5 lines, and classes under 3 or 4 methods. Paint the fence; sand the floor. That’s OK. Practice like that for awhile anyway. A lot of new learning has this rote quality while principles and deeper meaning are internalized.

If you are truly committed to learning this Agile Programming magic deeply, you will make this leap of faith like many of us have. And after awhile, you will indeed get a feel for it. You will be able to spot SRP violation.

Eventually, you will go from keeping modules small in order to approximate SRP-compliance, to mastering SRP-compliance in a way that automatically results in small modules.

And then, magical things will begin to happen in your code. Magical things that you have difficulty expressing to people sometimes, much less justifying.

And if you are like me, Extract Method and Renaming will become a way of life: I use them all the time to learn new code, to learn new problem domains, to spike problems. I use them continually in production code, test code, and toy code.

Extract Method and Renaming are two refactorings that are especially useful for learning to keep software truly soft.

Next Up: We’ll dive more deeply into microtests, and test-driving our KataBankOCR problem domain.

Agile Programming: Why Bother?

I’m working on a series of posts that outline a book’s worth of suggested curriculum on how to learn some of the technical practices that Agile Programming involves. The first post is here.

The Myth of Affordable Technical Debt

Along the way, I’ve re-discovered that too, too many software professionals still misunderstand, fundamentally, what Agile Programming and Software Craftsmanship are about, and why we bother with them. So I’m summarizing my view on that here. I’m not recapitulating the history now (later, perhaps). I’m just trying to gently make the case that sufficient long-term quality is something we can never not-afford.

In the heat of the production pressure to get a User Story out the door by hook or by crook, the discussion about “how much craftsmanship can we afford here” does keep arising (with the best intentions on all sides, no doubt). That’s my context for how and why programmers should learn deep craft. I suggest that almost no-one in software is ready to ask that question.

By Agile Programming, I Do Mean Computer Programming

My definition of Agile Programming is computer programming done at a high level of craft. (Note that in this series of posts, I am not talking about the entire superset of Agile Software Development practices and principles, some of which are technical, and some of which are not.)

Another definition of Agile Programming is Clean Code. These are ways to think about what it is. But why do we do it? It really is all about the smoothest flow of value through the code and the team. It is about highest ROI, and lowest TCO. Put another way:

Any programming practice or choice that does not make or save money for someone, somewhere, someday is not agile, and is not software craftsmanship.

Truly: Why Do we Bother? Are we Being Quality Fanatics?

We are not being fanatics. We are avoiding lots and lots of pain.

To a new regular-army combatant fresh to Guadalcanal in 1942, it may appear that the combat-veteran marines are completely obsessive-compulsive about cleaning their weapons. That’s because those fresh arrivals have not had their weapons jam frequently enough in heated enough battle in a nasty jungle. Field stripping a weapon in a way that appears obsessive and fanatic is just pain prevention. Pure and simple.

The idea of the high software craft inherent in true Agile Programming is to provide exactly, and only, the highest short-term ROI and long-term ROI, and the lowest short-term and long-term TCO for a codebase as simply another asset to be managed (around which, for example,  CAPEX and OPEX expenditures must be carefully predicted and managed these days).

Too frequently, we are asked to build straw houses when only brick houses will do, and we have no way of making it clear how false an economy this is. In a comment thread on a great blog post of a friend of mine’s, the debate rages about, essentially, the economics of Clean Code (“CC”) versus Quick and Dirty (“QD”).

Two things: I have frequently found that the advocates of QD have not bothered to learn CC deeply. They are arguing from at least partial ignorance and fear. (That is not true, BTW, of Naresh Jain, who is a very advanced Agile Programmer.) If you really are as good at Agile Programming as Josh Kerievsky, or Ron Jeffries, or Mike Hill, or Naresh Jain, then yeah, I’m willing to have User-Story-specific discussions about specific short-term “speed vs quality” tradeoffs. Otherwise, I’m not willing to have that discussion with you. All these guys have spent more than 10,000 hours test-driving. Once you’ve done that, let’s talk about CC vs QD again.

Only if you are deeply, deeply expert at test-driving clean code does it make sense for you to talk about breaking the rules. Only then do you have a feel, and deep knowledge about, the rules for breaking the rules.

The other thing is that QD experts (the Duct-Tape Programmer) are not taking long-term ROI and TCO into account fully enough. Yes, I want to ship product as regularly and quickly as I possibly can, with least thumb-twiddling. But no, I don’t want an asset in my portfolio made of duct tape and WD40. What’s the duty cycle on that thing? One release? Two?

The QD experts are arguing that QD works for a User Story, or an Iteration, or a Release. Well, yeah, you can have a roof over your head faster if you build with straw. So what? Is that a way of saying “I don’t believe in storms.” ? In the specific context of getting good-enough releases out the door for startup ventures, are we making it clear enough when we are distinguishing between prototypes designed to win investment capital, and production code designed to live for 25 releases?

The questions “How much craftsmanship can we afford for this Iteration? or this Release? or this User Story?”,  when asked about anything but spikes or prototypes, are nearly always premised in false economy.

As my friend Dave LeBlanc says: Later == Never. You will never be able to “Refactor” your straw house into a brick house. It will need to be rebuilt from scratch, eventually. Factor this into your ROI and TCO estimates. Oh, again, in the context of CC vs QD for startups, there is another sad corollary I’ve seen borne out time after time:

Every prototype ships, and becomes long-term production code. Especially for boot-strapped startups, whose investors will not agree to purchase or fund prototypes.

High software craft, Agile Programming, are not knob polishing, or gold plating. Agile Programming is about getting it right the first time, and getting it right over and over again. It is about optimizing the long-term return on an investment.

How come our codebases cannot last as long as our best cars? Why do most codebases that have not exhausted their business value end up going through expensive, dangerous rewrites?  Aren’t these rewrites fundamentally avoidable? You bet they are.

The fundamental unit of waste in software development is also the fundamental unit of denial: the gigantic codebase rewrite.

We can do much, much better than that. That’s what Agile Programming and software craft is about.

This level of dedication to quality is very much like Honda dedication to quality. It’s not about burlwood on the dashboard, or spoilers on the trunk, or pinstripes down the sides (though if you want those in the GUI, we can do that too, as in an Infiniti). It’s about a 300,000-mile-duty-cycle drivetrain, come hell or high water. It’s about the industry growing out of its addiction to rewriting business-valuable codebases after a few releases, because we can’t work them anymore.

How many 1993 Chevies does anyone in Michigan see out there on the roads, still purring along with few issues? Few. But you know what? There are still lots of 1993 Toyota Camries out there on these snowy, badly-maintained roads. Until recently I drove one every day (and I still own it as a spare-loaner-beater). It’s a little dinged up. And the AC just gave out (after 18 years!). But it also still starts in 6 degree weather, and runs smooth at 80mph, and gets 30mpg. Because those Camries were made with highest long-term ROI and lowest long-term TCO in mind.

So, throughout the series of “How To” posts, please keep this definition of the what and why of Agile Programming in mind.

Agile Programming: Lesson One (Part One)

What is Agile Programming, Anyway? Why Bother?

Great questions: I’ve tried to address them in this other post. Meanwhile, for programmers who wish to learn how to become more skillful, valuable, and excellent, the paragraphs and exercises below are for you.

The Bad News: Lots of Learning, Poorly Arranged

Agile Programming with object-oriented programming languages takes a good long while to learn to do “well enough.”

One chunk of practices at a time, or in a single chunk of total mastery, the books ask you to pick up technical practices, principles, and patterns:  TDDRefactoring. Clean Code. Etc, etc.

It’s not entirely unfair to say this:

As currently described by the various books and training available, learning Agile Programming is itself a waterfall project.

Your journey will be incremental and iterative, but the books themselves are, without intending it, phase-like. Even when they provide good tutorial, they provide it in the context of an isolated practice or set of practices. Too many books presume you have read the other books already. And some of them are truly only reference, with no real tutorial at all.

Even if the learning path were better (and a better learning path is exactly what I hope to provide here), and even if you are an experienced programmer, it takes thousands of hours to master Agile Programming well enough that you can be reliably productive on a mature, healthy agile team in the real world.

“Good enough” Agile Programming skill, by that standard, requires an amount of learning equivalent to an engineering Masters Degree, coupled with many more hours of supervised residency. You’ll need to learn the material using “breakable toy” codebases, and then (in a sense), all over again in real enterprise codebases. And it’s a moving target: the craft keeps advancing in the direction of saving us and our stakeholders time and money, which is what all software craft is intended to accomplish.

By the way, please don’t call it “TDD (Test-Driven Development),” which is one critical practice in Agile Programming, but only one practice:

Calling Agile Programming TDD is like calling a car a transmission. Trust me: you’re gonna need the whole car.

The Good News: You Can Start Small

What you want is to learn a thin slice of Refactoring, along with a thin slice of TDD, and a thin slice of Clean Code, etc. One iteration at a time, one increment at a time. I intend to help you with a learning path that looks just like that.

So, as a programmer committed to becoming skillful in Agile Programming, where do you start? I have a suggestion that I’ve been marinating in for some time now. It might not surprise you, at this point, that I do not recommend starting by learning how to Test-Drive. Instead, I recommend starting with just a bit of Refactoring. Learn how to clean up small messes. Learn how to turn fairly small Java messes into spotlessly Clean Code.

Programmers should begin by learning how to make a few methods in a couple of classes spotlessly clean, exactly because only when they are truly expert and speedy at that, will they be able to make sensible choices when faced with horrific codebases, tight deadlines, mediocre colleagues, or any two of those three.

Once you’re good at cleaning small messes, and at test-driving small chunks of code, you’ll be ready for larger messes and larger test-driving projects.

Lesson One: Refactoring to a Clean Method Hierarchy

If life throws at you a decently test-protected, somewhat ugly Java class (which won’t happen all that frequently, I admit), learn how to get it into a decent Clean Hierarchy of Composed Methods — a Clean Method Hierarchy. This is my term for a class whose hierarchy is outlined and organized into short, clear, descriptive methods at each hierarchical level.

Get good at cleaning up small messes in this specific way, THEN learn how to test-drive a replacement for the refactored code. (In a separate post, I’ll try to justify why I think this makes a good starting point; meanwhile, I’m going to ask you to make that leap of faith.)

Hierarchy: Outline Within Outline

Complex systems are naturally hierarchical, and we use outlining as a tool to help the right hierarchy emerge. Books, symphonies, organizations — all tend to have natural hierarchies. (They also tend to be more network-like than hierarchical, but that’s a concern for later.) If you were writing a book called How to Build Your Own Garage, would it’s Table of Contents start like this?

  • Buy more two-stroke fuel
  • Fuel up the big chainsaw
  • Take down any big Willow trees out back
  • Rent a stump grinder
  • Remove the tree stumps
  • Cut the trunks into logs
  • Have logs and stump debris hauled away…

Well, I hope it would not. Like any thoughtful book author, you would likely outline your book, then re-outline, then outline again.

One outline draft, translated into Java, might something like this:

But of course, few books, like few Java classes, can get away with hierarchies this shallow. Again, being the thoughtful book author you are, you would sooner or later see that another hierarchical level is trying to emerge here. It seems that buildYourGarage() really wants to be just a few methods at a larger hierarchical level than the ones above:

The rest of our hierarchy might then look like this:

Believe it or not, and like it or not, the consensus of Agile Programmers is that this is the sort of hierarchy into which we need to organize our code.

This is your ends, the way you want your individual classes to read when you are done test-driving or refactoring them. A class that is a Clean Method Hierarchy reads like a really well-written book outline, with clear and sensible hierarchical levels: chapters, sections, sub-sections, and sub-sub-sections. You can see how a good author has thought through a book’s natural hierarchy in a really well-organized Table of Contents.

Most Java code I read does not read this way. Indeed, much of it reads not like a Table of Contents, but like an Index that someone has resorted in page-number order. Like an exploded view of a car’s parts. Like the inventory list for a large hardware store after a tornado. You get the idea.

So Again, Don’t Test-Drive Clean Hierarchies First

Again, for reasons I’ll provide elsewhere, based in my coding, coaching, and training experience, I really think you just start by refactoring code that could have been test-driven to a Clean Method Hierarchy, but WAS NOT. This drives home certain principles, patterns, practices, and techniques in a way that will make your test-driving more meaningful and easier to learn later. I predict. We’ll revisit the whole idea of test-driving Clean Method Hierarchies in another post.

Some Exercises to Start With

I’ll give you the following two little code laboratories in which to practice learning how to get to a Clean Method Hierarchy state. These are inspired by Uncle Bob Martin’s Extract Till You Drop blog post. The rules for the Challenges are fairly simple:

  • Keep all existing tests, such as they are, running green as much as possible. Also try to purposefully break them sometimes, to see what kinds of coverage they give you in the code (more on that later, too).
  • Rename each entity (project, package, class, method, variable) at least twice, as you learn more and more about what it should be doing
  • By the time you are done, no method should contain more than 8 lines of code (this also applies to test methods), and most methods should be in the neighborhood of 4 or 5 lines of code, including declarations and return statements.

The “Somewhat Ugly” Challenge

Learn to start with a somewhat-ugly class (an implementation of part of a kata called the BankOCRKata), and refactor the code till it’s gorgeous. Do that lots of times (dozens), several ways. Here are three ways, right off the bat:

  1. Refactor all of the methods in the AccountNumber class, without extracting any of them to new classes. You can change variable scope however you like.
  2. Same as first way, except try not to turn method-scope variables into fields on the class. Try instead to find some other way to accomplish your method extractions.
  3. Same as first way, except extract groups of related methods to new classes however you see fit. Try to keep each class to no more than 3 public methods, and no more than 5 private helper methods. As you extract behavior to new classes, write new tests for the public methods in new test classes.

You can find the Java code for this first Challenge here, in the Pillar Technology github repository. It’s set up to work out of the box as an Eclipse project. (BTW, as I describe here, it is now wrapped up in a little testing framework called Whack-A-Method, which can help you police your compliance with the above goals as you go.)

EM-Pathy: The “Way Uglier” Challenge

Then tackle a much uglier class, and refactor it till it’s gorgeous. Again, repeat this lots of times. You can find the Java code for this second Challenge here, in the Pillar Technology github repository. It too is set up to work out of the box as an Eclipse project.

Clean Method Hierarchy: Practices and Principles Under the Hood

In other posts, there are a few things I’ll introduce you to (should you not already know them) as you learn to master this technique of refactoring classes into well-organized hierarchies. For the exercises in this post, I believe the list includes the following items (I reserve the right to revise this list as smart people give me feedback):

You can read about several of these in more detail in this following post.

If you practice understanding and using these and only these principles, patterns, and practices, you can get to a point where, indeed, your methods and classes are very clean, and you have great test coverage.

And maybe, for a little while, that’s not just OK, but awesome. At that point, you will be ready for different learning.

What Not To Learn Yet

Eventually you will have to learn agile patterns, principles, and practices that touch on dependency injection, test doubles, storytesting/acceptance-test-driven development, complex object models (Gof Design Patterns, small and large refactorings), databases and data modeling, continuous integration, version control strategies, continuous deployment, heavyweight frameworks, convention-based frameworks, concurrency, pair-programming, domain-driven design, dynamically typed vs statically typed languages, etc, etc.

But not today. In fact, not this month, nor in the coming few months. You may very well want to explore some of that material; good for you if you do.

In the meantime, though, if you really are committed to becoming expert at Agile Programming, and if you really are puzzled about where to start, why not start by mastering the art of keeping one class gorgeously clean (which may involve breaking it up into a few classes), and gorgeously tested?

You might say that with this first bit of knowledge to master, what you are really learning to conquer is modular programming. That’s fine. We’ll have more for you to learn later, but truth be told, most programmers in the software industry have not yet mastered modular programming as initially described well more than 3o years ago — and Agile Programming gives us just the tools to measure how modular our code truly is.

I guarantee you this: Agile Programming Lesson One a bite-sized bit of learning to start with, and it cannot possibly hurt you. If you get stuck in the above exercises, try reading ahead to this next bit of explanation, to see if it helps.