Dynamic Languages, Blimps, TDD, Alpha Geeks, and “Compiler as Nanny”

Nannies for Blimps!

In the old days, the compiler was your nanny, because computer resources were expensive and delicate and huge, like a giant hydrogen blimp. You want to fly the blimp, baby, you better be really good at flight plans.

So static-typing was one of several ways to prevent us from blowing up the blimp at runtime. (This metaphor may not work for too many more paragraphs, but I am flying with it for now anyway.) Static-typing is really a sort of BDUF (Big Design Up Front) enforced at the language level. It imposes design straitjackets that only become plain once a dynamic language has removed them from you. (Wow! You can do THAT in this language? Really? Look at how much less code that is.)

The Nanny is Not Groovy, Man

Extending designs in Java is genuinely hindered by static typing. It’s no longer a political issue, it’s just plain fact. And I don’t just mean because it takes 400 characters to print “Hello World.” No, I mean the kinds of shenanigans imposed on every type you inherit or use or create. Think: generics, and how long it took for Java to get them, and what a pain in the arse they are. The Nanny really is everywhere in Java, to my mind.

Bruce Tate, Stuart Halloway, Justin Gehtland, and others have made passionate, convincing argument that (for example) convention-based Rails programming as a means of getting an enterprise web app “off the ground” (so to speak) may be faster than the lightest-weight J2EE frameworks by a factor of 10. Maybe more than one factor of 10. Sheesh, that Nanny is EXPENSIVE! How would your stakeholders like it if you could produce 10 times as many web applications to solve enterprise problems than your teams can today?

The catch, of course, is Ruby and its still relative dirth of supporting libraries, frameworks, and similar open-source support. And Ruby syntax looks wonky to us old Algol-based fuddy-duddies. This is why I am personally so attracted to Groovy and Grails. (Introduced to me by Andrew Glover of Stelligent and Chris Judd at CodeMash in Ohio last month.) All the convention-based goodness, plus leverage of my Spring and Hibernate experience, and I can still use the Java stuff that the mean Nanny pounded into my head over the years. (Just the good stuff. She’s not a completely mean, insane, dictator nanny, I can now see in retrospect.)

Look Nanny, No Unit Tests! (Uh Oh.)

So off we go to a dynamic language, full of our static-typing outrage. The catch, of course, is this: because you can send a “divideYourselfByZero” message or any odd message you like to the integer 42 in languages like Smalltalk, you can get runtime errors of the sort that would curl a Java programmer’s hair.

Does it suddenly make a bit of sense why the first member of the xUnit family was, in fact sUnit? My question for the Smalltalk crew is this: before you had sUnit, how the heck did any of you keep your jobs? The production deployment problems must have been Hindenburg-spectacular. (I’m partly kidding. Actually, it turns out, the really good Smalltalkers had other tricks up their sleeves to avoid runtime disaster.)

So the bottom line is this: with dynamic languages, You Have No Choice but to develop exhaustive, requisite suites of true unit-level isolation tests. Oh, plus end-to-end tests, and several other automated test varieties. This is the equivalent of venting out all the Hydrogen and replacing it with … Helium! Yay! Doesn’t explode, a bit more expensive, for sure, and not quite as “lifty,” but way safer, and you can still fly.

With dynamic languages, you have the authority and responsibility to be an adult, not a child. No more Nanny, so no more stepping out into the street before you look both ways.

Note to Recruiters: Hire the Guys Who Know Dynamic Languages, No Matter What They Charge

Power Programmers, Alpha Geeks, the ones who some now claim in public (reasonably, I believe) can outproduce “vocational programmers” by a factor of 10 or more (there is that same math, hmm)? It turns out that one of the primary indicators of one of those guys or gals is that they just cannot keep their hands off of lots of different languages, and operating systems, and computers, and you name it. Some of them actually play banjo! They are really good at comparing entire language systems and development systems to one another. They really like dynamic languages, because they are so much faster and cleaner and better. And they really, really like unit tests, because suites of them save their behinds so frequently.

Oh, BTW, you know what true alpha geeks are all on about these days? A good old idea come full circle: functional programming. Oh, and also, BDD. Whee, here we go!

Alpha Geek example: my alpha geek pal Dimitri says “dynamic languages are so much more expressive it’s not even funny.” He says that when he noticed that Ruby does not require generics, “I almost started crying.” He says “I think the whole thing can be summarized as: static typing breeds incidental complexity”. Great quote. And he correctly points out that in the emerging world of Domain-Specific Languages (DSLs), dynamic languages are absolutely vital.

So save the really big programmer salaries for the ones who (A) know unit testing backward and forward, including TDD, (B) know multiple languages, and program avocationally and recreationally, and (C) can spout endlessly about the benefits of once-seeming exotica like dynamic languages and functional programming. That, at least, would be my agile alpha geek definition. There are other kinds of alpha geeks, certainly. In my big enterprise app world, I need the agile ones.

Hire guys like Dimitri. Make sure your team has a ratio of at least 1 alpha geek to every 3 or 4 non alpha geeks. And be the kind of boss and organization that alpha geeks love to work for and with (yet another blog topic, for another time).

And be the kind of boss who enables non-alpha geeks to find their way to alpha, if they want it. Again, another blog for another time.

The Whiteboard-Space to Wall-Space Ratio (WBS/WS)

Filed Under: Seriously Cheap Wins

Why this is true, I really do not completely understand. I want to understand it, and not judge it, but I admit I have difficulty there.

In the kinds of companies at which I have been doing agile software development consulting — coaching, mentoring, training, development — over the past few years, there is an odd trend: lots and lots of wall space, and too little whiteboard space.

I have been seeing lots and lots of conference rooms, team rooms, and miscellaneous rooms in which software development works gets done. And there are acres of wall space around. And there are tons of ideas that must be worked through collaboratively. Brainstorming that must happen, and design and architecture, and project tracking, and planning, and learning and mentoring, and training, and you name it.

Yet, there is this incredible dirth of whiteboard space. As if whiteboards were made of platinum. My favorite example of this is the very large conference room with a 20′ table that seats 24, and at the end of it, a tiny, 4′x4′ whiteboard, folded away in a little closet of its own (as if to say, “Only to be used in dire imaginative emergencies!”). Oh, and best of all, those little round whiteboard erasers maybe 3″ in diameter. They don’t so much erase as they smear.

Closely related to this: the dry-erase marker to whiteboard ratio (DEM/WB), and the dry-eraser-size to whiteboard-size ratio (DES/WBS).

How in the world do people get any creative, collaborative work done in such environments? In high-function agile teams of yore, I have seen walls covered with whiteboard stuff, and we have blithely scribbled floor to ceiling and wall to wall on it, with genuinely useful information. When I walk into a high-function team room, this is one of the things I immediately look for: huge whiteboards slathered with passionate creation and communication and clarification.

At one past engagement, 7 or so of us on a client site shared a little room the size of a large walk-in closet, with no windows, and a single 5′ square whiteboard. We positively crammed that poor board with ideas, then took digital pix of it, then erased it and crammed it with ideas again.

Our ability to think and create and collaborate in software development can literally be constrained by the whiteboard space available to us.

Coming Soon: Whiteboards On Me

I haven’t begun doing this, but I suspect I shall shortly. When I am brought to one of those conference rooms with the tiny closeted whiteboard, I shall say “Hey, I’ll work for you tomorrow for free, if you’ll let me put up 80 square feet of whiteboard on that empty wall there, at my own expense.” I’m going to start building that into my bill rate. [My fall back position will be that suggested by my pal Mike Gantz in the comment below: I'll bring in several whiteboards on wheels.]

Meanwhile, here is my contention around Whiteboard-Space to Wall-Space ratio (WBS/WS). The higher it is, the more time it takes to get things done, the more waste and rework you are likely to have, and the more, in particular, people end up communicating across one week and 50 emails what could have been handled elegantly in 5 minutes with a decent whiteboard diagramming session. Talk about muda.

Go forth, agilistas, and shrink the WBS/WS. Increase the DEM/WB, and the DES/WBS. Every room should have at least one wall where at least half the wall space is covered with whiteboard. Every whiteboard should have at least 8 markers on its little ledge per 30 square feet. And you can get these awesome extra large erasers that clean the boards faster and better. Every whiteboard should have one of those, regardless of size.

Surely this falls under the “cheap win” and “low hanging fruit” category for agile coaches everywhere.

Maybe I should just become a whiteboard consultant. Then I could wear my leather toolbelt and tools everywhere. I love to wear that thing. It’s all pockets and loops.

Client Validity, Client Validation, Code Smells, TDD & BDD

Or, BDUF & Fast OO Karmic Resolution

I’ve been chewing this one over for awhile, and it is finally ready for the world to attempt to digest it. Or something like that. (Ewww.)

Why do our most carefully conceived UML diagrams of object models of any size fall apart? Wait. More precisely, how and when do they fall apart? How can we tell whether a given class exhibits, say, the “Inappropriate Intimacycode smell, or the “Message Chains” smell, or the “Middle Man” smell?

More to my immediate point, in the context of Software Karma and Justice: If you create a class or collection of classes that inflicts such suffering on programmers who later must maintain and extend your code, then who should be the first one to suffer as a consequence? Well, trick question, of course.

You should, baby. The karmic resolution should be this fast: you create a class with bad separation of concerns or a cruddy API or rampant duplication, and you are the first person who is made to suffer as a consequence. What goes around comes immediately around and smacks you in the head, like a tetherball. Ah, would that justice flowed thus swiftly in all realms! (spoken in Robin Hood voice from “Men in Tights.”)

Test-Driving: Delivering Smells to Your Nose First

So how is this possible in object design? Only through TDD and BDD, which require that in fact you be the first person to invoke the API of your new class. The unit test you are writing is the first client of the new behavior you are creating. In fact, you have to try to invoke the production code before it exists. You can’t pay karmic debt any faster than that. Pay in advance, baby! You try to instantiate the class, set up some state for it involving some new behavior, and make some assertions on that state/behavior. And you suddenly realize: Eeeew. That’s a horrible API (or at least this seems to happen to me a lot).

Most likely, for me, the class already exists, and I need some new behavior (fast, baby, fast!). What’s the first tool I reach for? The old procedural one from my Old Coder DNA: adding a method. Doh. And what happens? I test drive that new method and realize: Eeeew. This class is getting more bloated than MS Word. And I have some nasty duplication going on.

Oh, Man. Not only does this new method not belong there, but the last two methods I added elsewhere are just as bad as this one. In fact, I am going to have to split this thing into a whole new little object tree, and I think I am going to need a Template Method. I’ll be pulling stuff up and pushing stuff down my new little tree for the next several minutes, as soon as I get this ugly test to pass.

Cause My Client Told Me So

So the tube through which this smell arises for me, I am calling “Client Validation.” My point is this: only from the perspective of the clients of a given class’s API can we really tell how bad they smell. (I’m trying not to mix two metaphors of human sense here, but dude, it’s hard.)

Only once my test shows a bit of “Primitive Obsession” or “Message Chains” do I realize, from the perspective of my client test, that other programmers will likely find this API as stinky as I do.

So, one test method at a time, one API call at a time, I find a particular class to be slightly (or horridly!) client-invalid. From the client’s perspective, that class or method or method signature stinks.

And one test method at a time, one production method at a time, one production class at a time, I repair my design, refactoring it into a state that feels, from a client-test-method’s perspective, to be valid. I end up with an API that smells valid to its client tests (and production clients).

This is what I can never smell in my UML diagrams. I can never feel (oh great! Now my metaphors include sight, smell, and touch!) whether each of these calls is “Client Valid.”

This may be old news to some brilliant old Smalltalk farts I could name, but not to me. It is a useful little heuristic for my on-going TDD journey. Maybe this will all be easier in Groovy. Hmmm.

Anyway, I am going forth now, nose held high, to sniff out Client Validity in my object models, one horrid little test method at a time. If comments are code-smell deodorant, then refactoring is code-smell Febreze. My cube will be the one that smells of Febreze. That stuff is great, you know. It can eliminate all of the cat-urine odor from an entire 6′x6′ Little Tikes play-structure (moved indoors one Winter) that apparently served as the olfactory bulletin board for every feline in Oak Park, Michigan. But that, of course, is another blog, for another time.

Continuous Refactoring and the Cost of Decay

Refactor Your Codebase as You Go, or Lose it to Early Death

Also, Scrub Your Teeth Twice a Day

Refactoring is badly misunderstood by many software professionals, and that misunderstanding causes software teams of all kinds – traditional and agile – to forgo refactoring, which in turn dooms them to waste millions of dollars. This is because failure to refactor software systems continuously as they evolve really is tantamount to a death-sentence for them.

To fail to refactor is to unwittingly allow a system to decay, and unchecked, nearly all non-trivial systems decay to the point where they are no longer extensible or maintainable. This has forced thousands of organizations over the decades to attempt to rewrite their business-critical software systems from scratch.

These rewrites, which have their own chronicles of enormous expense and grave peril, are completely avoidable. Using good automated testing and refactoring practices, it is possible to keep codebases extensible enough throughout their useful lifespans that such complete rewrites are never necessary. But such practices take discipline and skill. And acquiring that discipline and skill requires a strategy, commitment, and courage.

So, First of all: Refactoring – What is It?

The original meaning of the word has been polluted and diluted. Here are some of the “refactoring” definitions floating around:

  • Some view it as “gold-plating” – work that adds no business value, and merely serves to stroke the egos of perfectionists who are out of touch with business reality.
  • Some view it as “rework” – rewriting things that could, and should, have been written properly in the first place.
  • Others look at refactoring as miscellaneous code tidying of the kind that is “nice to have,” but should only happen when the team has some slack-time, and is a luxury we can do without, without any serious consequences. This view would compare refactoring to the kind of endless fire-truck-polishing and pushups that firemen do between fires. Busy work, in other words.
  • Still others look at refactoring as a vital, precise way of looking at the daily business of code cleanup, code maintenance, and code extension. They would say that refactoring is something that must be done continuously, to avoid disaster.

Of course, not all of these definitions can be right.

The original, and proper, definition of refactoring is that last one. Here I attempt to explain and justify that. But first let’s talk about where refactoring came from as a practice.

What problem does refactoring try to solve?

The Problem: “Code Debt” and the “Cost of Decay” Curve

What is Code Debt?

Warning: Mixed Metaphors Ahead

Veteran programmers will tell you that from day one, every system is trying to run off the rails, to become a monstrous, tangled behemoth that is increasingly difficult to maintain. Though it can be difficult to accept this unless you have seen it repeatedly firsthand, it is in fact true. No matter how thoughtfully we design up front and try to get it entirely right the first time, no matter how carefully we write tests to protect us as we go, no matter how carefully we try to embrace Simple Design, we inevitably create little messes at the end of each hour, or each day, or each week. There is simply no way to anticipate all the little changes, course corrections, and design experiments that complex systems will undergo in any period.

So enough of dental metaphors for a moment. Software decay is like the sawdust that accumulates in a cabinetmaker’s shop, or the dirty dishes and pots that pile up in a commercial kitchen – such accumulating mess is a kind of opportunity cost. It always happens, and it must be accounted for, planned for, and dealt with, in order to avoid disaster.

Programmers increasingly talk about these little software messes as “code debt” (also called “technical debt“) – debt that must be noted, entered into some kind of local ledger, and eventually paid down, because these little messes, if left unchecked, compound and grow out of control, much like real financial debt.

The Software “Cost of Decay” Curve

Years ago it was discovered that the cost of correcting a defect in software increases exponentially over time. Multiple articles, studies, and white papers have documented this “Cost of Change Curve” since the 1970′s. This curve describes how the cost of change tends to increase as we proceed from one waterfall phase to another. In other words, correcting a problem is cheapest in requirements, more expensive in design, yet more expensive in “coding,” yet more costly in testing, yet more costly in integration and deployment. Scott Ambler discusses this from an agile perspective here, talking about how some claim that agile methods generally flatten this curve. Ron Jeffries contends, alternately, that healthy agile methods like XP don’t flatten this curve, but merely insist on correcting problems at the earliest, cheapest part of it. I agree with Ron, but I claim that’s only part of how agility (and refactoring in particular) helps us with software cost of change.

There is a different (but related) exponential curve I dub the “cost of decay curve.” This curve describes the increasing cost of making any sort of change to the code itself, in any development phase, as the codebase grows more complex and less healthy. As it decays, in other words.

Whether you are adding new functionality, or fixing bugs, or optimizing performance, or whatever, the cost of making changes to your system starts out cheap in release 1, and tends to grow along a scary curve during future releases, if decay goes unrepaired. In release 10, any change you plan to make to your BigBallofMud system is more expensive than it was in release 1. In the graph-like image below, the red line shows how the cost of adding a feature to a system grows from release to release as its decay grows.

Classic cost of decay curve.

The number of releases shown here is arbitrary and illustrative — your mileage will vary. Once more, I am not talking about how, within a project, the cost of detecting and fixing a problem increases inevitably over time, as the Cost of Change curve does. I am saying that we can use the cost of any sort of change (like adding a new feature) to measure how much our increasing decay is costing us. I am using the cost of a change to measure increasing cost of decay.

Back to the dental metaphor. If, in the last few minutes of programming, I just created a tiny inevitable mess by writing 20 lines of code to get a test to pass, and if that mess will inevitably ramify and compound if left uncorrected (as is usually true), then from the organization’s perspective, the cheapest time for the organization to pay me to clean up that mess is immediately – the moment after I created it. I have reduced future change costs by removing the decay. I have scrubbed my teeth, removing the little vermin that tend to eat, multiply, defecate, and die there (I never promised a pleasant return to the metaphor — teeth are, let’s face it, gross).

Again, if a day’s worth of programming, or a week’s worth of programming, caused uncorrected, unrefactored messes to accumulate, the same logic is imposed upon us by the cost of decay curve. The sooner we deal with the messes, the lower the cost of that cleaning effort. It’s really no different than any other “pay a bit now or pay a lot later” practice from our work lives or personal lives. We really ought to scrub our teeth.

Little software messes really are as inevitable as morning breath, from a programmer’s perspective. And nearly all little software messes do ramify, compound, and grow out of control, as the system continues to grow and change. Our need to clean up the mess never vanishes – it just grows larger and larger the longer we put it off, continuously slowing us down and costing us money. But before we talk about how these little messes grow huge, helping to give that cost of decay curve it’s dramatic shape, let’s talk about the worst-case scenario: the BigBallOfMud, and the Complete System Rewrite.

Worst-Case Scenario: The BigBallOfMud, and the Complete Rewrite

Most veteran programmers, whether working in procedural or object oriented languages, have encountered the so-called BigBallOfMud pattern. The characteristics of this pattern are what make the worst legacy code so difficult or impossible to work with. These are codebases in which decay has made the cost of any change very expensive. At one shop at which I once consulted, morale was very low. Everybody seemed to be in the debugger all the time, wrestling with the local legacy BigBallOfMud. When I asked one of them how low morale had sunk, he said something like “You would need to dig a trench to find it.”

With a bad enough BigBallOfMud, the cost of the decay can be so high that the cost of adding the next handful of features is roughly the same as the cost of rewriting the system from scratch. This is a dreadfully expensive and dangerous outcome for any codebase that still retains significant business value. Total system rewrites often blow budgets, teams and careers – unplanned-for resources must be found somewhere for such huge and risky efforts. Below we revisit the cost of decay curve, adding in a blue line showing how we strive to increase our development capacity from release to release. At best, we can achieve this growth linearly, not exponentially.

BigBallOfMud! Busted!.

At the point where the two lines cross, we have our BigBallOfMud. We are out of luck for this particular system – it is no longer possible to add enough resources to maintain or extend it, nor shall it ever be again. Indeed, the cost of decay, and the cost of making any sort of change, can only continue to increase from there, until it becomes essentially infinite – change cannot be made safely or quickly enough at all.

We are then faced with a total system rewrite, because we have lost all of our refactoring opportunity, along with our ability to make any other forms of change. How many expensive, perilous total system rewrites have you seen or taken part in, in your career? How many “legacy codebases” do you know of that just could not be maintained any longer, and which had to be replaced, at great expense, by a rewrite, perhaps in a new technology or with a new approach, perhaps by a completely new team? I have personally seen several over the years. They have not all gone well.

Continue reading