Enterprise Software Blight
I get hired to help teams learn agile software development practices. Most of the practices in my tool bag — not all, but most — come from experience, books, articles, blogs, conferences that focus mainly on greenfield development. And as an agile consultant pal of mine, Mike Hill, says, “First step when you are digging a hole: Stop Digging!” Turning around how we launch greenfield projects, and the standards of craft, quality, feedback, accountability, and ROI we establish for them — Hey, that’s obviously all good. Most teams, most enterprise are, in fact, still digging everytime they launch a new project. Still making bad enterprise situations much, much worse with more stinky code.
But they are making things worse in more ways than my favorite tools reveal. And perhaps I have been, we have all been, focusing on the wrong kinds of damage. That’s what I want to explore here.
We have spent a number of years trying to help enterprises learn how to, at least, stop digging holes in the object model, in the architecture, in how the team works.
The thing is, these tools in our toolbags really do work best in greenfield situations. Meanwhile greenfield opportunities seem to be slowly drying up. Over the last 10+ years, the software best practices community has been acquiring agile experts, expertise, books, conferences, entire processes, that I think are slowly turning around greenfield project standards. If you work on a project where the issue is how to get everyone up to speed on iterating, velocity, OO, TDD, CI, build and deployment automation, and simple design on a new project, well, good for you. Count yourself extremely lucky. It’s still darned hard to do, but it can be done. It is deeply gratifying work, given enough skill, knowledge, courage, discipline, and management advocacy.
But as arose as a topic at Agile 2008, and has been arising for me with clients a lot, most developers in the industry can work for years without the opportunity to start from scratch. For most of our careers, we are basically hamstrung by the legacy code issues that keep so many software development professionals living in worlds of constant emergency, constant production defect repair, very slow progress.
Worse than this, our legacy code is accumulating faster than we can cope with it. Our release schedules and iteration schedules are more pressing, while we are increasingly dwarfed by these enormous, stinky, towering piles of crap. We really, really need a way out of this situation — and not just for one team, but for the entire enterprise. And not just in the object model, and not just in the architecture.
Legacy Complexity: It’s All Over the Place
When we do start talking about legacy codebase repair, we often start talking about how to get part of the object model under test. How to start repairing the Java, or the C++, or the C#, or whatever. As far as this goes, this too is 100% goodness. We certainly need characterization tests, opportunistic refactoring in high-traffic, high-business-value neighborhoods of the code. Again, all goodness.
But I suggest that that too might be the wrong thing for us to start with, or at least the wrong thing for us to focus most of our consulting energy on. I suggest that without a better measure of overall complexity from the top to the bottom and from back to front of the enterprise, we don’t really know the best place to start.
I have seen more and more teams engaging in agile software development followed immediately by waterfall integration and deployment.
The more I work at this, the more convinced I am that the legacy complexity that is hurting us all the most is all of this contextual enterprise complexity. Our biggest problem, and biggest potential point of leverage, is the massive legacy bureaucracy that makes inter-system integration, promotion between environments, environment configuration, version control, configuration management, and production deployment such stupendously, horrific nightmares, release after release.
“Total Enterprise Software Complexity”
The main problem is not within the individual systems (as crappy as most of them are, and as tempting it is for us to start diving in and refactoring and test-protecting them). The main problem, as far as I can tell, is between all of these systems. I don’t care how many million lines of stinky legacy untested Java you have. I bet dollars to donuts most of your worst problems are actually between those piles of Java.
I read somewhere a great little discussion (I forget where) about how cyclomatic complexity, for OO code, captures or covers most of what is healthy or unhealthy about a codebase. All the other kinds of dysfunction you would likely find in stinky OO code, and might measure separately, can be covered by cyclomatic complexity. As readers of mine know, I would amend that only slightly, using Crap4J for example, to measure how well test protected the most cyclomatically complex code is. Anyway, the point is that if you are smart, you end up with a single number. Cool. I love single numbers.
So I want a new kind of number. For a given enterprise, before I start determining where to focus my consulting, the metric I want for Christmas would be a single number that blends or relates to at least the following objective and subjective categories of enterprise mess:
- How many total development teams do we have?
- How many total developers do we have?
- How many total systems do we have that are interacting with each other?
- How many distinct technology stacks do we have in play (e.g., .Net, J2EE, SOA, AS400, Tibco, Rails, etc)?
- How many distinct frameworks do we have in play (e.g., Struts, Spring, Hibernate, Toplink, Corba, EJB)?
- How many total languages are we using (including SQL, Perl, shell scripts, Groovy, Ruby, XML, etc)?
- How automated is the process of deploying from a dev machine or CI machine to a QA target? From a QA target to Production?
- How many total lines of XML are there in play in the enterprise? How many total lines of build-related properties files? XML is so nasty it really does deserve its own measure. XML is a powerful carcinogenic force in organic enterprise systems.
- What is the average ratio of the lines of code in each system’s build scripts to the total lines of code in the system itself? (Feel free to substitute a better measure of build complexity here)?
- How much automated end-to-end functional test coverage do we have? Granted (as I advocate elsewhere) you don’t want to lean forever on huge suites of automated functional tests. But as we start healing a quagmire, how many have we got?
- And yes, what is the complexity of the average object model? How bad off is the average Java or C# or C++ project? A Crap4J number is great here.
So. For the while frigging enterprise, I want a metric, on a scale from zero (ideal) to 1000 (doomed) that describes how much of this mess is interfering with everyone’s ability to make production deadlines, much less transition to a continuously improving, manageable, agile approach.
And I want to be able to tailor a consulting approach — somehow — for an enterprise with a “Total Enterprise Software Complexity” score of 200 very differently than I would for an enterprise with a score of 750.
That’s all I want for Christmas this year. Is that what you want too? Let’s talk about it.
Caveat Lector: This really is an early draft. I throw it out for feedback. If smart enough people review it, I’ll likely be able to refine it greatly. That is my hope. So smart people, please comment and email me.