So why do businesses fail? A google search will yield lots reasons people offer up with good intent. Look them over, it’s an instructive exercise. As you do so, ask yourself how much the lack of systematic focus on stakeholder requirements and the corresponding emphasis on accumulation of domain expertise is actually a secret, semi-conscious theme running through many of the reasons given. Also notice how seldom it is mentioned explicitly and clearly.

Most start-ups think they do all right when it comes to accumulating domain knowledge, or, more narrowly, when it comes to framing requirements. They can’t be bothered to add detail that isn’t needed right now to get the next milestone in on time. Companies that outsource learn the hard way that they need to work harder at it than they expect, or they do not survive.

Yes, most companies, most engineering teams think they’re doing fine, until, deep in the midst of their heroic labors, they are shocked awake by the realization that small oversights, wasted efforts, and little pieces of time lost to rework mean that the critical business opportunity has been missed. Numbed, they try to pick up the pieces (or watch as others do) in the aftermath.

It’s hard to escape conclusion that this is the result of a systematic shortcoming in our approach, that we’re suffering from some blindness born more of the habit of looking elsewhere than of any fundamental defect in our sight. What could be the source?

In the midst of our Promethean efforts to bring the next great technological miracle into the world, we forget that in the Greek myth the hero-titan who suffered to win fire for mankind had a twin brother from whom he was inseparable. Epimetheus’ strength was“hindsight”, his curse forgetfulness.

Software development is hard precisely because software’s utility approaches so close to the threshold of conscious, and our brains are designed (have evolved) to accommodate the details of tool use automatically, with practice, in our goal-oriented absorption in the task at hand.

What we need is the balance of a little (just enough) Epimethean perspective building to anchor our goal-driven hurry.

I’m suggesting that it has particularly to do with the process of requirements gathering, because by its nature it is something we have to do with deliberation, even though we are inclined (both organizationally and individually) to want to push the task ‘into the background’ of our awareness as soon as possible in the name of ‘productivity’.

We find it painful slowing down our process of understanding, stifling the impulse to think in terms of solutions (and often jumping prematurely at the first one) rather than ‘living in the question’ long enough to know we’re heading in the right direction.

<aside>
I don’t mean to go off into a long digression into mythology, philosophy or, God forbid, post-modern deconstructionism here, but I will pause to point out that I’m not by any means the first to draw this connection. The philosopher Bernard Stiegler was at great pains to point out that we systematically forget Epimetheus and his role in our world, with tragic results. At one point the authors of the Wikipedia article summarizing his work Technics and Time paraphrase Steigler as observing that “a constitutive blindness, forgetting and idiocy accompany the fact of being technical”. This isn’t an indictment of nerds. The intended use of the term ‘technical’ is broad enough to encompass most human tool use, and by extension, most human culture of any sort. It is intended as a statement about the make-up of our conciousness.
</aside>

I’m saying that we’re capable of being better at this, but that it will take systematic, organizational, collective effort and iteration after iteration after iteration. Unless that effort is part of our ongoing way of doing engineering, of doing business, we don’t have a ghost of a chance to master the situation.

We need to pause in the here and now, between our Promethean questing and our Epimethean regrets, to systematically grow and cultivate our awareness of the present, of what we know and of how to communicate that.

We pride ourselves at coming up with the best designs possible within short time frames, at working in realms where the requirements are partly or poorly understood, and somehow coming through anyway. We code defensively, follow good OO principles, use design patterns, and drive our development with unit test.

And we do all this not quite knowing how the market is going to evolve while we work, or what the competition is up to.

“Agile” is quite a popular term these days, even le buzzword du jour. When The Agile Manifesto came out, I read it with some excitement and hope, but then increasingly with a tinge of disappointment. It’s authors got carried away, over-emphasizing the novelty of their techniques, and in so doing failed to learn some of the lessons that hard experience in iterative systems engineering could have taught. It should be a clue when a methodology labels itself ‘extreme’ that it isn’t emphacizing balance.

And yet, isn’t dynamic balance at the core of what living, even life itself, is all about? We could think of running, and even walking, as graceful, rhythmic not-quite-falling. Making progress depends on a balance between falling forward and standing upright.

The metaphor applies to engineering as well. We need to balance our ‘heads-down-and-go’ focus on code with a ‘heads-up’ vision of what’s coming and where we need to end up.

In time, the foundations of agile processes have become clearer, and some of the debt to earlier practitioners has been acknowledged from time to time.

My take on what it takes to give a team the edge it needs to succeed in this environment has been strongly influenced by Tom Gilb and his “evo” or “competitive engineering” practices which have their roots in systems engineering. I select from his work what I think is best for the team and project I’m on at the moment, and bring in many other factors as well such as test-driven development, domain model driven design, patterns, and so on.

But the bottom line is this: let’s utilize the tools and techniques and, yes, processes, lightly, with a common-sense eye for what works, and with a practiced sense of perspective while we do it. Let’s balance getting done what needs to be done right now with doing it in a way that maintains and even improves our ability to be competitive on the next project.

In other pages and posts on this blog I’ll discuss particular techniques. The larger framework, however remains Competitive Engineering, in the larger sense of the term.