The Pinned Progress Curve

I often get into discussions about the level of skill in the industry.. often with people who are deeply involved in the cause at one company or another.  They see their efforts to raise the effectiveness of their teams and increase the skill level, and they make some generalization about how, someday, we'll all be Agile (notice the capital 'A') and the industry will have been saved from itself.  Rainbows and unicorns.  The goal we all strive for.  Except.. I don't feel that way.  Or, perhaps, after many years as a consultant, being lucky enough to have visited hundreds of organizations, I have a different view of progress in the industry.  If I had to put it in a nutshell, I'd say it like this:

The variance of effectiveness in software development advances much faster than the mean

Imagine a graph. The x-axis is the level of effectiveness of a small group of people developing software.  The y-axis is the number of groups at that level of effectiveness.  I don't know what the distribution would look like.  To make a real graph, first of all, we'd have to define effectiveness, and find some way to measure it. But, as a thought experiment, maybe we can imagine it looks something like a normal curve.  We have a high mean at some point, and it tails down to the left with teams that are using ancient technology with bloated processes, poor training, etc.  On the right, we have groups who've actively tried to improve and stay ahead.  They become more effective, they learn new technologies, and actively improve their game.  The thing is, this side of the curve tails down too.  There are fewer groups that are highly effective than there are ones that are moderately effective.  To become highly effective you have to fight organizational inertia and personal inertia.  And, often it helps to be in a market where extremely high levels of effectiveness are rewarded.

Okay, let's take this curve and advance it 5 years.  What can we say about the groups on the left.  Have they all advanced?  Unfortunately, no.  There are many cul-de-sacs in software development.  They may not be part of your immediate social network but there still are people writing Cobol out there.  I'm sure that some places there are still people writing in VAX assembly language.  Some of the thorniest cul-de-sacs are off-shored.  Teams in Asia using compilers, tools and environments that haven't been updated in years.  The companies that produced them may have gone out of business, but the code remains.  It can't easily be replaced.  Sometimes, the lack of advancement is banal.  Some teams work on projects that aren't very integral to the organizations they are in.  The aren't part of the company's value chain, so there is no real incentive to make things better. Teams can coast like that for a long while.

On the other hand, the right end of the curve keeps marching forward.  Early adopters try things out, solve problems, develop new things and then discard them for yet newer things.  The things they discard, though, rarely go away.  Moderately effective groups leap to them.  Sometimes they adopt them fully and sometimes they don't. Inertia often gets in the way, and they just stay there with that technology or practice.  Over time, the total number of people using it falls off and it becomes a cul-de-sac that marches off to the left side of the curve.

The effect of all of this is that the net effectiveness of the industry keeps advancing, but the variance increases faster.  Progress is real, but it has an odd shape.  The left side is pinned and the right side keeps advancing.

The lesson that I take away from all of this is that you can't have the goal of advancing the whole of the industry. It isn't going to happen.  But, you can make a local situation better and you can be one of those people who produces disruptive changes that raise us to even more dizzying levels of variance.  I don't think people ever think of the latter as an explicit goal but, regardless, it's what happens.  And, on balance, I think it's good.

Comments

[Steve Freeman]

If we think about another graph of value per head vs. number of teams, I would guess that it would be a power law. Most teams are so trapped by their history that they can't contribute very much, while there's a tiny number of teams who have the right tools and circumstances to bring in the revenue.

[Michael Feathers]

@SteveFreeman. Agreed. I think that organizational inertia is the elephant in the living room. I really like the model of orgs that Gordon MacKenzie presented in 'Orbiting the Giant Hairball' [ http://www.amazon.com/Orbiting-Giant-Hairball-Corporate-Surviving/dp/0670879835 ] fits very well. Orgs drown in their own history.

[Steve Knight]

Very thought provoking.

At the risk of being called a luddite, I sometimes think that some of the enormous effort invested in new technologies actually generates a drag at the front end of this thought-curve too.

Making the super-effective less-effective whilst they come up with new and more interesting ways to make that wheel turn slower, and more abstractly.

But not all advancement is bad and if we could pick the good ones in advance we'd all be very rich I suspect!

[iPhone contacts backup]

I like it though.