Every once in a while, I get upset about something that happens in software development. Sometimes I simmer, sometimes I smolder, but eventually, I do pipe up about it. The thing that's been bothering me for a while is the fact that agile really threw the baby out with the bathwater about 10 years ago. At that point in time, we we had many people with deep design knowledge in the industry. Hundreds of books had been written about the mechanics of object-oriented design, and structured design. Although this knowledge was not pervasive, you had the sense that if you walked into a room of people who cared about software development, they'd be familiar with many of the design principles, patterns, and concepts that were being thrown around then.
I'm not sure that has changed. People who care, still dig deep and learn. The thing that did happen, though is a de-emphasis of software design over the last ten years. I think that there are a couple of reasons. One was maturity. There was a a lot of material out there 10 years ago, and it was excellent guidance. Maybe we'd just discovered the foundation of design and we've all moved on to other things. Another was a well-deserved re-emphasis on the people part of our work. Nowadays, the hot topic is: how to best get people to work with each other to deliver software. The last reason to mention is TDD/BDD. With Test-Driven Development and Behavior-Driven-Development, we moved much of our design discussion into the realm of micro-process: you take these little steps and good design emerges. That tends to be true, but no one can deny that it requires guidance and you can only really guide the evolution of your code when you have good design chops.
So, where are we now? I think we're at the cusp of something very interesting in the software design space, and it involves looking at things a bit differently. Let me start with a question. The last time you saw a blog or an article about design, did it just give you a single snippet of code outlining an idea? Chances are, the answer is yes. An article about refactoring might give you a before and after shot. But, do you ever see anything like this in articles about software design?
This graph shows the history of a file in a project in terms of complexity. You can see that the complexity of the file has grown over time, but there are dips, and those dips likely indicate complexity-reducing refactorings that have happened over the life of the project.
Okay, how about this diagram? It shows the number of commits for each file in a project in order of increasing number of commits. As you can see there many files that have only a few commits and a few that have an incredible number.
Remember the Open/Closed Principle? The notion that in a good design, we have many abstractions that don't change very often. Well, guess what? We can figure out whether it really happens on a project, and how often. All it takes is a little SCM mining.
Some of you reading this might be thinking "Oh, no! Metrics." I understand. Metrics do have a well-deserved bad rap. There is nothing worse than managing by a code metric. It throws away too much information and there are always unintended consequences. This isn't really about that, though. It's about taking the data that we have at hand in our development work and really using it. If we are making a decision about whether to refractor a piece of code, we should be able to see its churn and complexity trends, and tie them back to events that happened over time, and the actual features which triggered the work. Right now, it seems that we often look at our decisions through the pinhole of the present, ignoring what we can learn from our code's past.
At the time I'm writing this, there are many people who have been playing with these ideas for a while. Steve Yegge and many others at Google have been doing quite a bit of code mining work. Keith Braithwaite has been exploring how TDD affects the complexity of code. Joshua Kerievsky of Industrial Logic has, more than anyone else, pushed the direction of these ideas. His dream is to have IDEs with historical information, information mined from production logs and quite a few other things at our fingertips. Jason Gorman, Tim Ottinger and Greg Wilson share an interest in gaining a deeper empirical understanding of software development through this kind of work, and Chad Fowler, Corey Haines (blog) and I are brainstorming ideas and actively developing tools to facilitate work in this area. Currently, we're working on a Ruby gem called 'Turbulence' which plots churn and complexity for Ruby projects. Brian Foote of 'Big Ball of Mud' fame looks on and makes sage observations about the shape of software.