I once read that people who end up as developers usually have one of two different temperaments: scientist or engineer. You either get a deep thrill out of learning things or a deep thrill out of creating things. The best developers are a mix. They are continually on the prowl for new ways of doing things and they reflect on their practice, but they also know that they have to ship and they get a great deal of satisfaction out of that. If you're at either extreme, you end up as an "architecture astronaut" or a hack who ships cruddy code day in and day out.
I know that I lean more toward the scientist of end of things. I love working with teams and shipping code, but at the end of the day, I measure myself in what I've learned about software development. What I've seen that works, and what I've been able to discover. One of the secrets of the software industry is that many people who end up as high-end consultants really are more at the scientist end of the spectrum. They amass a lot of knowledge by experimenting and thinking critically about what they do. In an age where pragmatism and the credential of shipping code is nearly deified, it's a sore spot for some of us. We know that we have to reflect, that we're good at it, but also that we are sometimes a little bit out of the cultural value stream. I'll never forget a time when I visited a team and I sat with a guy who was about to change a 300 line method. I suggested that we do some 'extract method' so that we could understand the context better and he gave me a derisive look and called me an academic. I spend more time than most people I know wading through and teasing apart intractable code, so I'm not concerned about my credentials, but I know that I have to be aware of my own inclinations and those of others so that they don't blindside me.
One of the downsides of having the scientist bent is that it is very easy to be seduced by ideas, especially if they are aligned with some sense that you already have of the way things should be. People with the scientist bent look for hidden order and when they find it, they get a bit giddy, or at least I do. You get this feeling that you are discovering something fundamental, and often you are. We had a lot of this back in the early days of Agile, especially around the topic of 'emergent design.' I remember that, over and over again, I would develop something with TDD and arrive at design patterns without planning to use them, and it fascinated me. Joshua Kerievsky and I used to debate this back and forth and he eventually wrote the (great) book 'Refactoring To Patterns'. At the time, I thought yes, it's true you can refractor specifically toward patterns, but it is very interesting that you often arrive at them without planning to.
The emergent design meme, for better or worse, became intimately woven into Test Driven Development. We found that, in the small, you could start developing a system without having a plan for its design, and, through the application of a set of rules and some reflection, arrive at very good designs. When I think back about this, I call this the "Look ma, no hands!" era of TDD. Critics would rightly point out that people who were doing this well were drawing on a lot of tacit knowledge of good design. For the most part, we agreed. We just felt that this knowledge of good design could be taught, and people could make continuous decisions across the development cycle to grow good design organically. You do have to bring your design knowledge to the table. Red/Green/Refactor is a generative process, but it is extremely dependent upon the quality of our input.
The thing that fascinates me the most these days is where TDD and refactoring fall apart. I'm most concerned with the "you can't easily get there from here" problem. In a nutshell, it seems that incremental development of a design works fine until you confront stories which force you to change internal structure drastically to accommodate them. A simple example would be something like a text editor. You can march along very far maintaining a document as a single string before you encounter situations that make that representation impractical. Of course, if you're using TDD, you aren't lost at that point. You have tests that pin down the current behavior and you can use them to build verify a new implementation as you develop it, or, if you are very practiced at it, find a way to slowly finesse in the alternative representation through refactoring. The thing is, though, that these representational changes are not trivial, and the limit the degree to which we can just merrily look only at the step in front of us and arrive at an ideal solution in a decent amount of time.
For the most part, we've ignored this when discussing TDD and it's pretty much because we can. It turns out that that these sorts of major representational shifts are relatively rare or something that people often see coming and then make adjustment for. On top of that, we can offer it as a challenge: "you may think you are stuck, but are you really? Apply more ingenuity." To me, this means that we are avoiding something. We are avoiding serious study about situations where switching gears is difficult. Sometimes these situations are triggered by our approach and sometimes they are triggered by our choice of language features. In either case, if we know more about them we can be more effective.
So, what are they? What are the things which lead to difficult transitions? Let's imagine the text editor scenario in more detail. One of the first stories we might have is adding text the editor's buffer. We accept keystrokes and formulate some way to examine the text. Later, we might want to see that we can move the text cursor up, down, left, and right. Representationally, all of this can be accommodated with a string and an index which maintains the cursor position. We may discover later, however, that maintaining a list of lines might be better. This can either come up through some story like "line cut, copy and paste" which would have an easier implementation with a line-oriented data structure. One approach that we can use is to just muscle through: work with our simple data structure and introduce a bit of complexity which would've disappeared with a line-oriented data structure. Another alternative would be to create a mini line-oriented API which works against our simple string representation so that the complexity is well-encapsulated. We could do that, or we could make the assessment that a line oriented data structure is a better representation for most of the features we can anticipate, bite the bullet, and move to it now. There is no "one true way" of TDD which gives us guidance for this sort of choice, and I think there is a very good reason for that. These decisions depend on a substrate of information that is both beyond principle and also sensitive to language. In short, it's deeply contextual.
We need a name for a particular concept. I'm not sure what to call it. But it is something like "language granularity." If you take a program with structure A and want to transform it to structure B, how large are the steps you have to take to get there? In a low-level language you can make many small changes to get from A to B, but as the level of the language rises, it's often easier to rewrite some bit of code than to refractor it into a different shape. As an example, if you are using zip or fold in a lazy functional language, there are fewer behaviorally equivalent waypoints along your path to a different structure than if you were using a loop in an imperative language. You can split a loop, introduce or remove temporary variables. To me, this is granularity. When you use lower level constructs, you have more waypoints. I don't think this is good or bad necessarily, but it does make the work different. When the grain of a language is coarse, we may have fewer steps when refactoring, but we may also have a bit more work to do to see the next step. When the grain is fine, we can often have more confidence that we are going to get to the structure we'd like to arrive at and we're tempted to take it a little slower, with many more intermediary steps.
Oddly enough, designs have a similar quality. They are the next level up from languages. If a program uses data structure A and we want to move it to B, the cost and ease depends not only the language granularity but also the amount of change that would ripple through the design as a result of the change. This is coupling, pure and simple, and in any long-lived system a design will have a grain, much like a language has granularity. There are some structural changes which end up being so costly that, even with tests, people hesitate before undertaking them. It would be easy to claim that, well, this is just a symptom of bad design, but design does involve tradeoffs and the costs are variable.
I think we need to investigate design grain and language granularity a bit more. We aren't going to find an ideal strategy for taking the next step when we practice TDD. It's something which is beyond the process. The answer is, like many other things, in our understanding of the state of our code and ways it can and can't easily change.