Making Too Much of TDD

I once read that people who end up as developers usually have one of two different temperaments: scientist or engineer.  You either get a deep thrill out of learning things or a deep thrill out of creating things.  The best developers are a mix.  They are continually on the prowl for new ways of doing things and they reflect on their practice, but they also know that they have to ship and they get a great deal of satisfaction out of that.  If you're at either extreme, you end up as an "architecture astronaut" or a hack who ships cruddy code day in and day out.

I know that I lean more toward the scientist of end of things.  I love working with teams and shipping code, but at the end of the day, I measure myself in what I've learned about software development.  What I've seen that works, and what I've been able to discover.  One of the secrets of the software industry is that many people who end up as high-end consultants really are more at the scientist end of the spectrum.  They amass a lot of knowledge by experimenting and thinking critically about what they do.  In an age where pragmatism and the credential of shipping code is nearly deified, it's a sore spot for some of us.  We know that we have to reflect, that we're good at it, but also that we are sometimes a little bit out of the cultural value stream.  I'll never forget a time when I visited a team and I sat with a guy who was about to change a 300 line method.  I suggested that we do some 'extract method' so that we could understand the context better and he gave me a derisive look and called me an academic.  I spend more time than most people I know wading through and teasing apart intractable code, so I'm not concerned about my credentials, but I know that I have to be aware of my own inclinations and those of others so that they don't blindside me.

One of the downsides of having the scientist bent is that it is very easy to be seduced by ideas, especially if they are aligned with some sense that you already have of the way things should be.  People with the scientist bent look for hidden order and when they find it, they get  a bit giddy, or at least I do.  You get this feeling that you are discovering something fundamental, and often you are.  We had a lot of this back in the early days of Agile, especially around the topic of 'emergent design.'  I remember that, over and over again, I would develop something with TDD and arrive at design patterns without planning to use them, and it fascinated me.  Joshua Kerievsky and I used to debate this back and forth and he eventually wrote the (great) book 'Refactoring To Patterns'.  At the time, I thought yes, it's true you can refractor specifically toward patterns, but it is very interesting that you often arrive at them without planning to.

The emergent design meme, for better or worse, became intimately woven into Test Driven Development.  We found that, in the small, you could start developing a system without having a plan for its design, and, through the application of a set of rules and some reflection, arrive at very good designs.  When I think back about this, I call this the "Look ma, no hands!" era of TDD.  Critics would rightly point out that people who were doing this well were drawing on a lot of tacit knowledge of good design.  For the most part, we agreed.  We just felt that this knowledge of good design could be taught, and people could make continuous decisions across the development cycle to grow good design organically.  You do have to bring your design knowledge to the table.  Red/Green/Refactor is a generative process, but it is extremely dependent upon the quality of our input.

The thing that fascinates me the most these days is where TDD and refactoring fall apart.  I'm most concerned with the "you can't easily get there from here" problem. In a nutshell, it seems that incremental development of a design works fine until you confront stories which force you to change internal structure drastically to accommodate them.  A simple example would be something like a text editor.  You can march along very far maintaining a document as a single string before you encounter situations that make that representation impractical.  Of course, if you're using TDD, you aren't lost at that point.  You have tests that pin down the current behavior and you can use them to  build verify a new implementation as you develop it, or, if you are very practiced at it, find a way to slowly finesse in the alternative representation through refactoring.  The thing is, though, that these representational changes are not trivial, and the limit the degree to which we can just merrily look only at the step in front of us and arrive at an ideal solution in a decent amount of time.  

For the most part, we've ignored this when discussing TDD and it's pretty much because we can.  It turns out that that these sorts of major representational shifts are relatively rare or something that people often see coming and then make adjustment for.  On top of that, we can offer it as a challenge: "you may think you are stuck, but are you really?  Apply more ingenuity."  To me, this means that we are avoiding something.  We are avoiding serious study about situations where switching gears is difficult.  Sometimes these situations are triggered by our approach and sometimes they are triggered by our choice of language features.  In either case, if we know more about them we can be more effective.

So, what are they?  What are the things which lead to difficult transitions? Let's imagine the text editor scenario in more detail.  One of the first stories we might have is adding text the editor's buffer.  We accept keystrokes and formulate some way to examine the text.  Later, we might want to see that we can move the text cursor up, down, left, and right.  Representationally, all of this can be accommodated with a string and an index which maintains the cursor position.  We may discover later, however, that maintaining a list of lines might be better.  This can either come up through some story like "line cut, copy and paste" which would have an easier implementation with a line-oriented data structure.  One approach that we can use is to just muscle through: work with our simple data structure and introduce a bit of complexity which would've disappeared with a line-oriented data structure.  Another alternative would be to create a mini line-oriented API which works against our simple string representation so that the complexity is well-encapsulated.  We could do that, or we could make the assessment that a line oriented data structure is a better representation for most of the features we can anticipate, bite the bullet, and move to it now.  There is no "one true way" of TDD which gives us guidance for this sort of choice, and I think there is a very good reason for that.  These decisions depend on a substrate of information that is both beyond principle and also sensitive to language. In short, it's deeply contextual.

We need a name for a particular concept.  I'm not sure what to call it.  But it is something like "language granularity."  If you take a program with structure A and want to transform it to structure B, how large are the steps you have to take to get there?  In a low-level language you can make many small changes to get from A to B, but as the level of the language rises, it's often easier to rewrite some bit of code than to refractor it into a different shape.  As an example, if you are using zip or fold in a lazy functional language, there are fewer behaviorally equivalent waypoints along your path to a different structure than if you were using a loop in an imperative language.  You can split a loop, introduce or remove temporary variables.  To me, this is granularity.  When you use lower level constructs, you have more waypoints.  I don't think this is good or bad necessarily, but it does make the work different.  When the grain of a language is coarse, we may have fewer steps when refactoring, but we may also have a bit more work to do to see the next step.  When the grain is fine, we can often have more confidence that we are going to get to the structure we'd like to arrive at and we're tempted to take it a little slower, with many more intermediary steps.

Oddly enough, designs have a similar quality. They are the next level up from languages.  If a program uses data structure A and we want to move it to B, the cost and ease depends not only the language granularity but also the amount of change that would ripple through the design as a result of the change.  This is coupling, pure and simple, and in any long-lived system a design will have a grain, much like a language has granularity.  There are some structural changes which end up being so costly that, even with tests, people hesitate before undertaking them.  It would be easy to claim that, well, this is just a symptom of bad design, but design does involve tradeoffs and the costs are variable.

I think we need to investigate design grain and language granularity a bit more.  We aren't going to find an ideal strategy for taking the next step when we practice TDD.  It's something which is beyond the process.  The answer is, like many other things, in our understanding of the state of our code and ways it can and can't easily change.

Comments

[Telanis_]

I think "refactor" has been autocorrected to "refractor".

[Anonymouse]

Overall, I agree with your main point (TDD pretends we don't need any design.) But I disagree with your example.

> We may discover later, however, that maintaining a list of lines might be better.

This should be an implementation detail. Are you saying the API will be different because we're storing text by lines instead of a single string? If so, your API needed fixing anyway. The API will always have different levels of granularity (select word, select line, select paragraph). The whole point of a good API is NOT to expose the representation that's internal to the class.

On an embedded system with little RAM, we may opt for "insert this byte by manual copy/shifting the remaining text" and "count lines via brute-force search for \n". For small text files, the caller will never notice.

The internal representations change for performance/maintenance needs only. If the new internal representation "leaks out" into the API, then that's a leaky abstraction. This should only be allowed when performance targets can't be met with a "pure" API.

So if a programmer makes a bad API, he'll be forced to fix it later. So eventually, he'll learn not to do that. (And start thinking ahead a little bit.) But I don't really think that's entirely TDD-specific. After all, people do the same thing outside of TDD all the time.

[Grok2]

Excellent writeup -- made me think! Also you should separate out the earlier paragraphs about the scientist and engineer types from the later paragraphs about tdd, language granularity and grain (the latter portion is what I found interesting, btw).

[Rachel Blum]

Worth considering:

The things that force the large steps are usually two kinds of stories, IME.

1) Stories that push the boundaries of your system. (Performance requirements get you off the string representation. Space requirements for docs larger than available RAM get you to a paged representation. Per-character formatting might push you into yet another direction)

2) Stories that require deeper insight and that can't be approached incrementally. (Famously, the Sudoku solver. But really, any algorithmic problem)

Both of those seem to require some amount of up-front design, just to gain understanding of the problem. One possible conclusion is that there is a certain class of stories, lurking on the outer perimeter of your system and at the very core, that should be answered on paper/napkin first, before you write large amounts of code.

[Iain_nl]

Same topic as Uncle Bob. Did you plan this? And what's your opinion about his article?

Link: http://cleancoder.posterous.com/the-transformation-priority-premise

[Joel Parker Henderson]

@Rachel, I agree and would add a third kind of thing that forces large steps, which is a sea change feature.

An simple example of this for the string editor would be adding internationalization. TDD could be especially valuable for detecting tricky bugs, for example due to string representation, string searching algorithms, and interactions between the app's strings and any of the app's libraries.

[Apo]

I like the "design grain" and "granularity" concepts. To me they summarize the feeling I've had that relying on plain unit testing is not enough, instead you should diversify your tests to cover the different granularity levels of the software.

That way you'll have simple unit tests for the insides, functional tests to cover the functionality etc. When you are going to change the internal implementation, ideally you don't have to touch functional tests, instead you just need to write unit tests for the new functionality.

Planning a good test coverage becomes an act of balancing between the different granularity levels and their pros and cons.

[Michael Feathers]

@Iain_nl It wasn't planned, but Bob's article did inspire mine. I think there's truth in his hypothesis, but I also think that there are real gulfs that we encounter when we work on problems which make cul-de-sacs unavoidable.

[Michael Feathers]

@Anonymouse I think that's a seductive mode of thought... that we only have to change representation for performance reasons, but sometimes it's just a matter of realizing that new features can be added in much more easily if we revisit an an earlier decision.

[Pbadenski]

The way I see it, there's more general problem to that - a lack of linguistic apparatus to constructively discuss software design. In the subject that's being discussed there's obviously some property, a part of something bigger and I can't help feeling it is something fundamental (same as you, if I do understand correctly). It's a pity that we're not able to talk about these basic things, that we do not have that model, that theory of software design with all the basic forces identified. Is it really that we haven't moved any further than coupling and cohesion? Or am I just an ignorant by not knowing more.

Obviously we're trading one thing for another. The question is what for what and why? I'm sure everyone could think of many concepts on the spot, but we're doing that development, that software evolution every day for some time now and we don't have that figured out yet, what the hell. It might be that it sounds like just "academic stuff", but if at the end of the day you're trying to convince each other speaking of beauty of particular design choice then guess what - joke's on you. Moreover I believe it's just a start because as soon as more complex ideas come into play, such as TDD in the article, we are lost like a child in the woods. That stuff's what we have not figured out at all.

Maybe, just maybe it's only because recently I'm fixated on the lack of methodical grasp on software design by programmers.. or it is my inner scientist who's responsible for myself seeing this problem that way :)

Can't help feeling there are that basic forces which shape our design - viewed both as a structure and as a process.. we're just not able to see them yet. I would really love to see people studying something like "philosophy of software design", people exploring, giving full rein to their imagination. We had alchemy before chemistry, astrology before astronomy, philosophy before.. pretty much every science - damn, we need that creative process!

Hope I didn't digress to much.. :)

[Paul Saieg]

@Michael
I enjoyed your post, especially the scientist/engineer distinction. Those really are the poles between which developers oscillate.

As far as the "you can't easily get there from here" problem goes, where a fundamental representation needs to change (like the string/array representations in the text editor example), I think you are perhaps a quarter-inch off the mark.

The real problem, it seems to me, is not that incremental design work via TDD can’t address this issue — its just that it’s hard to know to when to buy the complexity you need to keep the refactor simple and the cost of change low.

Since this kind change is a shear along what Uncle Bob calls "The Data/Object Anti-Symmetry", what you really seem to be saying is that, if you have been programming procedurally (with data structures or, worse, primitive obsession) changing the shape of that data structure will cause all of the functions which interact with that data to change. This form shotgun surgery is what makes the refactor hard; this is because BOTH the signatures of your methods must change (which stops potentially thousands of your tests from even compiling) AND the logic of your methods must change (which often invalidates your tests outright). The root of the problem is that deep knowledge of the structure of your data is duplicated throughout not only your production code base, but also the tests you need in order to insure the production code still works.

The trick, is think, is to know when to buy the complexity of a true object that exposes a tested API, because — in all likely-hood — by the time the requirements demand the complexity, the cost of change is already high and we need to do some kind of seam-introducing-acrobatics() && advanced-refactoring-ninjitsu() to move forward safely.

We are left thus in a bit of a pickle on this one because most programmers (even we “scientists”), when it came time to implement the design that would insulate the code base against these difficult, disparate changes, would call YAGNI and not do it until there was a requirement that made us.

@Pbadenski: I think something we do need, as craftsmen, is a better way to talk about when to introduce complexity. In the present example, the usual TDD minimalist axioms like YAGNI and friends end up costing us in design debt. At present, we have no language or axiom for this intuition, which guides every experienced TDDer to break these rules on occasion, but is hard to describe or justify to a larger organizational context.

[Rachel Blum]

@Joel I'd almost argue that "sea change features" as e.g. introducing i18n late in the game, mean that you built the wrong product. Unfortunately, while that allows me to complain about management once more missing the boat, it doesn't change the fact that I'll still have to do it ;)

Had we done it at the start, it would be a boundary story....

[Michael Feathers]

@Rachel I think that's one of the things we're struggling with right now.. we're not looking at the major refactorings as part of business as usual because there's the sense that "well, if we had done it right." To me, it seems that even good codebases end up with hard to change assumptions, whether from unanticipated features or late insight into a better way of structuring things. One of my favorite stories on the latter is in Eric Evan's book.. when a team discovered they needed share pies.

[Philip Schwarz]

@Michael

You said: "I once read that people who end up as developers usually have one of two different temperaments: scientist or engineer. You either get a deep thrill out of learning things or a deep thrill out of creating things."

Here is how Fred Brooks put it: "A scientist builds in order to learn; an engineer learns in order to build".

[Luca Minudel]

Found those thoughts interesting.
Till now I thought of what is described here as a black-box called 'creative process', curious to see what can come out digging into it.

Found the title 'Making Too Much of TDD' (maybe good to drive some more traffic) bit misleading

[Luca Minudel]

will rephrase "Red/Green/Refactor is a generative process, but it is extremely dependent upon the quality of our input." like this:

"Red/Green/Refactor is a generative process that depend upon the co-evolution of ones practicing it and their skills, knowledge, experience, intelligence"

[SEO services]

That was a great writeup, very unique of its kind, left me asking for more!!..

[George Harrison]

You guys are all such integers! TDD is not a lifestyle choice, it is genetic.