« Are FP and OO Incompatible Syntactic Styles? | Main | Are Nested Classes Really a Good Idea? »

June 12, 2008

Comments

MORITA Hajime

Very impressive! So I've translated the article into Japanese, that is available at
http://www.hyuki.com/yukiwiki/wiki.cgi?FlawedTheoryBehindUnitTesting

Please let me know if you have any inconvenience.
Thank you for your good writing.

Colin Jack

Really good post and I definitely agree with the sentiment.

Having said that whilst different testing styles might each result in quality improvements they can also result in tests that affect future development in very different ways.

For example if I do the little integration tests (which I believe you might have gone off since 2000) then I get tests that are more immune to low level refactoring but they do tend to be more complex and slightly slower. Whether they provide good documentation depends on what the view of the system the reader wants, which is hard to judge. Its these sorts of trade offs that I struggle with and that leave me constantly questioning my approaches to testing.

MoffDub

Nice post. Your post has been linked to in my post about the same topic: http://moffdub.wordpress.com/2008/06/16/the-getter-setter-debate/

Philip Schwarz

You say that quality is a function of precise thought and reflection, so any technique that reinforces this discipline increases quality.

In his afterword to Kent Beck's 'Test Driven Development by Example', Martin Fowler has an idea, which in my mind may suggest that TDD is better at reinforcing the discipline than say Clean Room, or writing tests after coding.

Here is the gist of Fowler's idea (I have turned excerpts of his afterword into bullet points):

* Programming is hard.

* It sometimes feels like trying to keep several balls in the air at once: any lapse of concentration and everything comes tumbling down.

* TDD helps reduce this feeling, and results in rapid unhurriedness (really fast progress despite feeling unhurried).
This is because working in a TDD development style gives you the sense of keeping just one ball in the air at once, so you can concentrate on that ball properly and do a really good job with it.

* When you are trying to add some new functionality, you are not worried about what really makes a good design for this piece of function, you are just trying to get a test to pass as easily as possible.

* When you switch to refactoring mode, you are not worried about adding some new function, you are just worried about getting the right design.

* With both of these activities, you are just focused on one thing at a time, and as a result you can concentrate better on that one thing.

* Adding features test-first and refactoring, are two monological flavours of programming.

* A large part of making activities systematic is identifying core tasks and allowing us to concentrate on only one at a time.

* An assembly line is a mind-numbing example of this - mind numbing because you always do the one thing.

* Perhaps what test-driven development suggests is a way of breaking apart the act of programming into elemental modes, but avoiding the monotony by switching rapidly between those modes.

* The combination of monological modes and switching gives you the benefits of focus and lowers the stress on the brain without the monotony of the assembly line.

Mike Bria

Wordled version ;-)
http://wordle.net/gallery/01304/Feathers:_The_Flawed_Theory_Behind_Unit_Testing

Ricard Mayerhofer

Hi Michael,
I learned to not think in implementation details when creating testing during tdd process (like described in TDD by Example). This is very useful, because rather than thinking in implementation details I think how my class is suppossed to act. Eg. caculateTax in this scenario should return 15. No matter how this is implemented.
This gives me freedom to refactor in order to make the code cleaner, with no fear of breaking my tests.
This also help me to achieve test as documentation, so users of my class can get examples of how to use it. As a user I want to know the visible changes of the class, not how it works internaly. A lot o expectations that has nothing to do with a real use o my class makes it harder.

Interaction based tests are harder to read and to find in a glance what it is suposed to test.

They are also more fragile to refactoring.

IMO...

Dean Wampler

In the 90's I wrote the comment for a method before implementing it. Combined with Design by Contract, It was my way of thinking through the design issues before coding.

I still think in DbC terms, but now I capture that same design information in my tests.

Patrick Welsh

Someone recently said that an effective unit/micro/isolation test suite is the single most important artifact on a software project. More important than the production code, even, since without the test suite, the production code will always slip into irreparable decay. If the production code is the house, the unit test suite is the foundation. Build it well.

I think that integration test suites are important, since they tend to catch real interactions between real components, using closer-to-real data.

But while integration test suites may often be cheaper to write (they are certainly cheaper to *learn* to write), they are not cheaper to maintain than good unit test suites. And they are less likely to catch all the defects, and they are less likely to catch the defects early enough. They will al.so localize failure less well.

For least cost per defect, in the long run, and for least total cost of ownership, per test suite, I don't think you can beat really, really fast isolation tests, complete with requisite mocking, and little tricks like in-memory databases.

That said, I always end up with a contextual, locally-adapted mix of true unit tests (using your definition, Mike), and integration tests. And they are managed differently for CI purposes (fast tests are always run more frequently than slow ones -- so we separate the fast ones out, and the build state is always contingent on their passing).

So what about early on for a newly agile team?

Early on in a team's adoption of unit testing best practices, do I care more about true isolation/mocking and fast tests, or about good semantic coverage, and getting people test infected? An early win?

I care more about the latter. Early on, I just want them to learn that automated testing is possible, worthwhile, and full of delightful unexpected benefits and joys.

Scott Saad

Great article Michael, thanks for sharing! I ran across another post that talks about the possibility that TDD is becoming a Cargo Cult.

http://www.mtelligent.com/journal/2008/6/26/the-cargo-cult-of-test-driven-development.html

Between your article and this one, I ended up writing something as it made me reflect on our practices as developers and how we can fall into the "trap" of forgetting our original purpose for picking up TDD as a tool.

http://blog.saadware.com/2008/07/02/purposeful-software-development/

Thanks again!
Scott

Michael DiBernardo

BlogReader: One example that I remember off the top of my head re: CleanRoom was in the NASA software process improvement report, as referenced heavily by Steve McConnell in Code Complete.

http://www.sei.cmu.edu/publications/documents/94.reports/94.tr.022.html

Brian Henderson

Summerizing...

What TDD is:
practices which help us achieve continuous discipline and a continuous state of reflection.

Why TDD works:
Quality is a function of thought and reflection - precise thought and reflection. That’s the magic.

David Hume

[speaking from the grave]

I am proud!

Jason Gorman

I like that you mentioned the Old Ways :-)

Kids today don't believe that anyone was delivering code of any quality or value before Extreme Programming Explained was written.

I think guided inspections (test/example-driven, of course) are due for a comeback.

George Brooke

This is a nice article and makes good points... essentially that the whole of TDD is more than the parts. Another point I should like to make is that designing code which will run smoothly in a test environment as well as in a production environment is itself a design constraint and should lead to more portable, reusable code with better encapsulation properties

Ajay george

Nice read. Totally agree with your last point.
" Quality is a function of thought and reflection - precise thought and reflection. That’s the magic. "

usuck

yer an idiot

Jim Gay

Great post. This makes me think of literate programming but without the overhead of all the muck left around your code.

Monkeyonahill

This ties in perfectly with Greg Youngs talk on testing http://skillsmatter.com/podcast/design-architecture/talk-from-greg-young

and his article http://codebetter.com/gregyoung/2008/02/13/mocks-are-a-code-smell/

Andrew Binstock

@JimBullock There is still a huge number of folks who think that TDD is primarily about testing, rather than about design. Alas, even though many TDD exponents recognize the primacy of the design aspect(as discussed here by me http://drdobbs.com/229218691 ), TDD evangelists often present it primarily as a testing mechanism. I think that's what Mike F. is getting at in this piece, although he never comes out and says that is the "flawed theory."

John Rusk

Michael, As per Mark's comment above, examples of OO code without getters would be of interest. Do you know of any that are publicly available?

(Even though, as your post suggested, precise thought probably trumps any particular technique -- even removing getters. Would still be interesting to see a good example.)

Brian Balke

I never understood how unit tests don't get you to integration - we have integrating objects that need unit tests, don't we?

I find the value of thorough unit testing is actually in refactoring and extension. Both typically involve redefinition of requirements, and the unit test repository is an efficient way of determining when I've broken a contract somewhere.

Gregory Nicholas

we need more end to end testing, with the ability for PIXEL DIFF testing!!! we need to test that actual rendering of pages is as expected

Dean Wampler

In the 90's had a similar experience with Design by Contract, applied to C and C++ code. We didn't have any fancy tools to support it, just asserts or similar constructs. I also wrote careful documentation for each function I was about to write, again a process of reflection before doing. I eventually replaced both practices with tests using TDD.

For the functional code I write today, I find myself going back to up-front thinking and ex post facto test writing. By using Big Words like ex post facto, I'm showing how smart I am...

The comments to this entry are closed.