Michael Feathers' Blog

Canalizing Design

When I was a teen, I read a lot of science fiction.  Robert Heinlein was one of my favorite authors.  He wrote intelligent adventures which were quite good on their own terms, but I always had the sense that they were window-dressing for what he really wanted to do: social commentary through the veil of his own brand of paleolibertarianism.  He’d set homesteaders on a planet and engineer situations which called on them to display frontier justice and deep individualism.  The characters he poked fun at where the ones who didn’t get it; who didn’t subscribe to his view of what makes the human animal tick in society.

One day, when I was reading one of his books, I came across a word I hadn’t seen before.  It was the word canalize.  One of the characters in Heinlein’s book spoke about a person being canalized in his thinking.  The notion was that as kids we are very malleable in our thoughts and actions but through interaction with the society we grow up in, we eventually assume a course (like a canal) which is confluent with our environment - so much so that we are unaware of it.  Once we are on that course, its very hard to change.  The thoughts and habits of a lifetime are hard to adjust radically.

I think that design can be canalized as well.  What do I mean?  Well, let’s imagine an example.  Let’s suppose that we have a long method which consists of a sequence of validations.  I’m sure most people reading this can imagine the sort of code I’m talking about.  It’s just series of blocks, about five or ten lines apiece in which something (parameters, input data, etc) is being validated bit by bit.  This sort of code arises in most systems.  People do the first piece: a single validation, and then time goes on and they discover a need for a second validation and they add it.  For many people, it just never seems worthwhile to extract the validations to separate methods.  The code is rather clear.  There is no deep nesting; it’s just a long sequence of similar blocks.

Now, think about a canal. 

The water is flowing and carving its course.  What is the course in the code I just described?  Well, it seems that it doesn’t have one, really.  People can introduce just about any logic that they want to in that sequence.  The could start to do completely different work just because it is a convenient place to do it.  But, what if we started to extract each validation into a separate method?  People could still throw in random bits of logic, another responsibility, but they’d have to make a choice.   They could throw it into one of the extracted validations or throw it into the original method.  Either way, the code is a bit better off.  If the logic is thrown into one validation method, the other ones are fine.  If it is thrown into the top-level method, the one we extracted from, we at least have our courses in: the code shows a direction and people coming along later will at least be able to see that the application has a pattern in place for validation.

If you’re thinking that this is related to the “no broken windows” theory, you’re right.  But, I think there is a little bit more.  When we factor our code finely, we make some changes much harder than they would be otherwise.  Large methods are like stagnant pools.  All sorts of computation can be bound together in a big tangle.  However, once you factor a bit, and set a course, chances are no one will go through the process of inlining those methods.  It’s just not a natural thing to do.  We’ve given the code a bit more structure, and moved toward a design where validation is seen as a separate thing - people kind of know where it goes now.

Now, let’s try something else.  We can make the observation that our validations can be performed in any order.  The sad thing about most languages is that we have to put things in a particular order, even if order doesn’t really matter.  Text, in a file, has a natural order: top-down.  Could we do something a bit different?  One thing that we could do is turn our little validation methods into objects and put them in a set.  Then we could write code to iterate the set and fire each of them off.  Notice that I said “set.”  We could use a list, but lists have the same issue as a sequence of method calls in the text of a larger method: there is an ordering, but it doesn’t matter.  One the one hand, people might look at the ordering and think that it is significant.  On the other hand, people might start to take advantage of this accidental ordering and couple things together in a way in which it is hard to change.  In either case, seeing that the code uses a set rather than a list, at least betrays something about the intent of the designer.

Donald Norman uses the term affordance to describe qualities of things which tend to encourage or discourage particular uses. In software, Kevlin Henney has written about the affordances of interfaces: how they can communicate proper use to users.  I think that we can use the concept a slightly different area.  There are ways of writing code which encourage or discourage particular future changes in a code base. It’s never perfect, but it helps.  However, you can have too much of a good thing.  If you canalize too deeply, it can be harder to refactor your code toward a vastly different structure.   The paths you lay down are structure, and while structure is great for supporting our work, it also canalizes and reinforces itself over time. 

In the end, software development is a natural process.  We shouldn't be surprised to see the same sort of inertial change patterns arise that we see in nature.

Intelligent Design and Conway's Life

As much as I'm irritated by Creationism and Intelligent Design, I do think that I appreciate the depth of feeling behind them.  If the notion that humans are special to the degree that the universe is centered around them is part of your world-view, then evolution is quite a blow once you understand it.  It's much more of a blow than the notion that the Earth rotates around the Sun rather than the other way around. 

Knowledge moves in fits and starts.  We're okay with Copernicus's insights today and I think that it's because a Sun-centric model of the solar system is fairly easy to grasp.  Evolution, on the other hand, is a tough one.  The time scales are huge and the notion that life can spring from non-life just isn't intuitive for most people.  Fortunately, though, it can be.

One of my favorite little books is Turtles, Termites, and Traffic Jams. It's a book with an agenda. It seeks to make de-centralized behavior intuitive.  One of the examples in the book is a flock of birds.  We look at a flock and we see the lead bird and we kind of assume the flock will go wherever that bird leads it.  The truth is much more interesting.  It turns out that you can accurately model the actual behavior of a flock (the swooping and swirling) with a set of simple rules which each bird applies independently, sensing only the birds in its immediate vicinity.  And, in fact, this is how special effects designers create computer-generated flocks in the movies today.

The lesson of the book is that many of the patterns we see are the result of forces which are much more decentralized than we expect, but we have a cognitive bias against that point of view, so we don't see it.  We can, however, gain an accurate view of the "workings" of the world through simulations.  They can help us develop intuitions which make the truth a bit more plausible.

When I was a kid, I was very curious about the world.  I'm a bit less curious now, but not much.  For me, the notion that life could spring from inanimate matter seemed theoretically possible, I understood randomness and I understood the timescales, but I had trouble with the "shape" of the living behavior.  Birth, growth, and death are not the sorts of things that you see rocks doing.  Life is different somehow.  How could it be an extension of mechanics and physics?  

The thing which made it click for me is a simple simulation called Conway's Life. It's a simulation which plays out in a grid.  Each cell in the grid can be "on" or "off."  You start from an initial configuration, a set of "on" cells close to each other and then let it run.  The simulation  applies the same four simple rules over and over again.  New cells turn on and off and the net effect is that grid fills with extremely organic looking patterns which seem to have been generated out of thin air.

You can try it out Conway's Life here. Create a set of five or ten "on" cells close to each other (possibly touching, but not densely packed), then press start. Sometimes the cells will disappear quickly but sometimes they will balloon out into a web of activity.

If I could do one thing in today in the educational system it would be to get this simulation into the hands of children in elementary school.  The rules behind it are simple to explain and it provides a path toward understanding how life can be a consequence of the way the world is. That's as handy to know as the fact that the Earth revolves around the Sun, regardless of whether you are religious or not.

How To Fail With External DSLs

Software development, like any other industry, has trends.  Over the past couple of years it's been fun to watch the ascendance of the Domain Specific Language meme.  I’ve loved language design since I first started programming and I like to see people work in that design space.  Language design is challenging and immensely satisfying work.  It's been said that one of the oddities of software design is that we suffer from a lack of constraints.  At times, we have severe memory or time constraints, but for the most part, there is nothing that forces us to choose, say, one object structure over another other than our understanding of how it might impact future change.  When you design an embedded DSL -- a DSL built of constructs in a surrounding programming language, the structure of your classes has an immediate impact on the expressivity of your language.  It's API development writ large.

But, embedded DSLs are just one part of the DSL space.  The rest of the space is occupied by external DSLs -- standalone languages that compile down to some other language or a run-time.  In general, they give you a freer hand.  You can create a complete syntax and tune it directly to a particular domain.  Most of your constraints evaporate.  It’s a very tempting approach but it does have some pitfalls. When you are a consultant, you end up seeing many proprietary technologies that you can’t talk about.  There are more external DSLs out there than most people realize and more often than not, I see companies trying to get rid of them.

Why?

Well, there are a couple of reasons.  One is the cost of maintenance.   Someone in the organization needs to know enough about the DSL to make the inevitable changes.  In a way, this isn’t much different from the cost of maintaining a framework, but many organizations see it as a difference in kind because the tools that you use in an DSL implementation are often not the same as the tools that you use in conventional business applications.  It’s one thing to hire developers who know C# or Java or Ruby and yet another to hire or make sure that you have some who know ANTLR or YACC.  In some companies this is absolutely no problem.  They hire very knowledgable developers as a matter of course and they are sufficiently polyglot to make the cost of maintaining a language implementation negligible, but on the other hand, there are many companies that look at the cost side.  They write software, but their business isn’t software, so they are constantly trading off the cost of maintaining their DSL with the cost of dedicating those people to work that is closer to the business line.

One other reason why companies move away from external DSLs is that they can complicate hiring.  Years ago, I designed an external DSL for an organization that felt that they could save money by developing a high level language that their domain experts could use directly.  I spent a good amount of time on the implementation and it was used successfully for long time.  After a while, however, they dropped it. The domain experts were technical enough to use a mainstream programming language, but beyond that, moving away from the DSL made it easier to hire them.  Let’s face it, if you are looking for a job what do you want to have on your resume?  A language used by precisely one company or one that is known or used throughout the industry?

Like the cost of maintenance, hiring cost is just one piece of the total cost of ownership for a DSL but it can be the one which tips a decision.  One very public example is Erlang.  People are paying a lot of attention to Erlang today, but we shouldn’t forget that it was dropped by Ericsson in part because they were concerned with their ability to hire and train developers.  Many people know the story of Erlang, but what they don’t realize is just how common this scenario has been in the industry.  I get to see a lot of legacy code in my work, and a good amount of it is in homegrown languages that companies have trouble supporting.

The thing to remember about DSLs is that they are a commitment and the commitment doesn’t end when the initial implementation is done.  This is definitely true for external DSLs, but it’s also true for embedded DSLs, albeit at much smaller scale.  When you make a commitment, you have to make sure that you have the will to follow through and you also have to make a sober assessment about whether you will keep the commitment even under adverse business conditions.  If you can, it’s worth doing.  DSLs are powerful technology.

Enough about failure.  How can you succeed with DSLs?

Here are a couple of ideas:

1. If you are creating an external DSL, consider open sourcing it or, at least, consider forming some sort of an industry group around it.  This partially externalizes the cost of maintenance and avoids hiring trouble. 
2. If you can’t or won’t open your DSL(s) to the outside world, make sure that your developers are not expected to use them exclusively.  If you don’t, you can expect to pay higher wages for developers of equivalent skill than you would otherwise.  In negotiations, they will factor in the cost of working with little known technologies.
3. Consider embedded DSLs as an alternative. Often they are easy to construct and they can give you many of the same advantages.  Dynamic languages like Ruby are ideal for embedded DSL construction, but embedded DSLs have been written in just about every mainstream programming language.

I like external DSLs and I think we'll see more of them.  But remember, success is a function of far more than the technology.

Speaking in Russia

The week after next, I’ll be visiting Russia to do some training and some University lectures.

The lectures will be at Ural State University, Ekaterinburg. Each starts at 18:30:

Monday, Dec 8th Recovering a Code Base

Wednesday, Dec 10th Error Handling as a First Class Consideration in Design

Thursday, Dec 11th Design Sense – Reaching Consensus on Excellence

I’ll also be speaking to the Agile Russia group in Moscow Friday, Dec 12th.

The internet tells me it might be colder than Miami. I will probably bring a jacket.

Team Leadership Secrets of Sun Ra and Lope de Aguirre

It’s one of my favorite “beer in hand” statements when I get together with people at conferences: "I'd like to teach a day long seminar about team leadership.  I'd start with showings of Werner Herzog's Aguirre: Wrath of God, and Sun Ra: A Joyful Noise, then I’d follow it up with discussion."

Whenever I say that, I usually get stunned silence and furtive looks at the bartender (cut this guy off!) but I am serious.  I think that those two movies are not only entertaining, but a great launching point for a discussion of team leadership.

Aguirre: Wrath of God is a quasi-fictionalized account of Spanish explorer Lope de Aguirre.  Lope was part of an expedition across the Amazon to find El Dorado (the city of gold).  On the path down the Amazon, he killed two leaders of the expedition (one after the other) assumed leadership, destroyed villages and demanded that the other members of the expedition declare him Prince of Peru, Tierra Forma, and Chile.  In real life, he was shot and quartered.  In the film, he was the last man standing from the expedition, flailing in madness, lording over a pack of monkeys on a deserted river raft as it crept down the Amazon.  The film was a bit more poetic in that regard.

Herzog’s Aguirre (played by Klaus Kinski) is a textbook portrayal of ambition, narcissism, and sociopathy. Kinski’s mechanisations and visual ticks as his character leads his "team" on the mother of all death marches are incredible to watch, and learn from.  If anyone around you acts like Aguirre, run.

Sun Ra: A Joyful Noise on the other hand, is a documentary about a mid-20th century African-American jazz musician (Herman Poole Blount a.k.a. ‘Sun Ra’) who claimed that he was from Saturn, recruited one of the most promising saxophonists of his generation (John Gilmore) and  started a band called ‘Sun Ra Arkeststra.’  The Arkestra dressed up in space aged costumes, created their own rituals and avant-garde music and toured the US and Europe for years.

I'm not going to say much more about Aguirre: Wrath of God right now.  Instead, I'm going to leap into the Sun Ra film. Throughout it, we are led into the world of the band.  We hear Sun Ra philosophize about the world - how it is made of myth and how his myths are as good as anyone else's. Through it all, you notice how much the band is like a cult and you wonder why all of these people fell out of society to go along with something this goofy.  But on the other hand, the film is oddly mesmerizing.  You get the feeling that the band members really didn’t believe that he was from Saturn – that they gave up their normal lives to join this touring band so that they could play, not just in the musical sense but in the broader sense as well.

Everyone knows that cults can be dangerous, recent history is full of examples, but what about a cult that plays on the edge of "play"? Where everyone knows with a wink and smile that they are just playing? I can imagine it being a rather liberating experience.  In the film Sun Ra winks at us. It's all a game, but one he decided to play for fun. It's worth thinking about how much playing we do in real life, really - play which isn’t much fun because the nod and wink aren't there. 

Chef/author Anthony Bourdain once wrote that having a restaurant kitchen is like running your own pirate ship.  If you see work that way, how can it be anything other than interesting?

For years I’ve looked at software development teams and asked myself what makes the good ones tick.  It's hard to single out answers because teams can be good in many different ways.  One thing that I come back to often, though, is the sense that a good team has an inner life.  People joke with each other.  The feel somewhat protected against the other parts of the organization, and they have people who make it fun.  That last bit is an act of leadership and often it comes about because of a strong leader, someone who defines the mythos of the team – who we are, what makes us different?  The person doing this doesn't have to be a manager, but I have seen managers do it.  People who adopt this role instinctually look for the minor dramas that bind people together and foster them.  It's fun to see it when it happens.

There’s a vogue now for self-organizing teams in software development.  That works, to a degree, but as often as not, there’s a certain listlessness to many of them, a sense within them that everyone is always waiting for someone else to step up. Ultimately, I think leadership is an individual action, and it can be an act of play that facilitates good work.  I don’t suggest you jokingly tell your coworkers you’re from Saturn, but there are many actions south of that which might help.

Pessimistic Personal Finance: The Three Fuses

One of the nice things about blogging is that you get feedback.  You take an idea and you throw it out there.  If there’s a better way of seeing things, someone is bound to tell you.

With that introduction, I turn to the topic of the day: pessimistic personal finance.  These are scary times.  We’ve all seen the stock market plunge, but really the stock market is the least of our worries.  Unless credit frees up, we’re going to see economic disaster over the course of the next year.  We already know it’s going to be bad, but no one knows how bad.

I live by the motto that pessimists are rarely disappointed and often pleasantly surprised, so with that in mind, I looked at our family finances and calculated three metrics that I call fuses.  What is a fuse?  A fuse is a duration of time until  a bad thing.

The first fuse is what I call the normal fuse.  Assume some set of catastrophes knocks out all of your income.  How long can you and your family live on what you have now, without changing your spending at all?  You can calculate this fuse by taking your cash (pretty much anything liquid and not at the mercy of the market) and dividing it by your monthly spending.  The number you get is a duration in months.  That’s your normal fuse.

The second fuse is the ‘bunker’ fuse. Assume the same set of catastrophes and calculate your fuse, but this time divide your cash by your monthly spending in ‘hunker down’ mode.  In ‘hunker down’ mode, you’re spending only what you need to for rent/mortgage, insurance, and groceries.  You jettison all non-essential spending.  Again, don’t factor in things that can evaporate in the stock market.

The third fuse is the big one: the Armaggedon fuse.  Take the total of all of your assets, your retirement, your house, your investments, and any money on hand, and divide that by your essential monthly expenses.   Then, factor in the inflation rate, and you get your fuse. 

There you go.  Even if you aren’t a pessimist, I think these are good numbers to know.  If you know them, you can rest at night knowing exactly how you’d fare during a personal tragedy, an economic meltdown, or even a zombie attack.  Although for the latter two scenarios, guns and canned soups might be better than cash.

And, now a question: I can’t be the first one to have thought of these fuses.  Are these concepts floating around under different names?

Beyond Technical Debt

What happens to code when you don’t refactor?  Anyone with any experience knows the answer.  It gets messy.  It becomes hard to change and the rate at which you can add features slows to a crawl.

In the Agile Development community, we use the term “technical debt” to refer to this problem.  When you don’t keep your code clean, you incur debt, and as in other areas of life debt reduces your options – it can become crippling over time.

More and more, however, I wonder whether there could be a different way of looking at the messy code problem, a way that takes people's behavior into account.

The other day, I saw a news report that offered some advice for people who were cutting back on their spending.  The advice was to throw away your credit cards and use cash.  It turns out that people spend less when they are handing cash over to a cashier. 

When you think about it, it makes sense.  There is something fundamentally different about parting with something valuable that you own.  I have no doubt that credit card companies know this also.  People feel much freer spending money that, technically, isn’t theirs.  And, to the degree that the transaction doesn’t feel “lossy”, people will spend.   If recent history tells us anything, it tells us that in general people just don’t seem to mind accumulating debt.

The idea that I have in mind is to flip technical debt around.  We need to see clean flexible code as an asset and count it as an asset.  When a non-technical person wants something faster, we need to be able to talk about how much of their asset they would be losing if they want a rush job.  They also need to know how much it would cost to restore the asset, and it’s almost always far more than the original development cost.

The devil, of course, is in the details, but I do believe that we’re better off aligning with human nature.  People like to accumulate assets and they hate losing them.  We need to recognize the value of clean, supple code and make sure people understand the business impact of losing it.

Are Nested Classes Really a Good Idea?

It was so tempting to start this blog in the classic style: ‘Nested Classes Considered Harmful’, but people turn you off when they see that sort of thing and rightly so.  There are few hard and fast rules in software development, and when you take a strident position you end up in a flame war. You can’t have a conversation.   So, forget I wrote the “considered harmful" part.   

Let’s start this again.

I have strong feelings about nested classes.   I tend to avoid them and I don’t think I'm being irrational.  Here’s why.

1. Nested classes can make code less readable.  If they are short, it’s no problem, but it’s not uncommon to see one that spans half a page.  When that happens, the class it's embedded in is unavoidably less readable: it’s interrupted by the nested class.  Want to scan up and look at the imports or see what it inherits from?  Well you have to scroll over the nested class, and it’s worse when there are declarations above the nested class.  You lose the big picture.
2. Nested classes (in most languages) have special access to their enclosing scope.   “So what?”, I hear you say.  Well, whenever you have nested scopes you have the possibility of name conflicts.  In many languages, a new scope can silently hide variables and methods.  I’ve seen people spend 10 to 20 minutes debugging problems related to nested scopes.  Don’t get me wrong, I think it’s great to have a global scope and it’s great to have method scopes.  Namespaces are great too, but every level increases the chances of misunderstanding.
3. Nested classes are often hard to find.  “Pfaaaa!” I hear you say, “I use modern tools, I can find anything in a few keystrokes.”  Well, yes, but it’s amazing the number of times that you have to ask yourself where something is in a project, times when you are outside of a tool and file names are your only guidance.  Even when you are in your editor or IDE and able to just leap to a definition, you confront the semantic problem: when you arrive at a nested class you have to ask yourself “Gee, what does it pull from the outer scope?  Why is it here?  Is it hiding anything?”

So, at this point, I guess you can see why I’m not pressing this, why I’m not saying “nested classes considered harmful.”  It’s a style thing.   I just find that I don’t need them.  If I need a non-anonymous class, I have no problem promoting it to normal class scope.   I know that many people don’t like that.  They look at the cases where their little class is used only inside of another one and say “This is perfect, when I make this class an inner class I'm documenting the fact that it's used locally.  I avoid polluting the outer name space.”  That’s fine, but, really, if you’re that concerned with polluting the outer name space maybe it's too large already.  It’s something to think about.

To me, code is much clearer when nested classes are used sparingly. 

But, that’s just me.

The Flawed Theory Behind Unit Testing

I have Google’s blogsearch set to give me notifications about unit testing.  On an average week, I read dozens of blogs and mailing list discussions about the topic. Occasionally, I read something new, but there’s lot of repetition out there.  The same arguments crop up often.   Of all of them, though, there is one argument about unit testing which really bugs me because it rests on a flawed theory about testing and quality and sadly, it’s an argument that I fell for a long time ago and I’d like to lay it to rest.  Hopefully, this blog will help, but I have to relate a little history first.

Back in the very early 2000s, I had a conversation with Steve Freeman at a conference.  We were talking about Test-Driven Development and Steve had the strong feeling that most of the people who were practicing TDD at the time were doing it wrong - they'd missed something.

Steve was and is part of a close-knit community in London who have been practicing XP and TDD from the very beginning.   Among the fruits of their labor was the entire notion of mock objects.  Steve Freeman and Tim MacKinnon wrote the paper that introduced the idea to the broader community.  The rest is history.  There are mock object frameworks out there for nearly every language in common use.

Mock Objects, however, are part of a larger relatively unpublicized approach to TDD.  The story I heard was that it was all started by John Nolan, the CTO of a startup named Connextra. John Nolan, gave his developers a challenge: write OO code with no getters.  Whenever possible, tell another object to do something rather than ask.  In the process of doing this, they noticed that their code became supple and easy to change.  They also noticed that the fake objects that they were writing were highly repetitive, so they came up with the idea of a mocking framework that would allow them to set expectations on objects  - mock objects.

When Steve told me about this approach, I thought it sounded okay but I there was one thing that I couldn’t wrap my head around – Steve, and Tim, and the people who had been on that team were using mocks extensively.  In fact, they used mocks whenever they could.  This was a bit different from the way that I was practicing TDD.  What I did, in general, was use tests to drive a class and then I extracted new classes from the class I was designing as it became bulky.  Some tests would cover just one class, but others would cover several classes working together.

The problem that I saw with the mock object approach was that it only tested individual classes, not their interactions.   Sure, the tests I wrote were nominally unit tests, but I liked the fact that they occasionally tested real interactions between a class and its immediate collaborators. Yes, I liked isolation but I felt that this little tiptoe into integration level testing gave my tests a bit more power and a bit more strength.  But, there was only one problem.  The team at Connextra that was using mocks extensively was reporting extremely low defect rates.  I just wasn’t sure how they were getting them.  After all, it didn’t seem like there was any integration testing going on.  Their application should have been rife with integration errors.  Or should it have?  Let’s examine our reasoning.

One very common theory about unit testing is that quality comes from removing the errors that your tests catch.   Superficially, this makes sense.  Tests can pass or fail and when they fail we learn that we have a problem and we correct it.  If you subscribe to this theory, you expect to find fewer integration errors when you do integration testing and fewer “unit” errors when you do unit testing.  It’s a nice theory, but it’s wrong.  The best way to see this is to compare unit testing to another way of improving quality – one that has a very dramatic measurable effect.

Back in the 1980s, there was a movement to use something called Clean Room Software Development.  The notion behind Clean Room was that you could increase quality by increasing the rigor of development.  In Clean Room, you had to write a logical predicate for every little piece of your code and you had to demonstrate, during a review, that your code did no more or less than the predicate described.  It was a very serious approach, and it was a bit more radical than what I just described: another tenet of Clean Room was that there was to be no unit testing.  None.  Zilch.  When you wrote your code it was assumed correct after it was reviewed.  The only testing that was done was stochastic testing at the functional level.   

Amazingly, Clean Room worked.  Clean Room teams demonstrated very high quality numbers.  When I read about it, I was stunned, but then I came across a passage in a book that I was reading about the process. The author said that many programmers wrote their predicates after writing a section of code, but that experienced programmers often wrote the predicates first..  Gee, that sounds familiar, doesn’t it?  In TDD, we write the test first and the test is, essentially, a specification of the behavior of the code we are about to write.

In the software industry, we’ve been chasing quality for years.  The interesting thing is there are a number of things that work.  Design by Contract works.  Test Driven Development works.  So do Clean Room, code inspections and the use of higher-level languages. 

All of these techniques have been shown to increase quality.  And, if we look closely we can see why: all of them force us to reflect on our code.

That’s the magic, and it’s why unit testing works also.  When you write unit tests, TDD-style or after your development, you scrutinize, you think, and often you prevent problems without even encountering a test failure.

Now, as you’re reading this, you might think that I'm saying that we can get away with doing nothing as long as we sit back in our chairs, rest our chins on our hands, and think about our code.  I don’t think so.  I think that approach may work for short periods for some people, but software development is a long-haul activity.  We need practices which help us achieve continuous discipline and a continuous state of reflection.  Clean Room and TDD are two practices which, despite their radical differences, force us to think with absolute precision about what we are doing.

I have no doubt that a team could do well with, say, Clean Room, but personally, I like the fact that the tests we end up with using TDD give us additional leverage: they make it easier to change our code and know that it is still working without having to re-reason through the entirety of the code base.   If you have to change your code often it’s not the best use of your time, especially when you can write tests that have recorded and embodied that reasoning and run them at will.  With those tests in place, you free yourself up to reason about other things that you haven’t reasoned about in the past, rather than repeating yourself endlessly.

But, enough about TDD. 

My point is that we can't look at testing mechanistically.  Unit testing does not improve quality just by catching errors at the unit level.  And, integration testing does not improve quality just by catching errors at the integration level.  The truth is more subtle than that.  Quality is a function of thought and reflection - precise thought and reflection.  That’s the magic.  Techniques which reinforce that discipline invariably increase quality.

Are FP and OO Incompatible Syntactic Styles?

I've been looking at Scala, OCaml, and F# recently and I've really been struck by their different approaches toward mixing OO and FP.  I can't say that I like any particular approach, really.  OO and FP do seem to be odd bedfellows in a language and I think that I see why now.  They use syntax in rather different ways.  If you aim for one unified syntax in hybrid FP/OO languages the things that you do to tune the syntax toward one end seem to detune from the other.

Let's take a look at how we structure function calls/applications across several languages:

C:                a(b,c);
Java:             b.a(c);
Smalltalk:        b a: c
Lisp:             (a b c)
Haskell:          a b c

The C syntax is a straight procedural.  We have a function a and we need to pass b and c along.  The Smalltalk snippet is in the object-oriented style.  We privilege b and say that it owns the a method.  We get polymorphic substitutability in the bargain.  Lisp doesn't privilege any particular argument, so it is sort of like procedural, although people do write OO code in that style in Scheme and CLOS.  The Haskell example, however, is very interesting syntactically, because it is tuned to make this natural:

let x = a b in x c

In this example, we've bound x to the partial application of a to b and then used it to form an expression x c which is semantically the same as a b c.

Partial function application is powerful.  People working in Haskell and the other ML-derived languages use it routinely to compose operations.  It's a shame, though that its syntactic style is so different from the OO style.  Here's an example that drives it home.  Imagine that we want to get the second element of a list in a couple of hypothetical languages.  Here's a Haskell-ish way to do it:

head (tail [1,2,3])

and here's the OO-ish way:

[1,2,3].tail.head

See the difference?  FP is more of a prefix style, while OO is postfix.  One could say that this is arbitrary, that you could have an OO language which uses the prefix style, but I'd argue that there is something deeper here. 

Let's imagine what our code would be like if we attempted to do partial function application with OO syntax. 

let remainder = [1,2,3].tail in remainder.head

The partial function application doesn't really buy us much here.  Partial application is powerful when it composes functions.  Often the data that we are working with is the least interesting thing to compose, yet it comes first in OO expressions and the natural style of partial function application in language design is to keep the first N terms and leave off some trailing ones.

It does seem that functional programming gets some advantages from its prefix syntax.  Scala, F#, and OCaml do show that you can mix the styles, but you can't really get a unified one without picking the prefix form or the postfix form.  The ML derived languages have definitely found a syntactic sweet spot and it does seem like it's a poor mix with object orientation.