Michael Feathers

Temporal Correlation of Class Changes

How do you know that you have the right architecture for your application?  First of all, we have to define what "right" is, and there are many opinions.  

One worthwhile goal for an application is to have classes that are reasonably independent.  If we have some functional change to make, we should be able to go to a single class and make the change.  If we have to modify several classes it could be because we are introducing a large feature, or it could be because of the way that our application is decomposed.  Adding a field to a model class might require changes in a view, and that's normal.  It's a side effect of separating representation and presentation. 

In many cases, though, we end up making changes in several places because we have faulty abstractions.  If many changes to one class necessitate change to another, it's an important piece of information.  It could be an indication of the code smell that Martin Fowler named 'Shotgun Surgery' in his book `Refactoring: Improving the Design of Existing Code' [Addison-Wesley 1999].  'Shotgun Surgery' is essentially, the code smell that you have you find that adding features requires you to make changes spread across wide areas of the code base.

 The sad thing about 'Shotgun Surgery'' is that we're pretty much left up to our day to day experience to detect it.  We might get the sense that we are touching too many areas of our code, but that is a general feeling.  Do we really know which classes are tied together in our problem space?  Do we know what other classes we are likely to touch when we touch the one we are working on? It turns out that we have all of the information we need to figure this out in our source code repositories.

Representing Code Change

Over the past few months, I've settled on an intermediate form that I use for data I gather from source code repositories.  It makes change analysis and many other forms of analysis rather easy.  The central data structure is a method-change-event.  Each method change event has the following fields:

- type (method add, change, or delete)
- method name (fully qualified with class/modules)
- method body length
- file name
- sha1
- commit date 
- committer

With this information, I can track the changes of individual methods across their entire lifetimes.  I can also do higher level analysis of qualities associated with files and classes.  

Approach

When methods from two or more classes are changed in the same day by the same committer, we can say that the classes are somewhat correlated in time.  When this happens often their correlation is high. For our purposes, it's probably be sufficient to compute the pairwise correlation of classes.  

Given that we have a set of events which represents the state of every method each time it is changed in our project's history, we should be able to find all of the classes that have been touched by each committer on each day.  We can then take all pairs of those classes and throw them into a set.  When we do this for each commit and total the number of times each pair occurs across all of the commits, we should be able to see the relative frequencies of the pairs.  When we sort them, we'll see which classes change most often with each other.

Fortunately, this sort of analysis is rather brief in Ruby.  Here is the whole of the computation after a few extensions to Array and my method change event class:

events.group_by {|e| [e.day,e.committer]}.values   
  .map {|e| e.map(&:class_name).uniq.combination(2).to_a }    
  .flatten(1).norm_pairs.freq_by {|e| e }.sort_by {|p| p[1] }

When you examine these sorts of frequencies, they typically have that power law-ish shape:

Once you have a sorted list of the highest frequency class pairs, you can march through them and see if there are any surprises. Temporal correlation of change across domain objects is particularly informative. Altering the correlation range from a day to a week can be useful too.

Finessing Away Errors

I think I've written about this before but I don't care.  I'm going to keep harping on it.  Error detection and handling should be on the edges of programs, not on the inside.  You shouldn't have to get halfway through your execution to discover that something is wrong.

It's nice to have this sort of philosophy, but how do you put it into practice?  Let's look at an example.  

This morning I was working on some code repository mining code and I needed to group some events.  I knew what to do, I reached for group_by in Ruby.  It's a method on arrays which takes a block and executes for each element in the array.  Every unique return value from the block turns into a key in a hash that group_by returns.  The values in the hash are all of the elements which produced the result value when the block was run on them.

Here's a simple example:

  events.group_by {|e| e.method_name }

When I execute that, I get a hash with method names as keys.  Each key in the hash is associated with all of the events that have that method name.

So, where do the errors come in?  Well, the method names in my system are fully qualified.  "Utilities::Config::search" is an example.  The name denotes a method named search on the Config class in the Utilities module.  Now, suppose that I want to group on the names of classes?

It seems easy enough.  Here's a Ruby function that which will let me do that:

  def class_name method_name
    method_name.split('::')[0..-2].join('::')
  end

  events.group_by {|e| class_name(e.method_name) }

The function takes the name and splits it on '::'.  Then it takes all segments except for the last and joins them up again.  That should be our class name.  There is a problem, though.  It doesn't work for methods without a class.  It returns nil because [0..-2] on an empty array is undefined.

Hmm.. What can we do?  Well, my first tack was to just return "" from class_name when there was only one segment:

  def class_name method_name
    return "" if method_name.split('::') <= 1
    method_name.split('::')[0..-2].join('::')
  end

but then we have to get rid of that entry:

  events.group_by {|e| class_name(e.method_name) }.select {|k,_| not k.empty? } 

More than a bit ridiculous, eh?

Is there a better solution?   Exceptions to the rescue!!   No, just kidding.

Whenever you have an error, it pays to see if you can turn it into a normal case.  In this case, it's very easy.  It turns out that all methods in Ruby are part of a class.  When you don't see a class in the source file, Object is the implicit class.

Here's the fix:

  def class_name method_name
    return "Object" if method_name.split('::').count <= 1
    method_name.split('::')[0..-2].join('::')
  end

Now all methods that are defined without a class are binned under Object.  For instance, our class_name method will be seen as Object::class_name.

It might seem that we got lucky here, that the design of Ruby gave us an easy default.  Well, the truth is more sublime.  Ruby is playing the same trick that we are.  In essence, Ruby is saying that methods that don't appear to have a class aren't really special.  They are part of Object.   Here's the nice thing.  If Ruby hadn't played this trick, we could've.  We could've created a name like "no_class" and used it as a bin for all methods that aren't "part of" a class.     

Now that we have our class_name function, we can group away in an error-less way:

  events.group_by {|e| class_name(e.method_name) }

And we can compose without checking for error cases:

  events.group_by {|e| class_name(e.method_name) }.values.map(&:count).freq

because there are none.

Will Rogers Does Software Development

There's an old paradox from the humorist Will Rogers that uses the migration of people during 1930's Great Depression as a backdrop.  Rogers said "When the Okies left Oklahoma and moved to California, they raised the average intelligence level in both states."  

It certainly sounds paradoxical, but if you think about it a while, you realize that it's possible if the bottom-most people in Oklahoma were smarter on average than the people in California.  They leave, Oklahoma's average IQ goes up and so does California's when they arrive.

It's a neat puzzle, and in medicine they've actually named a statistical characteristic of cancer studies the `Will Rogers Phenomenon' after this paradox.

Closer to home, for us, I think the Will Rogers Paradox applies to workplaces.  I've noticed that they stratify.  If you pick two companies, A and B in a geographical region, they are either rather close to each other in terms of developer skill level, or they are wildly different: the top developers at A would just barely being able to keep up at B, and if developers from B moved to A, they'd be bored senseless and they'd bolt back to B.

I don't have any hard evidence for this, but it does seem like you don't have as many places in the middle.  Pick two organizations at random in an area and it's likely that they are either on par or wildly different. 

Developers self-select and we end up with stratification of skill across the industry. 

Thoughts?

 

 

UML Out of the Box

"Do you know UML?"  I used to say these words often when I started to work with new teams.  Nowadays, I just just start drawing boxes and lines.  I find I that I can explain as I draw and people grok it pretty fast.  "So, you're saying that a Flippet has many Finkles?"  I draw the little asterisk to indicate "many" when I say this. 

There isn't much to UML.  It's just boxes and lines. 

A long time ago, people did try to turn UML into a big thing.  Maybe they succeeded.  No!  Don't go off and do a search for "UML" now.  You're better off not knowing what they did.  It's just boxes and lines that we use to explain ideas.  And, you know, you are the boss.  You can use ampersands instead of asterisks when you use UML in your team, and you can even call it UML.  No one is going to sue you.  

The thing is.. boxes and lines got a bad rap a long time ago.  They were generally seen as a way of avoiding "real work."  And, there's a lot of truth to that perception. 

In the past, people did spend ungodly amounts of times getting all of their boxes and lines in the row before writing code.  I worked on a team like that once.  Those were trying times. But, drawing boxes and lines before you write code is only one way of using UML.  The niftier thing you can do is draw them after you've written code.  "Why?" you might say.  Well, because it's good to get a picture of what you have that's bigger than the class in front of you.

It's nice to think that we "kind of know" what's connected to what in applications we work in day to day, but you'd be surprised.  Actually, let me put that another way - you will be surprised if you take the time to sketch out the design of your existing code.

Sketch. Forget about UML generating tools.  They are a pain.

Oh, and there's a lot we can say about critiquing a design once we have a nice sketched UML view of it.  But, here's a good short rule of thumb: 

Boxes good.  Lines not so much.  

When you have too many lines connecting things crazily, your code is in trouble.

Isn't it funny that when we look at a project the things that turn into boxes are more visible than the things that turn into lines.  Maybe that's part of why it happens. 

 

The Carrying-Cost of Code: Taking Lean Seriously

I've spent the past 8 years or so looking at ugly code.  This isn't uncommon in software development but in my case, I've been looking at different ugly code developed by different teams every couple of weeks.  One question that people often have is whether to refactor or rewrite.  It's never a simple call.  Usually, this is because the reason that people want to rewrite code is because they don't understand it.  Yet, rewriting code often requires us to understand it well enough to proceed with the rewrite, especially if there are existing customers who depend on all of the nuances of behavior that the system has consistently exhibited.

When you see enough of this, it's easy to take a bleak view.  There are many places in the industry where existing mountains of code are a drag on progress.  The worst thing is that quite often organizations are so close to the problem that they can not see its full extent.  When business sponsors receive the cold-shoulder enough times when they want particular features, they stop asking. They have learned that "changes to X are expensive" and they gradually shape their business in different ways.  The effects ripple out from there.  Younger organizations without as much software infrastructure often have a competitive advantage provided they can ramp up to a base feature set quickly and provide value that more encumbered software-based companies can't.  It's a scenario that plays out over and over again, but people don't really talk about it. When Agile came around there was a lot of talk about flattening the cost of change in code.  With good practice, you can do quite a bit of that.  However, entropy happens. Older codebases are typically harder to change.

A few years ago, I was telling some friends about an experiment I would love to run.  I'd like to have code base where every line of code written disappears exactly three months after it is written.  If you were able to get past people gaming the system by copying code someplace else and then copying it back in when it was deleted, you'd have a very interesting set of constraints.  Developers would be rewriting code constantly and they'd develop insights into ways to rewrite the code more compactly and (hopefully) more understandably.   Perhaps more importantly, the business would have to make very serious tradeoffs about the features.  If you are limited to a smaller number of features (because code keeps disappearing), you have to make sure that the ones you keep are really the ones which are making you money.  I have the suspicion that a company could actually do better over the long term doing that, and the reason is because the costs of carrying code are real, but no one accounts for them.

Recently, there's been a strong move toward Lean in the industry.  The roots of Lean are in manufacturing - the Toyota Production System, etc.  I like some of the concepts, but I remember being shocked early on by what the lean community considers to be "inventory."  In manufacturing, inventory is a very clear concept.  It is the extra stuff hanging around.. the things that partially done and queued for the next bit of work. In Lean Software Development, requirements are often seen as inventory.  If you spend a lot of time elaborating requirements for features you are not going to work on for a while your process isn't streamlined enough.  That's fair, but I think that the brutal reality is that we have something much more tangible that we can see as inventory: our code.

Let's go back to manufacturing.  If you are making cars or widgets, you make them one by one.  They proceed through the manufacturing process and you can gain very real efficiencies by paying attention to how the pieces go through the process.  Lean Software Development has chosen to see tasks as pieces.  We carry them through a process and end up with completed products on the other side.

It's a nice view of the world, but it is a bit of a lie.  In software development, we are essentially working on the same car or widget continuously, often for years.  We are in the same soup, the same codebase.  We can't expect a model based on independence of pieces in manufacturing to be accurate when we are working continuously on a single thing (a codebase) that shows wear over time and needs constant attention.

No, to me, code is inventory.  It is stuff lying around and it has substantial cost of ownership. It might do us good to consider what we can do to minimize it.

I think that the future belongs to organizations that learn how to strategically delete code.  Many companies are getting better at cutting unprofitable features in their products, but the next step is to pull those features out by the root: the code.  Carrying costs are larger than we think. There's competitive advantage for companies that recognize this.

 

The Urge to REPL, the Urge to Test

When I was working in C++ and Java, I had this strange thought.  I thought that much of what we were trying to do with unit testing was targeted at giving ourselves some of the benefits of a read-eval-print loop in a compiled language.  The nice thing about testing is that we can get rapid feedback.  A REPL gives us that too, albeit without the ability to run it over and over again. 

I wonder, though, whether we can have have it both ways.  A while back I did the StringCalculator Kata in Haskell, and even though I wasn't writing tests in a file, I adopted a particular rhythm of working.  I typed expressions at the REPL containing functions that didn't exist yet, then I wrote them and "arrowed up" in the history to reevaluate them.  It seemed to me, that if I dumped a transcript of that session it wouldn't take much work to fashion it into a set of tests.  The process could even be automated, although, doubtless, you probably have to go back and edit the source: drop redundancies, name test cases well, etc.

The neat thing about a tool like this is that it wouldn't have to be perfect.  And, perhaps, the user would be able to give hints at the REPL.  Sort like "break here when translating to tests."  

Someone has to have done this.  Anyone know? I know Python has doctest but it doesn't seem to be quite the same.

Blog inspired by Mark Simpson: http://verdammelt.posterous.com/tdd-and-repl-analogies

 

Map/Reduce in the Micro

One of the downsides of spending a good deal of time exploring functional programming is that you get to the point where you don't like this sort of thing much any more:

print ","; @method_names.each { | name | print name + "," }; print "\n"

Well, okay... upside, downside.. I'm not sure.

The issue with this code is that it mixes printing, iteration, and formatting. It such a small snippet, it's no big deal but, if we're game we can move to a functional style, peel off at least one of these responsibilities and push it off to someone else.

The trick is to use Map/Reduce.  Wait.. no.. stop.. I'm not talking about pulling out a Hadoop-like framework for a simple printing task.  I'm talking about the pattern.  How about this?

We can replace each with map and then use reduce to build up our result:

puts "," + @method_names.map { | name | name + "," }.reduce(:+)

The map method distributes our work, and the reduce method collects the results. 

Map/Reduce: it's not just for data centers any more.

Sidenote:  Haskell has a nice function called intersperse which takes an element and a list and produces a new list with the element interspersed.  It's easy enough to write, but then I wouldn't get to talk about mapping and reducing.

 

Beyond 'Conway's Law'

When I'm "out and about" talking to people about software recently, it always seems that I circle back to "Conway's Law."  It's been on my mind quite a bit recently.  I'm continually stunned by the number of people who haven't heard about it.

 In a nutshell, 'Conway's Law' is a maxim named in reocognition of Melvin Conway.  He said that "...organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."  In many restatements of the law, the word "constrained" is replaced with "doomed", but that is another story. 

The gist of the law is that people and social structures are a non-removable part of the development picture.  It's nearly impossible to have separated teams in a large project without someone developing some sort of a narrow software interface between the code that each team is working on.  And, that makes sense.  You communicate more with the people on your team and less with the people "over there."  As a result, you are more likely to ask yourself "what's the piece of that other system that I need to know about" and the people on the other team start thinking "how do we provide some way for teams to use our stuff without mucking around in its internals."  The result of the clash of these two forces is often an API.  Oddly enough, even if people aren't thinking consciously about it, they still arrive at something which approaches a rough API.

When people first hear about Conway's Law, they often get a jolt of recognition and then a sense of doom. They remember all of the big systems they've worked on and they realize that, in a way, their architectures were much more at the mercy of organizational politics than they had hoped.  But, there is a bright side.  When an organization is aware of Conway's Law, it can leverage it to build great systems.

So, oddly, when I started this blog, I didn't want to talk about Conway's Law.  I wanted to talk about something slightly beyond it.  Conway's Law is about software structure mirroring organizational structure.  I think there's a corollary related to process - the way that we work.

Peter Provost at Microsoft related some stories in 'The Butterfly Effect', a presentation he gave at Agile2010.  He described the effect of having a quality-oriented culture that attempts to solve quality problems by making it harder to check in.  These sorts of cultures aren't uncommon.  I've run into quite a few of them.  Often things have to get pretty bad before a high "checkin-gate" is put in place, but sometimes it's just a response to scale.  If you have many developers on a project, you want to have some control, and requiring code reviews, and QA reviews, and style checks, etc can be a way to do it.  How can that be bad?  Well, for one thing, people do respond to disincentives.  When you put a time and frustration tax on checking in code, people just don't do it as often.  Transparency of work suffers and, quite like a dropping a pebble in a pond causes ripples, you can have delays of higher duration rippling out from that slow cadence of commits. Interestingly enough, quality can suffer as a result. Chalk this one up to the 'Law of Unintended Consequences', but still it's not quite what I was aiming to write about. 

 The thing I wanted to write about is how all of this affects the code.  In the day, I used to visit a lot of teams that used an "honest to god" waterfall process.  I know that some people today think that the waterfall process is some sort of a myth, but you know, it really happened.  All over the place.  The thing is, there was a studiousness to a lot of the code that I saw back then.  Today, we'd probably call it over-engineered, but it did seem like was shaped a little bit differently from a lot of code I see today.  More macro coherence than micro coherence.  It's hard to get at what it is without over-generalizing so I'll spare you.  But, really it should just stand to reason that this is true.  Our cycle times for change are far more rapid than they used to be and that has to have some influence on shape.  The thing that has me very curious right now, is how 'continuous deployment' might affect the shape of the design of our code.  When you are in a continuous deployment environment there is even more pressure toward small steps, and people are more likely to use devices like feature toggles when they want make larger changes.  And, well, that's sort of like a tax also.  We shouldn't be surprised if our code ends up having a different shape.

So, back to Conway.  I think there's a related law that has to do with process, but I don't think it's as a simple as Conway and I don't think it is as simple as saying that designs mirror the process structure used in their production.  Perhaps, over time, we'll know more.

 

Sensory Modes in Programming

Today I got caught up reading a back and forth in twitter about a proposed assertion syntax for Ruby testing frameworks.  It was interesting, but yet again it was about how to make tests 'read well.'  It's hard to disagree with that, right?  Well, I don't in principle, I just think about the amount of time we spend trying to warp programming syntax into English and I wonder whether it is really worth it.  If it is, maybe we need more malleable languages.. more malleable than Ruby.  Or, maybe we don't.  Maybe it's just too much fun.. maybe there's too much 'game' in molding an existing language into what we want it to be.  But, I'm getting off topic.

The thing I wanted to blog about is the clash between this natural language style of programming and the other sorts of programming we do.  In the agile community, seeded by Kent Beck and Ward Cunningham, there is this meme that the code should tell a story.  Luckily, our languages are nimble enough to help us with this.  We can use evocative names and string together code lines so that they read like a book.  Yes, it's work, but it is very doable.  OO was made for that sort of thing. 

Let's look at an alternative (avert your eyes if you must):

`git log #{@filename}| grep ^commit`.split(/\n/).map(&:split).map {|fields| fields[SHA1_COLUMN] }

This is a line of code that I wrote to pull the sha1s out of the output of a git log command.

Ugly?  Yes, but I can make it better.  Let's pull out the bash by saving the shell call's result into a variable called 'text':

text.split(/\n/).map(&:split).map {|fields| fields[SHA1_COLUMN] }

Better?  No, I can still hear you saying "yuck!"  I'm not going to argue with you, but I am going to say that that I think that this code is an example of an equally valid mode of expression.  It's just a visual mode rather than a narrative one.  

Let's bring it home.  Try to explain the code out loud: "take the text and split it into lines, then split those lines into fields and map each line to the field containing the sha1." 

Maybe some people do sub-vocalize that sort of thing to themselves when they see code like that, I just know that I don't.  When I see code like that, I construct a picture of what it is doing inside my head.  Words don't play into it much.

I don't think I'm alone in this.  We have different ways of making code understandable and making it 'read like English' isn't the only way. 

I remember years ago reading about something called Neuro-Linguistic Programming, and although I felt that much of it was popularized off the deep end, the idea that they had that we have different cognitive modalities is something I'll always remember.  According their theory, some people are far more visual.  Others prefer to get information in an auditory way, and often they subvocalize when they are thinking.  

No doubt this is a ruthless simplification, but what I notice about programming is that there aren't really any fixed boundaries.  We go back and forth between more structural (a.k.a functional, visual) ways of thinking about code and more narrative (auditory) ways. 

When I look at really good functional code (and believe me the piece I have up there isn't a good example) there is a balance between good naming and structure that reveals computation. 

It just isn't narrative. 

It's a different mode and I think we have to recognize it. 

Brutal Refactoring

Every once in a while, people ask me whether I'm going to do a 2nd Edition of 'Working Effectively with Legacy Code.'  I've dodged the question for quite a while because, on the one hand, it would be nice to move on.  There are so many other things I'm interested in.  On the other hand, I've learned a lot since the book was published and it would be a shame to just let that all slip away. 

These days, I'm much more aggressive in my approach to old code.  WELC was fully ingrained in that "if we don't have tests, we can't do much" attitude.  I think that part of that was a sign of the times, and part of it was a reflection of my natural fear as a consultant.  You walk in and you don't know anything about the code base so you can't even come close to accurately assessing risk.  Nicely, though, people who are embedded in teams often can, and I've learned a lot from people who've tried things out long term in their code and have lived to tell the tale.   

Beyond that, I've also learned at a deep level that what we call ugly code is really just code that was shaped by different forces.  It's always going to bear the mark of those forces and we have to go with what is there. What software wants to be is often very different from what we want it to be.  It turns out that by recognizing and aligning with these forces we can gain some leverage. Some of the funk we see in out existing code isn't going to go away, but that doesn't mean that we can't get out the pruning shears and make some rather large changes to make it all manageable.

So, what I've decided to do is organize the material I have and build it up into another book.  The working title is Brutal Refactoring.  The goal is describe how to take the big steps that make things easier and how to deal with software in the transition.  Along with that, there are smattering of topics related to the ROI of refactoring, the economics of code change and new strategies for large method and large class decomposition.

On Monday, April 4th, I'll be teaching my first course on this material in Chicago.  If you'd like to attend, there's more information here.