Michael Feathers

How To Fail With External DSLs

Software development, like any other industry, has trends.  Over the past couple of years it's been fun to watch the ascendance of the Domain Specific Language meme.  I’ve loved language design since I first started programming and I like to see people work in that design space.  Language design is challenging and immensely satisfying work.  It's been said that one of the oddities of software design is that we suffer from a lack of constraints.  At times, we have severe memory or time constraints, but for the most part, there is nothing that forces us to choose, say, one object structure over another other than our understanding of how it might impact future change.  When you design an embedded DSL -- a DSL built of constructs in a surrounding programming language, the structure of your classes has an immediate impact on the expressivity of your language.  It's API development writ large.

But, embedded DSLs are just one part of the DSL space.  The rest of the space is occupied by external DSLs -- standalone languages that compile down to some other language or a run-time.  In general, they give you a freer hand.  You can create a complete syntax and tune it directly to a particular domain.  Most of your constraints evaporate.  It’s a very tempting approach but it does have some pitfalls. When you are a consultant, you end up seeing many proprietary technologies that you can’t talk about.  There are more external DSLs out there than most people realize and more often than not, I see companies trying to get rid of them.

Why?

Well, there are a couple of reasons.  One is the cost of maintenance.   Someone in the organization needs to know enough about the DSL to make the inevitable changes.  In a way, this isn’t much different from the cost of maintaining a framework, but many organizations see it as a difference in kind because the tools that you use in an DSL implementation are often not the same as the tools that you use in conventional business applications.  It’s one thing to hire developers who know C# or Java or Ruby and yet another to hire or make sure that you have some who know ANTLR or YACC.  In some companies this is absolutely no problem.  They hire very knowledgable developers as a matter of course and they are sufficiently polyglot to make the cost of maintaining a language implementation negligible, but on the other hand, there are many companies that look at the cost side.  They write software, but their business isn’t software, so they are constantly trading off the cost of maintaining their DSL with the cost of dedicating those people to work that is closer to the business line.

One other reason why companies move away from external DSLs is that they can complicate hiring.  Years ago, I designed an external DSL for an organization that felt that they could save money by developing a high level language that their domain experts could use directly.  I spent a good amount of time on the implementation and it was used successfully for long time.  After a while, however, they dropped it. The domain experts were technical enough to use a mainstream programming language, but beyond that, moving away from the DSL made it easier to hire them.  Let’s face it, if you are looking for a job what do you want to have on your resume?  A language used by precisely one company or one that is known or used throughout the industry?

Like the cost of maintenance, hiring cost is just one piece of the total cost of ownership for a DSL but it can be the one which tips a decision.  One very public example is Erlang.  People are paying a lot of attention to Erlang today, but we shouldn’t forget that it was dropped by Ericsson in part because they were concerned with their ability to hire and train developers.  Many people know the story of Erlang, but what they don’t realize is just how common this scenario has been in the industry.  I get to see a lot of legacy code in my work, and a good amount of it is in homegrown languages that companies have trouble supporting.

The thing to remember about DSLs is that they are a commitment and the commitment doesn’t end when the initial implementation is done.  This is definitely true for external DSLs, but it’s also true for embedded DSLs, albeit at much smaller scale.  When you make a commitment, you have to make sure that you have the will to follow through and you also have to make a sober assessment about whether you will keep the commitment even under adverse business conditions.  If you can, it’s worth doing.  DSLs are powerful technology.

Enough about failure.  How can you succeed with DSLs?

Here are a couple of ideas:

1. If you are creating an external DSL, consider open sourcing it or, at least, consider forming some sort of an industry group around it.  This partially externalizes the cost of maintenance and avoids hiring trouble. 
2. If you can’t or won’t open your DSL(s) to the outside world, make sure that your developers are not expected to use them exclusively.  If you don’t, you can expect to pay higher wages for developers of equivalent skill than you would otherwise.  In negotiations, they will factor in the cost of working with little known technologies.
3. Consider embedded DSLs as an alternative. Often they are easy to construct and they can give you many of the same advantages.  Dynamic languages like Ruby are ideal for embedded DSL construction, but embedded DSLs have been written in just about every mainstream programming language.

I like external DSLs and I think we'll see more of them.  But remember, success is a function of far more than the technology.

Speaking in Russia

The week after next, I’ll be visiting Russia to do some training and some University lectures.

The lectures will be at Ural State University, Ekaterinburg. Each starts at 18:30:

Monday, Dec 8th Recovering a Code Base

Wednesday, Dec 10th Error Handling as a First Class Consideration in Design

Thursday, Dec 11th Design Sense – Reaching Consensus on Excellence

I’ll also be speaking to the Agile Russia group in Moscow Friday, Dec 12th.

The internet tells me it might be colder than Miami. I will probably bring a jacket.

Team Leadership Secrets of Sun Ra and Lope de Aguirre

It’s one of my favorite “beer in hand” statements when I get together with people at conferences: "I'd like to teach a day long seminar about team leadership.  I'd start with showings of Werner Herzog's Aguirre: Wrath of God, and Sun Ra: A Joyful Noise, then I’d follow it up with discussion."

Whenever I say that, I usually get stunned silence and furtive looks at the bartender (cut this guy off!) but I am serious.  I think that those two movies are not only entertaining, but a great launching point for a discussion of team leadership.

Aguirre: Wrath of God is a quasi-fictionalized account of Spanish explorer Lope de Aguirre.  Lope was part of an expedition across the Amazon to find El Dorado (the city of gold).  On the path down the Amazon, he killed two leaders of the expedition (one after the other) assumed leadership, destroyed villages and demanded that the other members of the expedition declare him Prince of Peru, Tierra Forma, and Chile.  In real life, he was shot and quartered.  In the film, he was the last man standing from the expedition, flailing in madness, lording over a pack of monkeys on a deserted river raft as it crept down the Amazon.  The film was a bit more poetic in that regard.

Herzog’s Aguirre (played by Klaus Kinski) is a textbook portrayal of ambition, narcissism, and sociopathy. Kinski’s mechanisations and visual ticks as his character leads his "team" on the mother of all death marches are incredible to watch, and learn from.  If anyone around you acts like Aguirre, run.

Sun Ra: A Joyful Noise on the other hand, is a documentary about a mid-20th century African-American jazz musician (Herman Poole Blount a.k.a. ‘Sun Ra’) who claimed that he was from Saturn, recruited one of the most promising saxophonists of his generation (John Gilmore) and  started a band called ‘Sun Ra Arkeststra.’  The Arkestra dressed up in space aged costumes, created their own rituals and avant-garde music and toured the US and Europe for years.

I'm not going to say much more about Aguirre: Wrath of God right now.  Instead, I'm going to leap into the Sun Ra film. Throughout it, we are led into the world of the band.  We hear Sun Ra philosophize about the world - how it is made of myth and how his myths are as good as anyone else's. Through it all, you notice how much the band is like a cult and you wonder why all of these people fell out of society to go along with something this goofy.  But on the other hand, the film is oddly mesmerizing.  You get the feeling that the band members really didn’t believe that he was from Saturn – that they gave up their normal lives to join this touring band so that they could play, not just in the musical sense but in the broader sense as well.

Everyone knows that cults can be dangerous, recent history is full of examples, but what about a cult that plays on the edge of "play"? Where everyone knows with a wink and smile that they are just playing? I can imagine it being a rather liberating experience.  In the film Sun Ra winks at us. It's all a game, but one he decided to play for fun. It's worth thinking about how much playing we do in real life, really - play which isn’t much fun because the nod and wink aren't there. 

Chef/author Anthony Bourdain once wrote that having a restaurant kitchen is like running your own pirate ship.  If you see work that way, how can it be anything other than interesting?

For years I’ve looked at software development teams and asked myself what makes the good ones tick.  It's hard to single out answers because teams can be good in many different ways.  One thing that I come back to often, though, is the sense that a good team has an inner life.  People joke with each other.  The feel somewhat protected against the other parts of the organization, and they have people who make it fun.  That last bit is an act of leadership and often it comes about because of a strong leader, someone who defines the mythos of the team – who we are, what makes us different?  The person doing this doesn't have to be a manager, but I have seen managers do it.  People who adopt this role instinctually look for the minor dramas that bind people together and foster them.  It's fun to see it when it happens.

There’s a vogue now for self-organizing teams in software development.  That works, to a degree, but as often as not, there’s a certain listlessness to many of them, a sense within them that everyone is always waiting for someone else to step up. Ultimately, I think leadership is an individual action, and it can be an act of play that facilitates good work.  I don’t suggest you jokingly tell your coworkers you’re from Saturn, but there are many actions south of that which might help.

Pessimistic Personal Finance: The Three Fuses

One of the nice things about blogging is that you get feedback.  You take an idea and you throw it out there.  If there’s a better way of seeing things, someone is bound to tell you.

With that introduction, I turn to the topic of the day: pessimistic personal finance.  These are scary times.  We’ve all seen the stock market plunge, but really the stock market is the least of our worries.  Unless credit frees up, we’re going to see economic disaster over the course of the next year.  We already know it’s going to be bad, but no one knows how bad.

I live by the motto that pessimists are rarely disappointed and often pleasantly surprised, so with that in mind, I looked at our family finances and calculated three metrics that I call fuses.  What is a fuse?  A fuse is a duration of time until  a bad thing.

The first fuse is what I call the normal fuse.  Assume some set of catastrophes knocks out all of your income.  How long can you and your family live on what you have now, without changing your spending at all?  You can calculate this fuse by taking your cash (pretty much anything liquid and not at the mercy of the market) and dividing it by your monthly spending.  The number you get is a duration in months.  That’s your normal fuse.

The second fuse is the ‘bunker’ fuse. Assume the same set of catastrophes and calculate your fuse, but this time divide your cash by your monthly spending in ‘hunker down’ mode.  In ‘hunker down’ mode, you’re spending only what you need to for rent/mortgage, insurance, and groceries.  You jettison all non-essential spending.  Again, don’t factor in things that can evaporate in the stock market.

The third fuse is the big one: the Armaggedon fuse.  Take the total of all of your assets, your retirement, your house, your investments, and any money on hand, and divide that by your essential monthly expenses.   Then, factor in the inflation rate, and you get your fuse. 

There you go.  Even if you aren’t a pessimist, I think these are good numbers to know.  If you know them, you can rest at night knowing exactly how you’d fare during a personal tragedy, an economic meltdown, or even a zombie attack.  Although for the latter two scenarios, guns and canned soups might be better than cash.

And, now a question: I can’t be the first one to have thought of these fuses.  Are these concepts floating around under different names?