I hate long methods. Not just in the academic "Oh, I've read Refactoring and long methods are kind of ugly" sense, but rather in a deep visceral sense. I see more than my share of them when I help people with legacy code issues. It's like stepping into a new yet familiar swamp every single time.
Scroll, scroll, scroll. "Oh I'm in that method." Scroll, scroll, scroll. "Oh, that's happening in this method too?"
You get tired of it. You start to search desperately for answers.
A friend of mine, Alan Francis, had an interesting idea a while back. He thought it would be nice to have some language enforcement for method size. Imagine a language which only allowed methods with, say, three statements? Wouldn't it all be better then? Sadly, I don't think it would. People would resort to something I once saw at a company I visited. The edict came down: methods could only be X lines long. The results that policy produced were predictable. The code was littered with artificial sub-methods. Imagine a method named processTransactions() calling methods named processTransactions1(), processTransactions2(), and processTransactions3(). It was that bad.
It's easy to imagine that this is just ignorance; that in right and capable hands all methods would be well named and less than ten lines long, but when was the last time you saw a code base like that? Does it ever happen?
A few years ago, I read an interesting post on a mailing list. Someone mentioned that method size in projects is a long tail distribution.
Before reading that, I hadn't given it much thought. I'd assumed that, in a typical code base, the vast majority of methods might be about 20 lines or so, with an equal number of smaller and larger methods. The normal distribution holds in so many other areas, I'd just assumed that it held for method size. But, it turns out that it doesn't.
I saw this for myself when I produced a frequency histogram of method size for a Python project I was working on. It looked like a power-law distribution. I had dozens of one line methods, fewer two and three line methods, and the curve bottomed out at a couple of 20 line methods. As I looked at these 20 line monstrosities (indulge me), it did seem that each of them were long for a reason. Sure, I could have broken them down but in each case it would've been worse. In some cases, I just couldn't name the sub-parts well enough. In other cases, the logic was just clearer in one piece. But, regardless, I knew that if I did break up each of these methods I wouldn't have a distribution that looked substantially different. It was as if was going to whack a rock repeatedly with a hammer. I'd still end up with a couple of big chunks outnumbered by smaller chunks.
Statistics has always fascinated and frustrated me. When I was in school it was never enough for me to know what distribution applied in a problem, I always wanted to know why. What physical process was at work? It's something that the textbooks never seem to address. And, I've been having the same frustration with the method size. Fortunately, there is some speculation about the processes that lead to power law distributions.
Power law distributions seem to arise in processes where there is preferential attachment. The "hub and spoke" routing system that airlines use is a good example. If you are an airline running flights from "backwoods country airport", you are usually better off if you route flights through a larger airport rather than direct to a neighboring little airport. The flights you schedule to the larger airport can carry people who want to travel locally and people who want to travel further along in the network. You get economies of scale. If you graph a frequency histogram of airports by number of routes, you'll get the familiar curve.
It turns out that there has been quite a bit of power-law research recently. People have noticed that power law distributions or distributions that look like power law are all around us, in biological systems, social systems etc. A while back, Gareth Baker, James Noble and a few associates published a paper (PDF) on a project of theirs called the 'Shape of Java'. They'd assembled a large corpus of closed and open source Java code and ran an incredible number of measurements upon it, looking at distributions for method size, class size, number of implemented interfaces, etc. One of the observations that they made was that program qualities that programmers tend to be aware of tend to not quite fit the power law mold. There's always some limit that programmers won't go past. I can confirm, at least, that my years of looking at legacy code, I've never seen a one hundred thousand line method, for instance. But, that long tail effect does seem to be there. Why?
One theory that I have is that there is a preferential attachment process that we adhere to, whether we are conscious of it or not, when we are coding. When we add a method we have to call it from some other method. Do we commonly pick bigger methods to call from or smaller ones? On balance, I think we choose the bigger ones because they are closer to our original intention, our goal when we are programming. If we see work that has to happen before or after some other work, it is easy to go to an existing method and add the call there. Sure, we refactor and break our bigger methods down, but that doesn't seem to change the shape of the distribution, it may just jiggle the coefficients. It's like hitting that rock with a hammer again.
I still hate long methods, but I've given up hope of getting rid of all of them. It seems that software adheres to some natural laws. A good code base consists of a large number of small methods and dwindling number of larger ones. And, if that's the case, we shouldn't strive for the absence of long methods, we should strive to make them rare.
Maybe those last few long methods in each code base can serve a purpose - they can remind us of what we'd like to avoid.
Comments (17)
I think the natural inclination is to have methods that fit in your editing window. For many of us, that is around 25 lines.
I may be drawing this realization from Code Complete, I forget. But it remains true. I want to be able to look at a whole method at once. If it's too long, I (often unconsciously) look for ways to factor bits out.
Posted by Jim Kiley | January 24, 2008 9:03 PM
I agree with Jim. Methods do appear to be best understood when they fit our editors. In my case that would be Visual Studio, programming at 10pts Courier New. I counted and 37 lines of code in my editor window. Being a nerd, I quickly enlarged the window so that it fitted 42 lines - but 25 would same more natural.
A nice example though. I Recently came across this ASP.NET web page (written in VB.Net to make things worse) which contained 4912 lines of code - in a single class. This class had near a 100 methods. A total disaster.
Posted by Sander Hoogendoorn | January 25, 2008 2:39 AM
Jim, Sander: I agree that methods are better when they fit on a screen, and that feels more natural, but in the end, what happens? I bet that if you did a histogram on of method lengths on your projects, you'd find something that looks kind of like a power law distribution.. a long tail going down to a small number of big methods.
Sometimes I wonder whether we can learn something about the health of a code base, by looking at that distribution.
Posted by Michael Feathers | January 25, 2008 2:58 AM
Michael, as you know, I have also been looking to the distribution of cyclomatic complexity in Java code, and also found there what looks a lot like a power-law (it's notoriously difficult to confirm that one really has one). Bil Kleb has done the same analysis of some of the FORTRAN his group uses and found the same shape (you don't want to know the largest complexities seen in legacy numerical code :)
One point to note about this sort of quality in an enemble is that the distribution is scale-free. In practice this means that, yes, there will be mostly smaller methods in your code and a very few methods much (much!) larger than the many short ones--but that doesn't mean that the longest long method needs to be particulalry long in absolute terms. Indeed, what I've found is that in code that was developed TDD the most complex methods are less complex than the most complex methods in code not written TDD. Using TDD seems to make the distribution "steeper", so that the long tail is thinner.
I also find myself wanting to know why a distribution should apply. Power-law distributions can be unpacked in a certain sense, which might help. In seeking to explain Benford's Law (another power-law phenomenon) it has been found that if one builds an variate by picking successive random values from successive randomly determined distributions (of any sort, IIRC), then that variate will have a power-law distribution. Not sure that helps terribly much, but it is food for thought.
Posted by Keith | January 25, 2008 5:37 AM
1: In very Object Oriented environments like Smalltalk, you end up with mature frameworks where all methods are small, all classes have few methods, and when you try to trace code, you find that all decisions are made by the presence/absence of methods and rerouting of calls to other objects. You get programming by fragmentation of responsibility - which is fine if you understand the architecture of the classes in the framework.
2: The Method Object pattern as described by Kent Beck was a real eye opener when it came to the handling of long methods. That refactoring is easier in Smalltalk, but still doable in Java and presumably in Ruby/Python, C#, C++ etc.
(Basically you turn the method into a "perform" method in an new class. This class has variables pointing to everything the old method required: internal vars, instance vars, static vars, parameters. The "perform" method may then be sliced, diced, subclassed, abstracted etc until it looks like documentation)
Posted by Arne D H | January 25, 2008 6:28 AM
Arne: Thanks. Yes, I've seen some Smalltalk frameworks and I do really go for the finely-grained factoring approach. It does strike me, though, that still there always seem to be a few scraggly long methods. Have you ever seen any frameworks that don't seem to fall into this power-law mold?
Re Method Object, yes, it's one of my favorite last resort patterns. It's particularly useful when you want to start making progress again in C++ code bases with massive compile times. You can start to add new headers without touching old ones, and that can be very convenient.
Posted by Michael Feathers | January 25, 2008 6:40 AM
Keith: Thanks, I'll look further into it. The random of randoms process that you describe sounds somewhat close to the physical metaphor that I keep coming back to: bash a rock and its pieces randomly and you end up with a few larger ones and many small ones. Still looking for ways to understand this.
Posted by Michael Feathers | January 25, 2008 6:44 AM
When I write applications for a business purpose the first cut on the code will contain long methods. This is natural for me until you start to get into the "flow" or "Mojo" of the logic your trying to invision. Most of the time I will go back and break them up into smaller chunks.
Sometimes due to constrains with business needs it stays "Long".
I've been writing code for a long time. Back in the days it was "Write clean/small efficient code". Now nobody cares!
Posted by David Camacho | January 25, 2008 8:23 AM
Length is not exactly the same as complexity, of course. With block-scoping, like C++'s {}, a function can be a sequence of almost isolated chunks. That case is easy to handle, and often better than jumping in and out of sub-functions (which obscures what might be an essentially sequential structure).
Posted by Harrison Ainsworth | January 25, 2008 9:33 AM
OOP is all about real life programming. There are people (read "reasons") who like huge halls in their house and there are people who would prefer to have 4 smaller rooms then a big hall.
I had a colleague who wouldn't like files more than x number of lines.
Compared to languages like C, fortunately with OO languages, we have useful features likes classes, exceptions, inheritance, etc which essentially breaks the code into smaller pieces. When they get transformed into assembly code, they ultimately take the same form.
In my not-so-humble opinion homo-sapiens should keep distance from suggesting hard-rules to break down the function.There are constructs like the "if" and the "case" statements which are heavenly signs to programmers to re-factor the code.
Despite all these, we do many a times find functions which are gigantic. Well, there are deserts and oceans in the world. Shouldn't God have re-factored them?
Posted by Shuva | January 25, 2008 12:37 PM
Fascinating post. I want to run some shell scripts on our code base now.
Posted by Ivan | January 25, 2008 1:16 PM
I think you gave us the reason why we still see long methods even though we try our best to avoid them. It's simply because a long method is sometimes the perfect encapsulation. Size in itself doesn't matter, how well the code communicate intention does.
I wrote a post on this subject some time back: http://www.hans-eric.com/2007/10/30/abstraction-is-the-name-of-the-game/
Posted by Hans-Eric Grönlund | January 25, 2008 2:18 PM
Editor viewing size is important. I think it also enforces a length maximum on a single line of code with the tendency of programmers to break that line into multiple lines if it is too long. It struck me that if you had a situation where that happened a lot, you would get longer methods because each statement would take up 3 lines of code. Thus long and descriptive names may be extending the length of the code. The more complex your system the more complex your naming structure has to be to differentiate the various pieces.
I also feel that the length of a method is determined often by grouping together like actions that you never want to call apart. For example, let's say you want to read into an object all the entries your user has made into a form that has 40-50 fields [note: yes I realize that a 40-50 field form is not ideal, but sometimes that's just how it goes with the requirements of the program.] You aren't going to only read in the first 10 of those fields by themselves; you want all the values read in. And, if each of those reads takes a single line of code (hopefully) then you get methods that appear longer, but are easy to understand and logically seem to flow together. Situations like this are probably contributing to the tail, even though most programmers would rather have shorter methods.
Breaking a method like that mentioned in the previous paragraph into a "master" method and sub methods seems a lot like premature optimization. You are effectively creating more functionality (the ability to read in only some of the values) than is necessary for the problem to be solved.
Posted by Joshua Volz | January 25, 2008 5:05 PM
Long methods are are to understand, hard to work on. That's why I'm also trying to limit methods to one screen.
Of course, cutting a method in a stupid way (like your example processTransactionN) and doesn't give any value. When cutting into smaller methods you need to make sure that these methods are compact and independent from each other.
Of course that's not always trivial, and you may need to rethink your design to have it broken in simple blocks. But when you succeed, your code is much more readable and easier to maintain.
Posted by erwan | January 25, 2008 5:23 PM
"20 line monstrosities"? How I envy you. In the legacy code I'm currently suffering through I have to deal with 1000-line methods with seven levels of nesting. It's enough to make a grown man cry.
Posted by Dan | January 25, 2008 5:30 PM
You might like these two papers by Les Hatton:
Posted by Greg Buchholz | January 26, 2008 1:07 PM
Greg: Thanks very much for the links. I've been interested in that topic for a while. I remember a note in McConnell's Code Completethat made reference to a similar study wrt module size, but I wasn't able to follow it up.
I've glanced at both the papers (will read in depth later) but the one thing I find very frustrating is the lack of a clear code notion of a component/module. I appreciate that they are looking at behavior that is seems to be true across several code organization schemes, but to me 200 line packages, 200 line classes and 200 line methods are completely different animals.
Another thing I often wonder about is whether effects like these could be mitigated with tools which give you an inlined view of a method, showing all code called at any level.
Again, thanks for the links. I look forward to reading them in depth.
Posted by Michael Feathers | January 26, 2008 2:23 PM