Beautiful Code

Thanks for this post!
I've been following this blog silently for a while now, and some articles are, I have to admit, a bit too difficult for me to understand, but those that I could understand really helped me gain deeper insights, and this post is one of them.
Just thought I'd express my appreciation.
Keep them coming!

I found it indeed slightly disturbing that Programming Erlang by Joe Armstrong (Pragmatic Bookshelf, 2007) doesn't have anything on testing. This absence is particularly noteworthy given the publishers' otherwise strong focus on testing.

I don't intend to bash the Erlang as a language, but it is quite clear that writing tests for systems implemented in Erlang poses some new challenges over common OOPLs and other FPLs as it almost by necessity involves concurrency.

The first thought I had while reading your post was that tt wouldn't be hard to use a preprocessor of your choice (like say M4) with any of the functional languages you mentioned, except maybe Erlang if you were wanting to edit and compile automatically. Even that shouldn't be too hard to rig up though. Anyway, then you would at least be able to use the macro approach from the C world.

Actually, Haskell makes things even better here, because there is a huge barrier to writing "bad things which could happen during testing" in the first place.

Indeed, if the module/typeclass doesn't have an "IO" in the type, then these bad situations simply can't happen.

Yes, having well placed seams will simplify your testing, but I think you perhaps presume too much when you claim they will be there merely by the grace of dynamic dispatch. What about the case where your code isn't factored into methods that align with the needs of your tests? If your badMethod() was instead just inline code in another method you would lose the seam you depend on:

public class X {

    public void method() {

       // important logic

       ...

       // bad stuff

       ...

       // more important stuff

    }

}

I think your thoughts about "seams" are really just getting at the underlying issue of design for testability. It should be clear that in any idiom, having the "seams" of your design mesh well with the needs of testing it will greatly simplify the testing. So, the question becomes one of whether OO, or any other particular idiom, inherently tends to place your seams where you need them without thought from the programmer. I would say from my experience that is not the case, and you don't really present evidence to support such a claim.
Any language will have its sources of "seams" that let you instrument for testing. But whether they are dynamic methods, first class functions, module boundaries, monads, or anything else, I would ascribe their proper placement to the quality of the design and the designer far more than to the language or idiom in which they are implemented.

I saw your book at the bookstore and am increasingly glad I didn't buy it.

If I understand you correctly, you are talking about pluggability -- able to plug in various mock objects at various places so to isolate the object-under-test. Thus, we are talking about what mechanisms are provided in the lang in general to allow us to substitute portion of the logic. (I think this is called "points of variation".)

In OOP, the characterizing mechanisms are inheritance/polymorphism and delegation. Likewise in C, they are macros and function pointers, etc. In functional programming, it is high-order functions.

One may argue what if the code isn't written with structured with the proper high-order functions to provide the points of variation to plug in the mock functions. But we can argue equally, in your example, what if class X doesn't have other virtual functions one can override. The point is, regardless it's functional or OOP, the code needs to be structured to facilitate unit testing in the first place.

Am I making sense?

John, that's fair enough. You can use a preprocessor. The only language I do that in (today) is C, and even then I usually opt for function pointers or link substitution first, primarily because it involves less head scratching than C's very unhygienic macros.

The thing that really annoys me is that this testing issue is an afterthought. Very few language designers consider it.

Andrew, it is true that you often have to change your code to make it testable, but there are safe ways of doing it.

If you are working with a tool which performs adequate analysis before extract method, you can extract safely. If you don't, you can use sensing variables to perform the extraction. I often do this in C and C++ because of the "tool trust" factor.

It's true that seams don't always align with the grain of the testing that you want to do, but it definitely is easier in languages with good seam support both because of the seams that are there accidentally and because it is easier to open up new ones. I'm afraid I don't have any evidence on that, but I do have experience. I literally do little else these days other than travel around the world and help people get code under test, and the seams that are already there are very handy. Often you have a choice of levels when you approach testing problems. More seams equals == more possibilities. I'm not one to plug my book because I think that's tacky. I should get some of that material online.

I agree that "design for testability" is the real issue, but I also know (sadly) that no matter what we do, there will be systems where it doesn't happen. For that reason, recoverability is an issue that I care about a great deal.

John, you said "One may argue what if the code isn't written with structured with the proper high-order functions to provide the points of variation to plug in the mock functions. But we can argue equally, in your example, what if class X doesn't have other virtual functions one can override. The point is, regardless it's functional or OOP, the code needs to be structured to facilitate unit testing in the first place."

True, but doing what it takes to make something testable after the fact is easier in some languages than it is in others. We can use the Java example as a foundation.

You point out that if badMethodwasn't virtual we would be out of luck. Nicely, though, in Java virtual is the default. You can make a method non-virtual by making it final, but it is easy enough to make a final method non-final when you need to (provided you own the code).

In Ruby, things are much easier. You can actually replace a "bad method" at the object level rather than the class level.

C++, on the other hand, is harder. If the member function is non-virtual, you can make it virtual, but you have to check other things. For instance, if the class has a base class, you have to check for non-virtual overrides of the same member function. Also, if there are no other virtual functions on the class, you have to consider a virtual destructor and other aspects of the class design to make sure they are consistent. In most cases you will end up changing the class's header file, which (if it is in a legacy system) often triggers a massive rebuild.

C is a bit easier, because you can play some tricks that, by the way, aren't available in some functional programming languages. There is a big difference between a function pointer and a first-class function. Function pointers are mutable. So, instead of changing a bunch of calls to pass a function down to the point where you want to replace it, you can just declare a function pointer with the same name as the function you want to replace and initialize it to a different function before the test call. The calling syntax is the same.

These language things do seem to matter when we are trying to recover.

My understanding of the functional approach is that I should be writing, er, functions that take inputs and return outputs, so programming should be the act of assembling these components which should, in turn, provide seams for testing.

I can also believe that there might not be the same style of testing in the functional community, but then there didn't used to be in the OO world. As you point out, we've all seen enormous stream-of-consciousness methods with nowhere to get in.

I'd be very interested to hear from people working on significant functional code bases to see how they do it.

Functional languages make it easy to structure your code with all that nasty stuff seperated off at the top-level and keep your core logic pure. Unless you're fighting against the functional language to write your code in a non-functional idiom, you just don't get the IoC and seperation-of-concern issues you do with imperative OO code.

And yes, Monads help a lot here too, effectively they let you write one piece of code that creates (on-the-fly) a list/stream of IO commands, and another piece of code consumes the stream and decides how to execute those commands.

The situation is pretty much the opposite to what you fear. You don't usually create those testing problems in the first place when you write functional code (if you have you should probably refactor right now). Unit testing is so much easier when your units are guaranteed truly independent! Or at worst the dependencies are clear and explicit.

Let me apologize if this seems like a functional programming bigot email, but I was honestly boggled by this article, since my experience with testing tends to lead me to the opposite conclusion from yours. That is, my impression is that the heavy use of state in conventional object-oriented programming is detrimental to testing. Since you are obviously a thoughtful and experienced maker of software, this suggests that there may be points in both direction.

I am primarily a Common Lisp programmer now, although at one time I wrote C code for a living. As a CL programmer I am not a pure functional programmer, as a Haskell programmer might be --- Common Lisp (like ML, and OCaml, which you describe as functional) is a multi-paradigm programming language.

My experience in writing tests is that the functional style can be immensely helpful in simplifying testing. When my code is well structured, then tests can typically be structured in terms of function calls and comparisons between the results of the functions and expected results. Of course that is oversimplifying --- there are issues of setup and teardown, dynamic scoping, and checking for conditions, among other things --- but in general the tests are simple to write. My experience in dealing with heavily object-oriented CL code is less happy --- writing tests is complicated by the need to build up large bodies of interconnected objects in order to set the stage for the tests, and there may be additional complications in checking that the expected side-effects have occurred.

I believe that my impression is substantiated by all the discussion of "mocks" and so forth in the OO testing literature.

As someone who uses a hybrid of functional and OO programming, I'm also a bit confused by all the discussion of the need for injecting "mock functions" for testing. When testing functional aspects of my code, I just don't find myself encountering this problem. I don't wish to use mock functions; I simply test the real functions. Since they are generally not side-effecting, there is no need to use a mock function.

In those cases where there are side-effects, and one might want the equivalent of a mock function, surely the behaviors one wants would be encapsulated in separate program entities to permit swapping in and out? What functional programming system would make this hard? In a purely functional framework such as Haskell, any of these problems would have to be encapsulated in a Monad; there's simply no other way to handle side-effects, so one would simply need some "mock Monad." I'm not that much of an Ocaml programmer, but I found it hard to follow your claims about that language. Indeed, I'm inclined to think it's precisely the non-functional, side-effecting and object-oriented aspects of Ocaml that are the ones that create the problems to which you refer, or perhaps more specifically the combinations of Ocaml's type discipline and its object oriented features.

Common Lisp, of course, has OO constructs, but as I said above, I find their use to be more a hindrance than a help in testing. The OO-related constructs of Common Lisp that seem to me to be most helpful are the ones that are more commonly referred to as "Aspect Oriented." In particular, I find the ability to dispatch on multiple arguments very helpful (to abuse your terminology, this might be called "even later dispatch"), and the ability to create around methods is very handy for test fixtures. But neither of these is offered by more conventional OO frameworks.

As an outsider to the programming community from which you come, it seems to me almost that you are complaining that functional programming frameworks make it hard to test programs with side-effects. But of course, those are precisely the kinds of programs that are either avoided (by constructs like the Monad) or that are admitted by non-functional parts of hybrid languages like ML, Ocaml, and Common Lisp.

Part of the reason I don't get your argument is that I'm writing different sorts of program from ours, using different tools. This is certainly substantially my fault. But surely this is also a problem with your argument --- I think there must be some assumptions there that are so obvious to you that you felt that they didn't need stating.

Even though the seam may not be present in a FP function

method = do
    ...
    badMethod
    ...
    return x

it seems straight forward to create a seam

method = method' badMethod

method' f = do

    ...

    f

    ...

    return x

Now all testing can be carried out on method' without those side effects in badMethod that are undesirable for testing by substituting in a dummy or mock 'f'