Unit Tests are Like Antibiotics

Tuesday, 28 May 2013 08:14

What do unit tests have in common with antibiotics? Well, both are fantastic at killing off a whole class of bugs, certainly, but there is another thing they have in common: they are overused.

Don't get me wrong: I love writing unit tests, and I think they are actually essential to the success of a project. Only with a suite of unit tests which can be used for regression testing can developers change code with confidence. And, while a high test coverage level does not indicate that production code is really being tested properly, a lack of test coverage indicates it is not really being tested at all.

Another reason to love unit tests is that they simply make you look better. How so? Well, like many developers, I am prone to thinking faster than I can write code and introducing silly bugs into new modules. I'm talking about obvious things like off by one errors or null pointers. Having these errors happen after you commit a new module to the SCM is just embarrassing. By having tests in place and making sure they pass before you commit new code eliminates a whole class of bugs which means your code is automatically a lot better than it would have been without them.

No, the reason why they are overused is that, firstly, it is not possible to check how well the unit tests actually test the code except by inspecting them. Code inspections of test code? I hear you ask. Well, actually, yes. Of course, I don't mean you have to review test code as rigorously as you would review production code, but you really do want to know that the tests assert that certain pre- and postconditions are met and do not simply run through the code just to make the code coverage statistics look better and please the Code Sheriff.

A good indication that your tests are not very much use is when you change production code in multiple ways but no tests ever seem to fail. If this is happening too often, alarm bells need to ring. Or, of course, if you have a module with a very high level of code coverage which is still very buggy. How to judge this is of course a matter for the development team, but everyone has worked on projects where certain code modules are notorious for producing new defects.

There is another problem with unit testing. Pure unit tests really just test a certain module and use mocks or stubs to fake the behaviour of collaborating modules, especially those lower down the architectural food chain. These tests are essential for testing difficult-to-reach parts of an application's business logic, for example, but they really just tell you how the module under test works in isolation. This is only part of the story. In addition to (pure) unit tests, you need to add integration tests.

Integration tests are typically heavyweight tests designed to exercise the whole system (or as much of it as possible) with either a real or nearly-real configuration and backend systems. Only when such tests pass can you know that your system holds together. Of course, integration tests require a lot more infrastructure than pure unit tests, and typically setup and tear down is hugely expensive in terms of runtime. For this reason, you may only choose to run them every hour or every day during the nightly build, but ideally developers should run them before any checkin if they want to stay on the safe side and not end up in the Build Breakers' Hall of Shame.