Thoughts on Unit Testing

Back when I was a new engineer out of college, I was excited about automated unit testing. It makes sense to use a computer, which is good at performing repeated operations quickly, for testing that the software works or that it still works. I still hold out hope for automated unit testing, but it is not the miracle it feels like it could be. It is good for catching some bugs and hopeless for others.

Extremes

In my career I've met engineers who thought unit testing was a waste of time. They thought it took too long to write, didn't catch all that much, and was busywork. This is a bad dismissive attitude. You must try a practice and give it a chance to see where it works best. As developers we have powerful computers at our disposal, it isn't a good use of resources to manually poke through code for every release to test it. Computers are good at this sort of thing.

I've also met engineers who were at the other extreme. All code must have unit tests, target 100% code coverage, or maybe write your unit tests first. This also is crazy. Not all code equally benefits from unit tests. You are better off concentrating your efforts on unit testing code where you get the most value than to try to do everything. I've never worked on a project where everything was specified ahead of time to be able to write tests before code. Engineering is generally more fluid than that. If you are working on a strict waterfall maybe this is possible, but that sounds like a terrible project to be on.

What Works Well

Algorithms and data structures often benefit from unit tests. It is great when working with a reversible algorithm as it is a simple test write to apply and reverse and can catch a great number of problems. Some algorithms are very complex to solve a problem but also very simple to check. Algorithms and data structures tend to be more isolated code. They don't generally depend on your database server, some web service, or operating system calls. So, there isn't much work needed to test them in isolation.

Really any problem that you can throw a ton of data at it and can validate its behavior with the data works well for a unit test. From a single unit test, you can try a large number of conditions.

For one project I had to design an algorithm that would significantly change the data storage format of what was essentially simplified program code. Unit testing worked well on that. Give the algorithm different scenarios and make sure it handles them ok. Apply the transformation and reverse it and verify that it reverses ok. This found a ton of bugs out of a small list of unit tests.

What Doesn't Work Well

Sometimes the kinds of problems that introduce the largest number of bugs also are the hardest to unit test. Integration with another software system or with hardware often behaves vastly different than you think it will or has been specified to behave. These kinds of bugs tend to show up when code is moved to production, and you probably won't have logging around the unexpected integration behavior because it is unexpected.

What is or isn't integration testing vs unit testing isn't something everyone agrees on. But a common technique for isolating modules is to mock out the integration points. The same assumptions you made when writing your code will also be put into the mocked objects and unit tests. You are then testing that your code handles the way you think the integration behaves and not how it behaves. This can catch some issues, but the most difficult bugs that it would be great to catch slip through.

These kinds of bugs of missing information, false assumptions or not realizing something could behave a certain way are quite expensive. Unit tests may catch them by accident or when working on unit testing you suddenly realize something could happen. But in general, they aren't great for this.

Don't Only Do Unit Testing

One bug I had to deal with that was very frustrating was in an interrupt handler for a input pin. On occasion, my microcontroller would just reset (probably watchdog or other check). I eventually traced it down to the timing of the input pin. The code I was running was a scripting language and the interpreter would fail out it the interrupt was triggered faster than it could run the script. This failed for even an empty interrupt handler.

This is an obvious case where unit testing won't catch the problem at all. Mock out an input pin? Well, the mocked code would always work. It wasn't a code logic issue, the problem happened even when the interrupt handler was empty.

This may seem obvious, there are many more QA practices than unit testing because there are all sorts of bugs in software. The best way to mitigate bugs is through multiple approaches. Don't spend so much time on code coverage and unit tests that you don't get to other forms of testing, review, or QA practices. Depending on the project you are working on, maybe the other QA practices are more valuable. Every project is different.

100% of code coverage also isn't a good target. How many possible states are you testing? Are edge cases covered? These kinds of problems are hard for your brain to think about at the same time and more likely to have bugs. If you are concerned about validating every line of code, it is better to use code reviews for that.

Use Your Debugger

I've seen an odd mentality in software developers lately that there is a view that using a debugger is lazy and you should be unit testing and using logging. This seems to go along with the increase in cloud computing where debugging a running application is either difficult or simply not possible.

Yes, write unit tests! Yes, use log files! But a debugger is a wonderful tool to find bugs! Learn your debugger well. Hopefully it has something like a REPL where you can execute code snippets while you are running your program. Avoid development tools or languages that don't have a good debugger. Shipping code because the unit tests all pass isn't far removed from shipping code because it compiled. Great, it can catch some things, now finish testing it.

Sometimes it takes looking at a scenario in a debugger for the actual cause to be apparent. Code is complex, it is hard to keep all the possibilities in your head. This is even harder when digging through logs or writing unit tests. Sometimes it really helps to see what is going on.