Code Coverage and Testing: Used the Right Way

By Teresa Hansen

What is code coverage? How can I take advantage of code coverage to improve a codebase? What should I look out for? These are all questions that should be asked when looking into code coverage.

I have been using code coverage as a tool to improve legacy testing suites as well as new features for over a year.

The Basics:

Code coverage is a way to measure the amount of code that runs given a testing suite. It will tell you how much of the code is being touched by the tests. As a result of breaking down a test suite into a single number, a lot of important information can be lost. Code coverage can show the quantity of tests but does little to show the quality of the tests.

Quality testing is arguably more important than the quantity of tests in a test suite. There are many ways to add a test that can increase your code coverage number without actually testing the functionality of the code. By doing this the code coverage may look great but the functionality of the code is not tested. The purpose of the tests should be to show that functionality changed if someone goes in to change the code. If just the code is tested, not the functionality, the test will not help in the future when you need to change the code but not the functionality.

Although, quality is very important when testing, quantity is also important. If you only have one test, that only goes through one path, you are leaving a lot of your code untested. The quality of your test may be great, but the quantity is lacking. One of the biggest benefits of code coverage is to show what part of the code is not tested. By showing what is not tested, a developer can go back and add tests to check the functionality of the untested code.

For a good testing suite, you need your tests to be of good quality and have a quantity of tests so that all functionality is covered. Code coverage does not show all of the story when it comes to a quality test suite.

Take Advantage:

While code coverage may not be the end all be all for if a test suite is good, it is important to be able to take advantage of the positives that come with code coverage. Some of the benefits of code coverage include increase mindfulness to test, ability to show what code is not tested, and a better overall confidence in the codebase.

Often testing takes a back seat when it comes to creating a new feature or changing something from the past. While there are many reasons why one is because testing can be easily forgotten and is not a priority to most developers. By adding code coverage testing can become more important and closer to the forefront of the mind when writing new code or changing the old. Having good tests can lead to a host of positive results in being able to debug as well as enter into code that you have never worked in before.

What I believe to be the best benefit for code coverage is the ability to see what code is not tested. While having 100% code coverage does not mean that all of your code is well tested, having 10% code coverage means that you have do not have a well-tested codebase. A developer can use the code coverage to help them find places that have no tests so that they can add them. Knowing what code is not tested can help future developers improve the test suite by adding tests that should be there.

Confidence in a codebase is difficult to achieve when you can only rely on human testing. Most code has many paths that do different things based on given parameters or situations. When relying on human testing trying to change the existing code to be better written or more efficient can bring about questions such as, how we know everything is working correctly and what is the potential for failure with the change. By having a good testing suite that covers all of the different features and scenarios can help developers in having confidence that their changes did not cause problems with the system.

Take Heed:

The benefits of code coverage can be overtaken by negatives if taken too far. While having low code coverage is bad having high code coverage is not necessarily good. There are many ways that someone can artificially increase the code coverage for a project without having well tested code. Without the right mindset, code coverage amount can do nothing but increase the amount of work on a developer with none of the advantages.

Many developers see adding tests as extra work. Another task that they have to do to appease those in charge, with little effect on the functionality of their project. Writing tests can take time away from other work that needs to get done and the tests may be harder to write than the code itself. With this mindset, code coverage can be seen as another hurdle they need to jump through before their code can be used in production. This deters programmers from writing good comprehensive tests for their code. Before starting with code coverage, developers need to be on board with wanting to write good tests.

Code coverage is not a good metric in the ways that quality tests are not needed to make a high code coverage. One way to get high code coverage without writing quality tests is to write a test with no assert statement. Without an assert statement, the program calculating the code coverage will record code being tested that tests for nothing. These tests don’t test functionality and at best will test that no unexpected exceptions are thrown. A second way is to mock incorrectly. By incorrectly mocking you can end up skipping the functionality for parts of the code you are trying to test. Third, testing only the easy code and avoiding writing tests for code that is harder to test. The more complex the code the more that code should be tested, but if all of the simple, getters and setters, are tested the code coverage number may be high with a little testing of functionality.

When talking about code coverage, a lot of people ask what percent coverage is needed. We need to remember that even if your code is 100% in code coverage, this does not mean that it is 100% well tested. A change of how people view testing is the best way to get a better testing suite than adding a requirement based on a metric.

Like most tools, code coverage has a range of what it can do, and with the right application can be helpful for the developers work. Though, also like most tools, if used improperly can be a difficult tool to wield and bring up more issues than originally known. With this info I hope that you have gained a better understanding of the benefits as well as the downfalls that come with code coverage.