Code coverage
A powerful tool that will help you write better quality code is measuring code coverage. This post will explain the basics how I use this in my foundation library project (https://github.com/rampantpixels/foundation_lib) written in C, but the principles apply to other languages and toolchains as well. All code I publish is released under public domain and free to use for any purpose.
Code coverage measures how many times each line of code is executed while running your test suite. By looking at the lines that have not been executed you can find potential errors in your code or learn which test cases that needs to be improved in order to test all code paths. I use the free service at https://codecov.io to visualize the results.
For an example, look at https://goo.gl/TtNJl1 to see the coverage results of a module in the library. The lines that have been executed at least once are marked in green, the lines that have never been executed are marked in red, and the lines that are not generating any instructions are unmarked.
Coverage is pretty good, but in the function profile_end_block there is a while loop marked with the comment "Walk sibling list backwards" that is never executed.
Is this dead code that can be removed (as in, will this case never happen due to logic in other parts of the code) or are the test cases lacking the proper tests? If the latter, we have no idea if the code actually works or not, since it has not been executed. Finding these untested parts of the code is that goal of code coverage analysis.
If you have an already existing code base it might seem a daunting task to start using code coverage as you will most likely find many parts of the code which is not covered. But once you find an fix the first bug in one of these dark areas of the code it will be worth it! It's gamification at its best, you will work hard to reach that elusive 100% coverage, and your code quality will rise quickly.
In order to generate the code coverage measurements I use the built-in functionality of the clang compiler by passing the "--coverage" argument to compiler and linker. When the code is compiled a .gcno metadata file is generated by the compiler for each object file, and once the final test case binary is executed it will generate a corresponding .gcda coverage data file.
These files can then be parsed together with the corresponding source file to find the execution count per line per file using the llvm-cov tool. This tool can output a gcov text file with the wanted data (similar/compatible to what the gcc toolchain also uses).
I wrote two python scripts to do the job invoking llvm-cov, parsing these gcov files and upload the results to codecov.io. Read a gcov file at https://goo.gl/bH4AUm, invoke llvm-cov and build a report for codecov.io at https://goo.gl/bH4AUm.
As a closing remark also note that code coverage can be used with profile guided optimization by feeding the coverage data from your final executable back to the compiler. But this is a topic for another post.
Code coverage measures how many times each line of code is executed while running your test suite. By looking at the lines that have not been executed you can find potential errors in your code or learn which test cases that needs to be improved in order to test all code paths. I use the free service at https://codecov.io to visualize the results.
For an example, look at https://goo.gl/TtNJl1 to see the coverage results of a module in the library. The lines that have been executed at least once are marked in green, the lines that have never been executed are marked in red, and the lines that are not generating any instructions are unmarked.
Coverage is pretty good, but in the function profile_end_block there is a while loop marked with the comment "Walk sibling list backwards" that is never executed.
Is this dead code that can be removed (as in, will this case never happen due to logic in other parts of the code) or are the test cases lacking the proper tests? If the latter, we have no idea if the code actually works or not, since it has not been executed. Finding these untested parts of the code is that goal of code coverage analysis.
If you have an already existing code base it might seem a daunting task to start using code coverage as you will most likely find many parts of the code which is not covered. But once you find an fix the first bug in one of these dark areas of the code it will be worth it! It's gamification at its best, you will work hard to reach that elusive 100% coverage, and your code quality will rise quickly.
In order to generate the code coverage measurements I use the built-in functionality of the clang compiler by passing the "--coverage" argument to compiler and linker. When the code is compiled a .gcno metadata file is generated by the compiler for each object file, and once the final test case binary is executed it will generate a corresponding .gcda coverage data file.
These files can then be parsed together with the corresponding source file to find the execution count per line per file using the llvm-cov tool. This tool can output a gcov text file with the wanted data (similar/compatible to what the gcc toolchain also uses).
I wrote two python scripts to do the job invoking llvm-cov, parsing these gcov files and upload the results to codecov.io. Read a gcov file at https://goo.gl/bH4AUm, invoke llvm-cov and build a report for codecov.io at https://goo.gl/bH4AUm.
As a closing remark also note that code coverage can be used with profile guided optimization by feeding the coverage data from your final executable back to the compiler. But this is a topic for another post.
posted by Mattias at
14:47
0 Comments
