Testing error handling in C
Writing tests is an useful tool in any programmers toolbox. For C programmers it tends to be slightly more cumbersome compared to other more high level languages. Especially if you are like me and fall in the "handmade" category (see the Handmade Manifesto) and don't just want to use any and every library and framework you can get your hands on.
Even so, I tend to write test suites and measure code coverage (see my other article) to make sure I get as close as possible to execute all potential code paths in my projects. In my foundation codebase I recently hit the wall where I was basically testing everything I could except for one thing. Code handling standard library or system call failures.
In some cases you can provoke a failure in order to test your error handling by sending in invalid values. But this might not always be possible, like if your function makes the standard library call with parameters independent of the arguments of the function you are testing. No matter what your test passes to the function, the standard library call will succeed. It will only fail due to external factors which are impossible or at least very difficult for your test to control. One simple example, using getcwd on POSIX-compliant system to get the path of the current working directory for the process. In my codebase the call looks something like
In this snippet, buffer is a local variable which is independent from whatever arguments the enclosing function has. No matter what my tests sends to the function, the call will most likely succeed. In order to make it fail, the test has to either make sure the test process does not have read access to a part of the working directory path, or has been removed. This could be hard to reproduce, or at least take a lot of custom wrapping code to get right without affecting other tests or be guaranteed to work on other systems.
However, those two cases might very well occur in real use of the code, and I would like to make sure the error handling is working as expected. Ideally the test should be able to provoke the error in a way that can be used for any standard library or system call, and not rely on writing lots of code just to test a single point of failure.
To do this, I came up with a simple solution. I create a mock library which is linked last in the list of libraries when linking the final test executable. The mock library adds a mock/proxy implementation of the standard library function, and use available platform support calls (like dlsym or GetProcAddress) to locate the real entry point of the standard library implementation of the function and depending on a toggle either call the real implementation, or produce the wanted error.
The test code then only has to call a function in the mock library to toggle the wanted error, execute the test, and then toggle off the mock implementation causing the real implementation to be executed in the next call. This can easily be reproduced for other standard library calls in an identical fashion (but with separate mock functions). For getcwd the mock wrappers could look like this:
A couple of caveats:
Even so, I tend to write test suites and measure code coverage (see my other article) to make sure I get as close as possible to execute all potential code paths in my projects. In my foundation codebase I recently hit the wall where I was basically testing everything I could except for one thing. Code handling standard library or system call failures.
In some cases you can provoke a failure in order to test your error handling by sending in invalid values. But this might not always be possible, like if your function makes the standard library call with parameters independent of the arguments of the function you are testing. No matter what your test passes to the function, the standard library call will succeed. It will only fail due to external factors which are impossible or at least very difficult for your test to control. One simple example, using getcwd on POSIX-compliant system to get the path of the current working directory for the process. In my codebase the call looks something like
char* function_we_want_to_test(int arg) { //Some code goes here char buffer[MAX_PATH_LEN]; if (!getcwd(buffer, sizeof(buffer)) { //Error handling code goes here return 0; } //Success code path... }
However, those two cases might very well occur in real use of the code, and I would like to make sure the error handling is working as expected. Ideally the test should be able to provoke the error in a way that can be used for any standard library or system call, and not rely on writing lots of code just to test a single point of failure.
To do this, I came up with a simple solution. I create a mock library which is linked last in the list of libraries when linking the final test executable. The mock library adds a mock/proxy implementation of the standard library function, and use available platform support calls (like dlsym or GetProcAddress) to locate the real entry point of the standard library implementation of the function and depending on a toggle either call the real implementation, or produce the wanted error.
The test code then only has to call a function in the mock library to toggle the wanted error, execute the test, and then toggle off the mock implementation causing the real implementation to be executed in the next call. This can easily be reproduced for other standard library calls in an identical fashion (but with separate mock functions). For getcwd the mock wrappers could look like this:
The test code could then be:static bool getcwd_is_mocked; static char* getcwd_ret; static int getcwd_err; void getcwd_mock(char* ret, int err) { getcwd_is_mocked = true; getcwd_ret = ret; getcwd_err = err; } void getcwd_unmock(void) { getcwd_is_mocked = false; } char* getcwd(char* arg0, size_t arg1) { if (getcwd_is_mocked) { errno = getcwd_err; return getcwd_ret; } static void* real_getcwd = 0; if (!real_getcwd) real_getcwd = dlsym(RTLD_NEXT, "getcwd"); return (*(char* (*)(char*, size_t))real_getcwd)(arg0, arg1); }
getcwd_mock(0, EACCESS); char* ret = function_we_want_to_test(arg); getcwd_unmock(); // verify that ret is 0 // verify that errno is EACCESS
A couple of caveats:
- The test is inherently aware of the implementation to test, and actually testing not the function as in given input results in the expected output, but rather that external conditions affect the expected implementation in the correct way. The function is no longer a black box, and the implementation cannot change without invalidating the test.
- Depending on the implementation, it can be hard to verify that the expected error path was entered. Maybe the success path can return the same value? But at least code coverage should show that the error path was executed.
- Link order will be important. By placing the mock library last it should resolve any references to the mocked entry points from the standard library.
posted by Mattias at
13:26
1 Comments
