Unit Testing for Legacy C/C++

I recently had the opportunity to work with some really old C/C++ code to try and uncover a heap corruption bug. Not wanting to recreate the legacy build environment, with its equally old compiler, I made the decision to pull the questionable code onto my laptop to get access to some modern tooling. I also needed to get some tests around this library to verify its correctness and that's when I stumbled across a portable, header based, unit testing library called Catch that was exactly what I needed.
The beauty of Catch is in its simplicity. Open your favorite editor, add a couple of includes, write a test, compile and run. When you execute the application, it runs all of your tests and reports the results. The following is the compilation and execution with a test failure.

john$ clang -g -Wall -fprofile-instr-generate -fcoverage-mapping -lstdc++ -o testhost tests.cpp irisbuffer.cpp nullterm.cpp

john$ ./testhost

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

testhost is a Catch v1.9.4 host application.

Run with -? for options

-------------------------------------------------------------------------------

resize

  Shrink Buffer, Expect bytes cleared

-------------------------------------------------------------------------------

tests.cpp:251

...............................................................................

tests.cpp:264: FAILED:

  REQUIRE( buffer.getBytesUsed() == expectedLen )

with expansion:

  3 == 4

===============================================================================

test cases:  15 |  14 passed | 1 failed

assertions: 132 | 131 passed | 1 failed

Writing a Test in Catch

The test constructs are simple, yet powerful. Each test is denoted by a TEST_CASE macro which contains the name as well as support for some contextual tags. At this point you can write your standard Arrange, Act, Assert unit test and you are ready to run. One of the features I really like is sections. These allow you to organize your unit tests into more meaningful functional groups. The following is an example test which shows the basic test structures.

TEST_CASE( "resize", "[CIrisBuffer]" )
{
    char actual[] = "1234567890";
    size_t actualLen = sizeof(actual) - 1;
    
    SECTION("Shrink Buffer, Expect bytes cleared")
    {
        char expected[] = "123";
        size_t expectedLen = sizeof(expected) - 1;
        char expectedMem[] = "123\0\0\0\0\0\0\0";
        size_t expectedMemLen = sizeof(expectedMem) - 1;
        CIrisBuffer buffer(actual, actualLen);
        size_t expectedCapacity = buffer.getCapacity();
        char* expectedBuffer = buffer.getBuffer();
        
        buffer.resize(3);
        
        REQUIRE(buffer.getCapacity() == expectedCapacity);
        REQUIRE(buffer.getBytesUsed() == expectedLen);
        REQUIRE(0 == memcmp(buffer.getBuffer(), expected, expectedLen));
        REQUIRE(0 == memcmp(buffer.getBuffer(), expectedMem, expectedMemLen));
        REQUIRE(buffer.getBuffer() == expectedBuffer);
    }
    
    SECTION("Grow with fill, Expect fill with nulls")
    {
        char expected[] = "1234567890\0\0\0\0\0\0\0\0\0\0";
        size_t expectedLen = sizeof(expected) - 1;
        CIrisBuffer buffer(actual, actualLen);
        size_t expectedBytesUsed = buffer.getBytesUsed();
        char* oldBuffer = buffer.getBuffer();
        
        buffer.resize(expectedLen);
        
        REQUIRE(buffer.getCapacity() > expectedLen);
        REQUIRE(buffer.getBytesUsed() == expectedBytesUsed);
        REQUIRE(0 == memcmp(buffer.getBuffer(), expected, expectedLen));
        REQUIRE_FALSE(buffer.getBuffer() == oldBuffer);
    }
}

What about code coverage?

So the tests I wrote exposed the memory bug pretty quickly, but how could I be sure that was the only issue? I needed to know how much of the code I had covered with my tests. Fortunately, there is some great tooling support which makes this really easy to integrate into your test execution. The LLVM toolchain contains several utilities which make generating the coverage report a snap. The key is telling the compiler to generate profile and coverage mapping information when the code is executed. Here is an example using the clang compiler.

clang -g -fprofile-instr-generate -fcoverage-mapping -lstdc++ -o testhost irisbuffer.cpp nullterm.cpp tests.cpp

With these compiler options, when the application is executed a default.profraw file is generated which contains all the profile information from the run. The data file by itself is pretty worthless, but run it through a couple of utilities and you get the desired magic output. In order to get the raw profile data into a form that the coverage tool can use, it has to be transformed into an indexed profile data file. The llvm-profdata tool does this for you when using the merge command.

llvm-profdata merge -o testhost.profdata default.profraw

Now the fun begins. llvm-cov is used to visualize the coverage information. It has two modes, a coverage report mode and a source code overlay show mode. The report command reveals what your coverage is for each source file.

llvm-cov report ./testhost -instr-profile=testhost.profdata

As you can see, it included the coverage of the tests themselves. These can be excluded with some compiler magic by excluding the coverage flags for those files. If you want a more granular report you can specify one or more filenames on the command line and it will spit out the function level coverage. Seeing this data at the file level is great for high level reports, but we need to see it at the line level to see what we missed. That's where the show command comes in handy.

llvm-cov show ./testhost -instr-profile=testhost.profdata

  218|       |bool CIrisBuffer::clear(void)

  219|     58|{

  220|     58|    usedBytes = 0;

  221|     58|    if (xbuffer)

  222|     58|    {

  223|     58|        if (!ZeroMemory(xbuffer, capacity))

  224|      0|        {

  225|      0|            error = BUFFER_WIN32_ERROR;

  226|      0|            supplementalError = GetLastError();

  227|      0|            logErr(LOGERROR, "ZeroMemory failed.", GetLastError(), LOCATION);

  228|      0|        }

  229|     58|    }

  230|     58|    return true;

  231|     58|}

Now that's what I'm talking about!! I can see that I am only missing coverage of some negative cases, which frankly, I can live without due to the difficulty in forcing them to fail. What's also nice is that you can see the execution counts for each LOC. I like this as it tells me which parts of the code are hot and could possibly benefit from optimization.

Summary

Honestly, I wish I had found Catch years ago. The simplicity and ease of use make it stupid simple to write and run tests for new code, or to integrate it into older code with a little massaging. So if you find yourself digging through some legacy codebase wondering if it was ever fully tested, throw Catch into the mix and you might be surprised with what you uncover.