Testing in C

In this lab, you will learn how to use a testing framework in C.

Objectives:

Describe the benefits of automated testing
Assemble a test suite using the Check software package
Add and modify tests in a test suite

Introduction

You may be familiar with the JUnit framework from Java. JUnit allows you to write automated tests for your code. In this course, we will be using Check, which is a C-based testing framework similar to JUnit.

Throughout this lab, we will be testing implementations of the following function:

    // takes an integer and returns the next prime number
    // that is greater than the parameter
    //
    int next_prime(int x)
    {
        return 0;   // TODO: finish this
    }

You should take a few minutes to consider how you would implement this function.

Lab setup

Here are the starter files for this project:

Download one of the archives and decompress it to a working folder. You will find three code modules (*.c files):

main.c - the main() program
primes.c - the implementation of the next_prime function
testsuite.c - the automated tests

You will also find primes.h, a header file for the primes.c module.

Because we will be using the Check framework, it is time to begin using the generic makefiles that we have posted on Piazza and/or on your course website. Download the makefile that is appropriate for the system you are working on and put it in the same folder as the other lab files, renaming it to simply Makefile (note the capitalization).

You will need to customize the generic makefile for each project. For this project, we will need to add a module target for the primes module to the MODS variable in the makefile. Find the appropriate line and add primes.o:

    # application-specific settings and run target

    EXE=main
    TEST=testsuite
    MODS=primes.o
    LIBS=

    ...

IMPORTANT: Even though you are adding an entry for the primes.c code file, you need to put primes.o (with the ".o" extension) in the makefile. Recall that the make build process runs on target information; i.e., a list of things that need to be built. All you need to do is tell make what you want to build, and the generic makefile has rules and patterns to help make figure out how to do it.

Also note that we didn't have to include main.o and testsuite.o in that list. We only have to list any extra code objects beyond the basic main.o and testsuite.o objects, which are already accounted for elsewhere in the generic makefile.

Test the build with make to ensure that everything is set up properly. You will need to use a similar process to set up all of the programming assignments in this course.

Part 1: Manual testing

In this lab, we will alternate between improving our tests and improving our code. This resembles a formal software development process called test-driven development (TDD). The idea behind TDD is that you should write tests for a program before you write the program itself. This requires you to think critically about the expected behavior of your program even before you begin to code it.

After you write some initial tests, you then write the minimal amount of code necessary to pass the test. After you have fixed any problems and all of the current tests are passing, you write more tests and then more code. Gradually, you will build up a large list of tests that help you verify that new changes don't break older features. This technique is also sometimes called the "test a little, code a little" strategy, and is a recommended approach for tackling the programming assignments in this class.

For this lab, we will (over the course of several exercises) gradually build up a suite of tests for the next_prime function. This suite will be tightly integrated with our build process (make) and thus it will be very easy to check that we haven't broken anything if we ever have to make changes to the code in the future.

Exercise 1: Ad-hoc tests

Let's write a test for next_prime. We'll start by writing an ad-hoc test in the main() function. This is a very rudimentary method of testing, but it is quick and simple.

DO THIS: Add the following code to your main() function:

    if (next_prime(2) == 3) {
        printf("x=2: correct\n");
    } else {
        printf("x=2: incorrect\n");
    }

This test encodes the following specification: "when given the number 2, the next_prime function should return 3." This is a concrete, testable statement that follows from the original function specification, which was the somewhat hard-to-test notion that the function "takes an integer and returns the next prime number that is greater than the parameter." When writing tests, you should always begin with the project specification and work towards concrete, testable test cases that are easy to code.

Compile and run the main program and see that the test fails.

DO THIS: Fix the code to make the test pass. For now, just change return 0; in next_prime to return x+1;. This is a minimal change necessary to make the given test pass.

Hopefully you can see that this does not fully implement the project specification for next_prime. Later, we will need a better implementation. This also exposes the inadequacy of our current test; it now passes, but that doesn't mean that the code matches the spec.

Exercise 2: Function-based tests

Ad-hoc tests in a main() function are quick to implement, but we want to be able to add lots of tests. If we have to duplicate all the code for a test every time we add a new one, we will end up with a very hard-to-maintain test suite.

We can improve the situation by extracting the test behavior into a separate function. Let's do that by adding a new test_next_prime function:

DO THIS: Move the testing code from main() into the test_next_prime function, which takes two parameters: a test input value and an expected test result value. This function should test next_prime with the given input value and check it against the expected result value. Thus, this function should work for testing any integer input. You will need to parameterize the code from the last section by changing the constants (2 and 3) to variable references.

Then, add the following tests to the main() function:

    test_next_prime(2, 3);
    test_next_prime(3, 5);
    test_next_prime(5, 7);

Compile and run to verify that the first test passes but the latter two fail.

DO THIS: Fix the code for next_prime so that all three tests pass. Here is an easy fix that should cause your implementation to pass all three tests:

    if (x == 2) {
        return x+1;
    } else {
        return x+2;
    }

Again, hopefully you can see the futility of coding to an incomplete test suite. Even though we have already "fixed" the function twice, we are not much closer to a true implementation of the project specification. This is why it is so important to have a comprehensive test suite.

Part 2: Automated testing

Thus far, we have confined our tests to ad-hoc and function-based tests in the main program module. However, it is much more robust to separate a program's tests into a separate module, enabling them to be maintained, compiled, and managed separately. This will also allow us to handle test crashes gracefully, and to use the main() function for actual program code.

As mentioned before, we will be using the Check framework for tests in this course. This will allow us to take advantage of robust testing code that someone else has written, meaning there is less work for us!

For this course, we will always put our test suites in a separate source code file that we will call testsuite.c. We have included a basic testsuite template in the starter files for this lab. Load that file now and look through it.

Actual tests are enclosed between START_TEST and END_TEST markers. These are actually macros, a copy/paste mechanism similar to the #include directive that we have seen already. The Check framework replaces these macros with code that runs the test, handles any crashes, and records the result. Every test is named, and the name is declared as a parameter to the START_TEST macro.

Inside of these tests, Check provides many convenience functions for actually performing tests. Here are the most important:

ck_assert(expr)

Evaluate expr; the check passes if the result is true and fails if it is false.
ck_assert_int_eq(x, y)

The check passes if the two integers are equal and fails if they are not.
ck_assert_str_eq(x, y)

The check passes if the two character strings are equal (according to the strcmp function) and fails if they are not.

Here is a reference to all of the checking functions provided by our testing framework.

IMPORTANT: Every test also needs to be registered with the testing framework using the tcase_add_test function; in our course framework, this is done in the test_suite function.

(Optional) Peeking behind the curtain

Read this section if you're curious about what the rest of the code in testsuite.c is doing; however, it is not strictly necessary to understand the content in this section in order to use the framework.

The test suite module (testsuite.c) contains two major framework functions: test_suite() and run_testsuite(). The former is responsible for building a Suite object that contains a list of all the tests in the test suite, and the latter is responsible for running the Suite and printing the results. The file also contains a main(), but it just calls run_testsuite.

Here is the code for the test_suite() function:

Suite * test_suite(void)
{
    Suite *s;
    TCase *tc_core;
    s = suite_create("Default");
    tc_core = tcase_create("Core");

    tcase_add_test(tc_core, TEST_NAME);

    suite_add_tcase(s, tc_core);
    return s;
}

The first four lines create Suite and TCase structures and initialize them. These structs are defined in the Check framework files (which are not part of our project). A Suite is a collection of TCase objects, and a TCase is a collection of actual test functions, which by convention should begin with the prefix "test_". The test functions themselves may contain multiple checks using the convenience functions (e.g., ck_assert_int_eq). Thus, there is a hierarchical, container-based relationship between all of these structures:

    + Suite struct
      + TCase struct
        + "test_*" function
          + individual "ck_assert_*" checks
        + "test_*" function
          + individual "ck_assert_*" checks
        + "test_*" function
          + individual "ck_assert_*" checks
        ...

For simplicity, we will only use a single Suite and TCase for all of the test suites in this course. We will separate tests into multiple "test_*" functions, each of which should have a small number of individual checks (usually just one).

The only modification that you should ever need to make to the test_suite function is to add each new test case using a call to the tcase_add_test function.

Here is the code for the run_testsuite() function:

int run_testsuite()
{
    int number_failed;
    Suite *s;
    SRunner *sr;

    s = test_suite();
    sr = srunner_create(s);

    srunner_run_all(sr, CK_NORMAL);
    number_failed = srunner_ntests_failed(sr);
    srunner_free(sr);

    if (number_failed == 0) {
        printf("SUCCESS: All current tests passed!\n");
    } else {
        printf("FAILURE: At least one test failed or crashed.\n");
    }

    return (number_failed == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
}

The first three lines allocate some variables, including a Suite struct (which is initialized in the fourth line using the test_suite() function described above) and an SRunner struct (which manages the actual execution of tests. These variables are initialized, and then the runner is executed using the srunner_run_all function. The Check library is then polled to see how many tests failed, and a summary message is printed. Finally, the function returns an exit value to indicate whether or not all of the tests passed.

You should not need to modify the run_testsuite function in this course.

Finally, if you look through the generic makefile, you will see a separate CFLAGS and LIBS variables for compiling the test suite:

TESTCFLAGS=$(CFLAGS) -Wno-gnu-zero-variadic-macro-arguments
TESTLIBS=-lcheck

The -Wno-gnu-zero-variadic-macro-arguments flag suppresses a particular (harmless) warning that is produced when compiling the Check header files, and the -lcheck flag is what tells the linker to include the Check library when it builds the final executable.

Exercise 3: Automated test

Let's re-implement our original ad-hoc test using the test framework.

DO THIS: Inside the test_prime_2 test case, add a check that the result of calling next_prime(2) is 3:

START_TEST (test_prime_2)
{
    ck_assert_int_eq(next_prime(2), 3);
}
END_TEST

This is much simpler than our previous code! We could also reformat the test to be even more compact:

START_TEST (test_prime_2)
{ ck_assert_int_eq(next_prime(2), 3); }
END_TEST

Such formatting would not be very readable for normal functions, but for simple test cases it is occasionally useful to be able to encode a large number of tests in a compact form.

In our generic Makefile, we provide a separate build target for automated tests: test. To compile and run the tests, just run make test. Verify that the test suite builds and runs. The output should look something like this:

Running suite(s): Default
100%: Checks: 1, Failures: 0, Errors: 0
SUCCESS: All current tests passed!

This output tells you how many checks were run, how many failures and errors were encountered, and what percentage of the tests passed. The last line is a general at-a-glance summary of the results.

Exercise 4: More tests

Now that it is easy to add more tests, let's add a lot more.

In particular, we should try to anticipate edge cases, which are input values that lie on the border of acceptable values. To find edge cases, think about what sorts of input values make sense for any particular function, and choose values that are unexpected or extreme. For example, we know that two is the smallest prime number, so we need to think about the behavior of our function for values smaller than two. what is the next prime of the value one? Zero? A negative number?

DO THIS: Add more tests for the following cases:

next_prime(3) should be 5
next_prime(5) should be 7
next_prime(7) should be 11
next_prime(14) should be 17
next_prime(42000) should be 42013
next_prime(1) should be 2
next_prime(0) should be 2
next_prime(-1) should be 2

IMPORTANT: For each test you add, make sure you remember to add a corresponding call to tcase_add_test in the test_suite function. Always verify that the number of checks run by the test suite matches your expectations.

Re-run the test suite. Which tests pass and which tests fail?

DO THIS: Write a proper implementation for the next_prime function. Does your implementation pass all the tests?

PA 0

Don't forget that PA 0 is due today. Hopefully you have already completed it; if not, make sure you do so. You may download the private test suite for this PA (and ONLY this PA) from Canvas. Just overwrite the old testsuite.c with the new one.

CS 240

Data Structures and Algorithms - Fall 2015