This course uses a testing infrastructure that
is similar to but very different from that used in CS 261.
Specifically for projects, you cannot run make test to
test and debug your code. You must follow these instructions in order
to get credit for your work.
While you may be familiar
with the automated testing framework used in CS 261, the information provided
by make test is insufficient for debugging code in this class.
Instead of re-running the automated tests and hoping for different results,
you should adopt a different strategy: use the automated tests just to
identify a test case to work on, then run the code manually or with a
debugger to trace the execution.
This document provides provides details on the testing infrastructure used
in this class and how it relates to the submission system.
Compiler Warnings as Errors
The first form of testing that you have to deal with is done by compiling
your code. All of the project source code distributions contain a
Makefile that specifies the compiler flags to be used. These
flags include -Werror -Wall -Wextra. These three flags enforce a
strict coding standard that you must follow. These flags specify additional
code checks beyond what is syntactically required. All code submitted
must compile cleanly with these flags in place.
This combination of flags prevents you from declaring a variable that you
do not use and using a local variable without initializing it. While both of
these are technically allowed by C, but they are a common source of bugs and
they are signs of sloppy coding. You should always
initialize local variables, especially pointers; 0 or
NULL are generally safe values. For unused variables or functions,
you need to add a GCC attribute to the variable to declare it as such:
int x __attribute__ ((unused));
int __attribute__ ((unused))
stub (void)
{
return 0;
}
int fun (void) __attribute__ ((unused));
int
fun (void)
{
return 0;
}
Testing Infrastructure
All project and lab distributions have the same structure:
project/src/ (projects only)
- This directory will contain all of the source code for the projects.
Some code will be provided, but you may need to create other files here,
too. For labs, the source code is in the main
lab
directory.
project/tests/
- This directory contains the testing infrastructure code.
project/build/
- This directory will be generated when you run
make to compile
your code.
The CS 361 testing infrasture contains several files and subdirectories that
are used to test your code. You may modify these files in your local instance
for additional testing if you'd like. However, we will grade your submission using
the original version included in the source code distribution.
project/tests/itests.include- This configuration file specifies command-line arguments that will be
used for integration testing. Lines in these files contain a test case
name and a list of command-line arguments in quotes, along with other
information.
project/tests/expected/- This directory contains text files with the expected output for
integration tests. For each test named in
itests.include,
there is a corresponding *.txt file in this directory.
project/tests/Makefile and
project/tests/integration.sh- These files are the drivers for the testing infrastructure. You should
not need to modify them, but you are encouraged to read through them.
project/tests/grade.sh and
project/tests/grade.sh.local- These are the scripts invoked with
make check and
make submit for testing and submitting your code.
project/tests/public.c- For labs, this file contains unit tests that focus on specific
functional units of the program. The tests we provide include both valid
and invalid input data to test your code for robust error handling. For
projects, this file is left (mostly) empty but is provided so that you can
write your own unit tests.
Lab Testing Procedures
The procedures for running tests in this class are different for labs and
for projects. This section describes the procedures specifically for labs.
Running all tests (labs only)
If you navigate to the lab directory and run the provided code
with the existing test cases, you will see the following output:
$ make test
make -C tests test
make[1]: Entering directory '/cs/home/stu/starkeri/cs361/lab/tests'
make -C ../
make[2]: Entering directory '/cs/home/stu/starkeri/cs361/lab'
make[2]: Nothing to be done for 'default'.
make[2]: Leaving directory '/cs/home/stu/starkeri/cs361/lab'
gcc -c -g -O0 -Wall --std=c99 -pedantic -Wextra testsuite.c
gcc -c -g -O0 -Wall --std=c99 -pedantic -Wextra public.c
gcc -g -O0 -o testsuite testsuite.o public.o ../build/helper.o -lcheck -lm -lpthread -lrt -lsubunit
========================================
UNIT TESTS
0%: Checks: 1, Failures: 1, Errors: 0
public.c:12:F:Public:add_2_3:0: Assertion 'add (2,3) == 5' failed: add (2,3) == 0, 5 == 5
========================================
INTEGRATION TESTS
Addition_3_5 FAIL (see outputs/Addition_3_5.diff for details)
No memory leak found.
========================================
make[1]: Leaving directory '/cs/home/stu/starkeri/cs361/lab/tests'
The first grouping of lines (before the first =======) show the
compilation of the test suite. This should not fail. If it does, the
problem is most likely that you have modified the public.c file in
a way that the header files are not correctly found. Also, make sure that your
tests use the START_TEST and END_TEST structure required by
the check framework.
The next grouping of lines show the results of the unit tests. The first line
indicates that the current code passed 0% of the test cases. There was 1 test
case (Checks) and it failed. The test cases can also detect
run-time errors, although this code did not produce any. After this first line,
you will be given a specific output about the test case failure. For instance,
this output indicates that add (2,3) should return 5, but it returned
0 instead.
The third grouping shows the output of the integration tests. While unit tests
focus on specific internal functions of the project code, the integration tests
exclusively compare the output produced with what was expected. Your code
must match the expected output verbatim to pass these tests.
Lab integration testing
While unit tests focus on individual components of your code, integration
testing will evaluate the complete functionality of your project. These tests
are based purely on the output produced by your code. It must match the
contents of the tests/expected/*.txt file verbatim.
If there is even a single extra space anywhere, the test case fails. Once you
have created an integration test case output file, you add the test case by
adding the following line to the tests/itests.include file:
run_test Add_two_negatives "-1 -2"
This line indicates that the Add_two_negatives test case should be
run with the following command line (add is the project executable
that was compiled in the project directory):
Correcting expected files
Each of the labs have integration tests that are intentionally incomplete.
The tests/expected/*.txt file has embedded instructions for
lines that you have to complete. If you do not correct these files,
these test cases are considered failed and you will not get credit for the
lab. As an example, consider the following expected file:
<<<< ./add 1 -2 >>>>
<<<< Replace the sum with the correct value >>>>
1 + -2 = << 0 >>
The corrected version of this file must eliminate all lines with the four
<<<< symbols while replacing the parts of lines with
two << symbols. The corrected version of the file above
would be (note there cannot be extra spaces anywhere!):
Project Testing Procedures
Although the lab testing procedures may be similar to what you are used to
from CS 261, the project testing procedures are very different. One major
reason for this difference is that the projects are divided into separate
features that have to be submitted as separate
claims. This section describes how features relate to the
testing infrastructure.
Running a single test
The first major difference to notice about the testing framework is that
make test does not run all tests in the project
structure. Instead, you will need to run specific tests or to run a different
command.
The structure of itests.include file is organized based on
the features required for the project. Lines in this file are structured
as follows:
BASIC positive evens "2 4"
BASIC positive odds "1 3"
These lines define BASIC (as opposed to MIN, INTER, or ADV) tests,
which reflect how they fit into the grading structure. (All BASIC and MIN
features are required for a grade of at least a C. See the syllabus for
more information.) Both tests are for the positive feature,
while the test names are evens and odds.
To run one (or more) of these tests, put their name(s) after the
make test command:
$ make test evens odds
========================================
INTEGRATION TESTS
----------------------------------------
evens pass
odds pass
----------------------------------------
SUCCESS: All tests passed
If one of the test cases fails, the testing code will tell you then
command line that was used to run the code.
Running a single test
Once you have passed all the tests individually, you can then test a
single feature (running all tests for it) as follows:
$ make check positive
========================================
CHECKING FEATURE
----------------------------------------
evens pass
odds pass
----------------------------------------
SUCCESS: All tests passed
----------------------------------------
BASIC-positive DONE
----------------------------------------
As you are working on new features, you should go back and re-run the
make check commands on older features to make sure your updates
have not introduced bugs or other issues. Once you feel that a feature is
complete and ready to submit, you must run make submit positive
(for the positive feature). This command requires that your
repository is set up correctly, and more details about what it does are
described in the Submission
Procedures Documentation.
Additional Notes
This section provides some additional information that may be helpful for
testing your code.
Unit Testing
Unit test cases are intended to test whether or not one particular function
or piece of code is working correctly. They should be very precisely
targeted in what they do. To create one, you can modify the
tests/public.c file. The structure of a test case is as follows:
/* This should never fail */
START_TEST (sanity_check)
{
ck_assert (1 == 1);
}
END_TEST
The test case is passed if the assertion in the ck_assert() is true.
You can specify as many assertions in a single test case as is appropriate. In
order to pass the test case, they must all be true. For more information
on how to create assertions, you should consult the
Check
API. You can also see more documentation (including a tutorial) on the
Check home page.
Once you have created a test case, you add it to the test suite by adding the
following line to the public_tests() function:
tcase_add_test (tc_public, sanity_check);
Most integration test cases use some sort of input file. These files are
typically placed in the project/tests/data directory. There is
also a symbolic link that makes these files accessible from the
project/data directory. You should not modify these files (which
might break real test cases), but you can reference them in your code or on
the command line. For instance, you could open the file data/foo.txt
and it will work regardless of whether you run the code from the
project directory or the project/tests directory.
Memory Leak Check
Our testing infrastructure also automatically runs
valgrind to check
for memory leaks. In short, if you do a malloc() somewhere but never
use free() to clean up the data, you have lost that memory. If you
program runs for long enough, these leaks add up, and you will never be able to
allocate new dynamic data structures. The following output shows what you would
see if you had a leak:
==919== LEAK SUMMARY:
==919== definitely lost: 4 bytes in 1 blocks
==919== indirectly lost: 0 bytes in 0 blocks
==919== possibly lost: 0 bytes in 0 blocks
==919== still reachable: 0 bytes in 0 blocks
==919== suppressed: 0 bytes in 0 blocks
==919== Rerun with --leak-check=full to see details of leaked memory
==919==
==919== For counts of detected and suppressed errors, rerun with: -v
==919== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
In this case, the program leaked 4 bytes of memory. Here is the program that
was used to generate this summary:
#include <stdio.h>
#include <stdlib.h>
int
main ()
{
int *p = (int)malloc (4);
return 0;
}
It is possible to get more information about the cause of the leak by running
valgrind on the test case manually with the -leak-check=full
option. That is, for the add program described above, you could run
the following line from the project/tests directory:
$ valgrind --leak-check=full ../add -1 -2
In this case, you would also see the following lines of output:
==922== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1
==922== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==922== by 0x400537: main (in /cs/home/stu/starkeri/cs361/project/tests/leak)
These lines indicate that the leaked memory was allocated by malloc()
inside of main(). At that point, I could look in my source code and
see that the variable p had memory allocated to it, but it was never
freed.
Error Checking
When testing your code, you MUST try to break it with invalid input.
Your code must detect this invalid input and react in an appropriate manner.
Your test cases must include the following conditions:
- invalid command-line flags
- missing required command-line arguments
- passing non-existent or empty files
- repeating command-line flags
- invalid arguments to flags
- invalid combinations of flags
- very large numbers and files
- passing invalid parameters to functions
- passing
NULL values and empty strings as arguments
