CS 240

Testing Lab

Objectives

The goal of today's activity is to practice working with assertions and unit testing in Python.

Part 1: Assertions

Assertions are used to verify that internal state of a program matches the programmer's expectations. Often, they are used to enforce preconditions for a function or snippet of code.

In Python, use the keyword assert and a boolean condition to check that condition at runtime. If the condition is false at the point it is checked, an AssertionError is raised. Here is an example demonstrating the use of assert in the Python interactive interpreter:

    Python 3.4.1 (default, Aug 13 2014, 16:11:43) 
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> assert True
    >>> assert False
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AssertionError
    >>> assert 4 < 5
    >>> assert 5 < 4
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AssertionError
    >>> assert 3 + 5 == 8
    >>> assert 3 + 5 == 9
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AssertionError

Often while you are writing code it is convenient to make assumptions about certain conditions in your program, such as a function parameter's data type. For example, a sqrt() function might assume that its parameter is a positive number. Such conditions may be true except in very rare circumstances (such as a bug in the code), so it doesn't make sense to write full-fledged error handling for the situation where the conditions are false. However, it is still good practice to document these assumptions, either in a comment or in an assertion. Here is an example:

def factorial(x):
    """ Returns the factorial of a positive integer. """
    assert isinstance(x, int)
    assert x > 0
    result = 1
    for i in range(1, x+1):
        result *= i
    return result

In the above code, the assertions at the beginning of the factorial() function will cause an exception to be raised if the parameter x is not a positive integer.

TASK: Download assertions.py and open it in an editor. This file contains several functions and a short main(). Each function has a docstring describing the behavior of the function and the desired properties of its parameters. Add assertions at the beginning of each function to enforce those documented requirements. You should also add some faulty calls to those methods in main() to test your assertions.

HINT: Recall that you can test an object's type using the isinstance() built-in function, like this:

    if isinstance(x, int):
        print("x is an integer!")

It is important not to overuse assertions. Python's dynamic type system raises automatic type errors when an object is used in a way that is not supported by its type, and it is not always useful to enforce type restrictions using assertions. In your code, you should weigh the benefits of the clarify provided by assertions against the loss of generality that they cause. For more information and some guidelines about the use of assertions in Python, see this wiki page.

However, for the purposes of this lab (to practice writing assertions), you should include assertions to enforce all of the restrictions included in the docstrings.

Part 2: Unit Testing

"Unit testing" refers to the practice of using tests to verify the behavior of individual code modules. This is distinguished from end-to-end testing of a larger piece of software, which is called "system" or "integration" testing. While all types of testing are important, Python provides a very simple way to write unit tests for your code, using the unittest module.

Here is an example of unit tests, taken from the blog post in today's reading:

import unittest
from primes import is_prime

class PrimesTestCase(unittest.TestCase):
    """Tests for `primes.py`."""

    def test_is_five_prime(self):
        """Is five successfully determined to be prime?"""
        self.assertTrue(is_prime(5))

if __name__ == '__main__':
    unittest.main()

The unittest module provides a main() function that looks for classes that inherit from unittest.TestCase, runs their setUp() method to initialize, then executes any methods whose names begin with test_. These methods should include calls to the assertion methods that the class inherits from TestCase. When the test suite is finished executing, the framework will report the number of tests run and the number that passed, as well as details on the ones that failed.

Here is a quick reference on the most useful assertions provided by TestCase:

In this lab, you will use the unittest framework to test implementations of the format_str function.

To begin, download testing_files.zip and unzip it into a folder on your computer. This zip file contains correct.py, which is a reference implementation of format_str from Lab 2 and Homework 1. We have also included several implementations (wrong##.py) that contain bugs.

We have also included two other files, one of which you will need to modify for this lab. Open tests.py in an editor and read it carefully. It contains a framework for testing the format_str function implementations. Currently it only includes a single unit test; you will need to add more.

class TestFormatStr(unittest.TestCase):
    """ This class contains unit test cases for format_str """

    def test_format0(self):
        """ Very basic test (just passthrough--no substitutions) """
        self.assertEqual(candidate.format_str("test", []), "test")

The single unit test is the method test_format0, which checks to make sure that the result of calling format_str("test", []) is the string "test".

The other file is run.py, which is a helper script that runs the tests in tests.py on all of the candidate code files (correct.py and wrong##.py). It will print results from each test run in sequence, starting with the correct version and then the incorrect versions. The script does this by creating for each test a temporary file candidate.py containing the contents of the file to test. Note that tests.py imports candidate. You shouldn't need to modify this file.

Here is example output from executing run.py:

    $ python3 run.py

    =====  TESTING correct.py  =====
    .
    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    OK

    =====  TESTING wrong01.py  =====
    test
    F
    ======================================================================
    FAIL: test_format0 (__main__.TestFormatStr)
    Very basic test (just passthrough--no substitutions)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "tests.py", line 26, in test_format0
        self.assertEqual(candidate.format_str("test", []), "test")
    AssertionError: None != 'test'

    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    FAILED (failures=1)

    =====  TESTING wrong02.py  =====
    .
    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    OK

    =====  TESTING wrong03.py  =====
    .
    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    OK

    =====  TESTING wrong04.py  =====
    .
    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    OK

    TOTAL FAILED 1/5

As you can see, correct.py passes the tests ("OK") but wrong01.py does not ("FAILED"). Check wrong01.py to make sure you understand why it does not pass the test in tests.py. Note that the test suite in tests.py is currently incomplete, because some of the incorrect versions pass the testing.

If run.py does not work on your computer, you can still complete the lab. Re-enable the commented-out lines (one at a time) in tests.py to import the individual candidate files. It will be slower this way, however, because you have to manually run the tests for each file.

TASK: Modify tests.py, adding tests until all of the incorrect versions fail at least one of the tests. Make sure that the correct implementation still passes all the tests (i.e., make sure that your tests are not overly restrictive). The final output for a correct test suite end with the following:

    TOTAL FAILED 4/5

You may find it helpful to use the diff utility to spot differences between the format_str implementations. Example:

    $ diff -u correct.py wrong01.py
    --- correct.py	2014-09-14 13:51:03.222753277 -0400
    +++ wrong01.py	2014-09-14 13:51:03.222753277 -0400
    @@ -1,5 +1,5 @@
     """
    -HW01: Basic Python (Correct Solution)
    +HW01: Basic Python
     
     Mike Lam
     August 2014
    @@ -24,7 +24,7 @@
                     new_str += str(values[index])           # replacement value
             else:
                 new_str += word                             # original text
    -    return new_str
    +    print(new_str)
     
     if __name__ == "__main__":
         pass

This makes it easy to see that the bug in wrong01.py is that it prints the result instead of returning it. That explains the "AssertionError: None != 'test'" message from the output of run.py. Figuring out the bug in each wrong version of the code may help you to write appropriate unit tests. You could also use some of the example test cases from the homework assignment page.

Submission

This lab will not be graded so there is nothing to submit. However, this lab may form the basis of a future homework assignment, and we may ask you to submit unit tests with future programming assignments. Make sure you keep a copy of your code for future reference. If you would like to discuss your solution or any problems you encounter while working on this lab, please come to office hours or make an appointment.