A Hierarchy of Tests

Now that you’ve established test functions as first-class citizens, the question might arise, should test-arithmetic be a test function? As things stand, it doesn’t really matter—if you did define it with deftest, its binding of *test-name* would be shadowed by the bindings in test-+ and test-* before any results are reported.

But now imagine you’ve got thousands of test cases to organize. The first level of organization is provided by test functions such as test-+ and test-* that directly call check. But with thousands of test cases, you’ll likely need other levels of organization. Functions such as test-arithmetic can group related test functions into test suites. Now suppose some low-level test functions are called from multiple test suites. It’s not unheard of for a test case to pass in one context but fail in another. If that happens, you’ll probably want to know more than just what low-level test function contains the test case.

If you define the test suite functions such as test-arithmetic with deftest and make a small change to the *test-name* bookkeeping, you can have results reported with a “fully qualified” path to the test case, something like this:

  1. pass ... (TEST-ARITHMETIC TEST-+): (= (+ 1 2) 3)

Because you’ve already abstracted the process of defining a test function, you can change the bookkeeping details without modifying the code of the test functions.6 To make *test-name* hold a list of test function names instead of just the name of the most recently entered test function, you just need to change this binding form:

  1. (let ((*test-name* ',name))

to the following:

  1. (let ((*test-name* (append *test-name* (list ',name))))

Since **APPEND** returns a new list made up of the elements of its arguments, this version will bind *test-name* to a list containing the old contents of *test-name* with the new name tacked onto the end.7 When each test function returns, the old value of *test-name* will be restored.

Now you can redefine test-arithmetic with deftest instead of **DEFUN**.

  1. (deftest test-arithmetic ()
  2.   (combine-results
  3.    (test-+)
  4.    (test-*)))

The results now show exactly how you got to each test expression.

  1. CL-USER> (test-arithmetic)
  2. pass ... (TEST-ARITHMETIC TEST-+): (= (+ 1 2) 3)
  3. pass ... (TEST-ARITHMETIC TEST-+): (= (+ 1 2 3) 6)
  4. pass ... (TEST-ARITHMETIC TEST-+): (= (+ -1 -3) -4)
  5. pass ... (TEST-ARITHMETIC TEST-*): (= (* 2 2) 4)
  6. pass ... (TEST-ARITHMETIC TEST-*): (= (* 3 5) 15)
  7. T

As your test suite grows, you can add new layers of test functions; as long as they’re defined with deftest, the results will be reported correctly. For instance, the following:

  1. (deftest test-math ()
  2.   (test-arithmetic))

would generate these results:

  1. CL-USER> (test-math)
  2. pass ... (TEST-MATH TEST-ARITHMETIC TEST-+): (= (+ 1 2) 3)
  3. pass ... (TEST-MATH TEST-ARITHMETIC TEST-+): (= (+ 1 2 3) 6)
  4. pass ... (TEST-MATH TEST-ARITHMETIC TEST-+): (= (+ -1 -3) -4)
  5. pass ... (TEST-MATH TEST-ARITHMETIC TEST-*): (= (* 2 2) 4)
  6. pass ... (TEST-MATH TEST-ARITHMETIC TEST-*): (= (* 3 5) 15)
  7. T