Better Result Reporting

As long as you have only one test function, the current result reporting is pretty clear. If a particular test case fails, all you have to do is find the test case in the check form and figure out why it’s failing. But if you write a lot of tests, you’ll probably want to organize them somehow, rather than shoving them all into one function. For instance, suppose you wanted to add some test cases for the * function. You might write a new test function.

  1. (defun test-* ()
  2.   (check
  3.     (= (* 2 2) 4)
  4.     (= (* 3 5) 15)))

Now that you have two test functions, you’ll probably want another function that runs all the tests. That’s easy enough.

  1. (defun test-arithmetic ()
  2.   (combine-results
  3.    (test-+)
  4.    (test-*)))

In this function you use combine-results instead of check since both test-+ and test-* will take care of reporting their own results. When you run test-arithmetic, you’ll get the following results:

  1. CL-USER> (test-arithmetic)
  2. pass ... (= (+ 1 2) 3)
  3. pass ... (= (+ 1 2 3) 6)
  4. pass ... (= (+ -1 -3) -4)
  5. pass ... (= (* 2 2) 4)
  6. pass ... (= (* 3 5) 15)
  7. T

Now imagine that one of the test cases failed and you need to track down the problem. With only five test cases and two test functions, it won’t be too hard to find the code of the failing test case. But suppose you had 500 test cases spread across 20 functions. It might be nice if the results told you what function each test case came from.

Since the code that prints the results is centralized in report-result, you need a way to pass information about what test function you’re in to report-result. You could add a parameter to report-result to pass this information, but check, which generates the calls to report-result, doesn’t know what function it’s being called from, which means you’d also have to change the way you call check, passing it an argument that it simply passes onto report-result.

This is exactly the kind of problem dynamic variables were designed to solve. If you create a dynamic variable that each test function binds to the name of the function before calling check, then report-result can use it without check having to know anything about it.

Step one is to declare the variable at the top level.

  1. (defvar *test-name* nil)

Now you need to make another tiny change to report-result to include *test-name* in the **FORMAT** output.

  1. (format t "~:[FAIL~;pass~] ... ~a: ~a~%" result *test-name* form)

With those changes, the test functions will still work but will produce the following output because *test-name* is never rebound:

  1. CL-USER> (test-arithmetic)
  2. pass ... NIL: (= (+ 1 2) 3)
  3. pass ... NIL: (= (+ 1 2 3) 6)
  4. pass ... NIL: (= (+ -1 -3) -4)
  5. pass ... NIL: (= (* 2 2) 4)
  6. pass ... NIL: (= (* 3 5) 15)
  7. T

For the name to be reported properly, you need to change the two test functions.

  1. (defun test-+ ()
  2.   (let ((*test-name* 'test-+))
  3.     (check
  4.       (= (+ 1 2) 3)
  5.       (= (+ 1 2 3) 6)
  6.       (= (+ -1 -3) -4))))
  8. (defun test-* ()
  9.   (let ((*test-name* 'test-*))
  10.     (check
  11.       (= (* 2 2) 4)
  12.       (= (* 3 5) 15))))

Now the results are properly labeled.

  1. CL-USER> (test-arithmetic)
  2. pass ... TEST-+: (= (+ 1 2) 3)
  3. pass ... TEST-+: (= (+ 1 2 3) 6)
  4. pass ... TEST-+: (= (+ -1 -3) -4)
  5. pass ... TEST-*: (= (* 2 2) 4)
  6. pass ... TEST-*: (= (* 3 5) 15)
  7. T