|
Allegro CL version 11.0 |
ANSI Common Lisp contains no functionality designed specifically for testing applications. Because testing is an essential part of application development, Franz Inc. is making public the test harness used internally for testing Allegro CL itself. (A test harness is a collection of macros and variables associated with testing, along with templates for test forms.)
To use the test harness, you must load the tester.fasl module. Do this by evaluating
(require :tester)
There are issues with running test in a multiprocessing environment. While these issues affect Lisp multiprocessing running on a single hardware processor (as happens in the non-SMP Allegro CL including all versions prior to 9.0), they are less serious in that case and so we have not addressed them previously. However, in an SMP Lisp, the issues are such that result output may be very difficult to read and interpret. See Running tests in multiple threads (Lisp processes) for more information.
All of the following symbols are exported from the util.test
package.
The test harness API includes the following variables, each described fully on its own page and briefly here.
These macros wrap around a form to be tested and supply the expected value (for the test macro) or the expected behavior, which is encoded in the macro name (e.g. test-error). For example:
(test 1 (+ 0 1)) ;; (testing that the result of (+ 0 1) is
;; the fixnum 1)
(test-error (+ 1 "2")) ;; (testing that an error is
;; signaled when a string is
;; passed as an argument to +)
Many more examples are given below.
with-tests wraps dynamically around a collection of test or test-* forms.
Note that many of the macros have fail-info and known-failure keyword arguments.
Each macro is described briefly here and fully on its documentation page.
Arguments: expected-value test-form &key (test #'eql) multiple-values fail-info known-failure
Perform a single test and compare expected-value with the value actually returned by test-form.
Arguments: form &key announce catch-breaks fail-info known-failure (condition-type 'simple-error) include-subtypes format-control format-arguments
Perform a single test to see whether form signals an error.
Arguments: form &key announce catch-breaks fail-info known-failure
Perform a single test to see that form does not signal an error.
Arguments: form &key fail-info known-failure
Perform a single test to see that form signals a warning.
Arguments: form &key fail-info known-failure
Perform a single test to see that
Arguments: (&key (name "unnamed")) &body body
Evaluates body, which should be a list of test forms, and reports on the results.
Arguments: (&optional (increment 1)
Increment one of the three test counters (atomically if necessary).
The following are simple examples using the test harness. The test forms themselves are trivial, and the purpose is to indicate the behavior of the test harness macros.
user(1): (require :tester)
; Fasl loading .../tester.fasl
t
user(2): (use-package :util.test)
t
user(3): (test 1 1)
t
user(4): (test 1 2)
* * * UNEXPECTED TEST FAILURE * * *
Test failed: 2
wanted: 1
got: 2
nil
user(5): (defun foo (x) x)
foo
user(6): (test 1 (foo 1))
t
user(7): (test 1 (foo 2))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (foo 2)
wanted: 1
got: 2
nil
user(8): (setq *break-on-test-failures* t)
t
user(9): (test 1 (foo 2))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (foo 2)
wanted: 1
got: 2
Break: *break-on-test-failures* is non-nil.
Restart actions (select using :continue):
0: return from break.
1: Return to Top Level (an "abort" restart)
2: Abort #<process Initial Lisp Listener>
[1c] user(10): :pop
user(11): (setq *break-on-test-failures* nil)
nil
user(12): (test 1 (error "foo"))
Error: foo
Restart actions (select using :continue):
0: Return to Top Level (an "abort" restart)
1: Abort #<process Initial Lisp Listener>
[1] user(13): :pop
user(14): (setq *error-protect-tests* t)
t
user(15): (test 1 (error "foo"))
Condition type: simple-error
Message: foo
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo")
Reason: an error (of type `simple-error') was detected.
nil
user(16): (setq *error-protect-tests* nil)
nil
user(17): *test-errors*
4
user(18): *test-successes*
2
user(19): (test 1 2 :known-failure t)
Test failed: known failure: 2
wanted: 1
got: 2
nil
user(20): (test 1 (foo 1) :known-failure t)
Expected test failure for (foo 1) did not occur.
nil
user(21): (test 1 (foo 1) :known-failure t :fail-info "This is bug666.")
Expected test failure for (foo 1) did not occur.
Additional info: This is bug666.
nil
user(22): (test-error (error "foo"))
t
user(23): (test-no-error (error "foo"))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo")
Reason: detected an unexpected error of type `simple-error'.
nil
user(24): (test-error (car '(10)))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (car '(10))
Reason: expected but did not detect an error of type `condition'.
nil
user(25): (test-no-error (car '(10)))
t
user(26): (test-warning (warn "foo"))
t
user(27): (test-no-warning (warn "foo"))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (warn "foo")
wanted: no warning
got: a warning
nil
user(28): (test-warning (car '(10)))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (car '(10))
wanted: a warning
got: no warning
nil
user(29): (test-no-warning (car '(10)))
t
user(30): (test-error (error "foo: ~a" 10))
t
user(31): (test-error (error "foo: ~a" 10) :format-control "foo: ~a")
t
user(32): (test-error (error "foo: ~a" 10) :format-control "foo: ~a"
:format-arguments '(10))
t
user(33): (test-error (error "foo: ~a" 10) :format-control "foo: ~a")
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: the format-control was incorrect.
wanted: "~1@<foo: ~a~:@>"
got: "~1@<foo: ~a~:@>"
nil
user(34): (test-error (error "foo: ~a" 10) :format-control "foo: ~a"
:format-arguments '(11))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: the format-arguments were incorrect.
wanted: (10)
got: (11)
nil
user(35): (test-error (error "foo: ~a" 10) :condition-type 'condition
:include-subtypes t)
t
user(36): (test-error (error "foo: ~a" 10) :condition-type 'simple-break
:include-subtypes t)
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: detected an incorrect condition type.
wanted: simple-break
got: #<standard-class simple-error>
nil
user(37): (test-error (break "foo: ~a" 10) :condition-type 'simple-break
:include-subtypes t)
Break: foo: 10
[condition type: simple-break]
Restart actions (select using :continue):
0: return from break.
1: Return to Top Level (an "abort" restart)
2: Abort #<process Initial Lisp Listener>
[1c] user(38): :pop
user(39): (test-error (break "foo: ~a" 10) :catch-breaks t
:condition-type 'simple-break :include-subtypes t)
t
When tests are run in multiple threads (also commonly known as Lisp processes, or Lisp light-weight processes) several issues must be addressed. These issues are most prominent in SMP implementations where threads can and do run simultaneously, but they can also occur (although with much lower probability) in both the virtual threads and os-threads versions of Allegro CL.
Tester messages from separate threads may become intermingled to make the output difficult to read. The latest version of the tester uses a lock on the output stream to make sure this does not happen to tester messages. The test application may also suffer from the same problem; it must do its own locking around messages or groups of messages.
It may be difficult to identify which thread produced which output message. The special variable *test-report-thread* is added to the tester to address this issue. When the value is non-nil
, each group of output lines from the tester is prefixed with a line identifying the thread that is printing the output.
When tests are run in the dynamic scope of a with-tests macro, the bound test counters accumulate counts only from the thread in which the with-tests macro was entered. Test counts from tests run outside any with-tests macro are accumulated in the global test counters. The global counters must be updated atomically in order to maintain correct and consistent counts. If the test application needs to update the test counters from user code, it must use the new macro inc-test-counter instead of incf or simple assignment.
Copyright (c) Franz Inc. Lafayette, CA., USA. All rights reserved.
|
Allegro CL version 11.0 |