A test harness for Allegro CL
1.0 Introduction
2.0 The tester module API
2.1 Test Harness Variables
2.2 Test Harness Macros
3.0 Examples
Index
1.0 Introduction
ANSI Common Lisp contains no functionality designed specifically for testing applications. Because testing is an essential part of application development, Franz Inc. is making public the test harness used internally for testing Allegro CL itself. (A test harness is a collection of macros and variables associated with testing, along with templates for test forms.)
The test harness facility will be part of the next major release of Allegro CL, and is available as a patch for Allegro CL 5.0.1. The patch, along with other patches providing new features (see the links in the Tech Corner[LINK] page) provides a preview of some of the technology available in the next major release of Allegro CL. The test harness facility is supported in ACL 5.0.1, and we encourage ACL customers to use it and to report problems and suggest improvements. Please address all correspondence (as with any technical issue, problem, or question) to support@franz.com.
To get the patch, evaluate (sys:update-allegro) in Allegro CL. This will automatically download and install all patches, including the one that implements the test harness facility (which will be in the downloaded file code/tester.fasl). Alternatively, you can manually download tester.fasl and tester.txt from the code/ directory of the patch directory (for the appropriate machine) from ftp://ftp.franz.com/pub/patches/5.0.1/.
Once the file tester.fasl is available, then (require :tester) is all that is needed to load it into Allegro CL.
2.0 The tester module API
All of the following symbols are exported from the :util.test package.
2.1 Test Harness Variables
*break-on-test-failures* [Variable]
Package: util.test
- When the value of this variable is nil, when a test failure occurs (that is, the behavior is not what is expected), information is collected but testing continues. When the value of this variable is true, common-lisp:break is called when a test failure occurs, allowing interactive debugging of the failure.
*error-protect-tests*
Package: util.test
- If true, protect each test from errors. If an error occurs, then that will be taken as a test failure unless test-error is being used. If nil, if an error occurs (except in a test-error form), an error is signaled and a break loop is entered.
*test-errors*
Package: util.test
- The value is the number of test errors which have occurred. This variable is used by with-test to total the errors during all the tests in the body.
*test-successes*
Package: util.test
- The value is the number of test successes which have occurred. This variable is used by with-test to total the successes during all the tests in the body.
*test-unexpected-failures*
Package: util.test
- The value is the number of unexpected test failures which have occurred. This variable is used by with-test to total the unexpected failures during all the tests in the body. (We say "unexpected" failues to distinguish them from expected failure caught by, say, test-error.)
2.2 Test Harness Macros
These macros wrap around a form to be tested and supply the expected value (for the test macro) or the expected behavior, which is encoded in the macro name (e.g. test-error). For example:
(test 1 (+ 0 1)) (testing that the result of (+ 0 1) is the fixnum 1)
(test-error (+ 1 "2")) (testing that an error is signaled when a string is passed as an argument to +)
Many more examples are given below.
with-tests wraps around a collection of test or test-* forms.
Note that many of the macros have fail-info and known-failure keyword arguments.
- fail-info, if non-nil, should be a string that will be printed if the test fails. Typical strings provide information about what is being tested, such as "This is bug2075".
- known-failure, if non-nil, affects what is printed when the test fails or succeeds. Thus a failure is reported as "Test failed: known failure: ..." and a success as "Expected test failure for [...] did not occur."
test
Package: util.test
Arguments: expected-value test-form &key (test #'eql) multiple-values fail-info known-failure
Perform a single test. expected-value is the reference value for the test. test-form is a form that will produce the value to be compared (using the predicate that is the value of the test keyword argument) to expected-value. If the values are not the same, then a failure is logged; otherwise a success is logged.
The comparison is done with the predicate that is the value of the test keyword argument. The default value of test is eql. Other standard comparison functions include eq, equal, equalp, string=, string-equal, etc., and you may, of course, write your own predicate.
Normally, only the first return value from the test-form is considered, however if multiple-values is t, then all values returned from test-form are considered. If multiple-values is true, expected-values must be a list. If the length of the expected-value list differs from the number of returned values, the test will fail.
The value of the fail-info argument should be nil or a string. If a string, it is printed if the test fails, providing more information with a test failure. (Typical strings are "bug2127" and "can fail when network is slow".)
known-failure if non-nil, affects what is printed when the test fails or succeeds. Thus a failure is reported as "Test failed: known failure: ..." and a success as "Expected test failure for [...] did not occur."
test-error
Package: util.test
Arguments: form &key announce catch-breaks fail-info known-failure (condition-type 'simple-error)
include-subtypes format-control format-arguments
Test that form signals an error. The order of evaluation of the arguments is keywords first, then form.
If announce is non-nil, then cause the error message to be printed.
If the catch-breaks is non-nil then consider a call to common-lisp:break an error.
The value of the fail-info argument should be nil or a string. If a string, it is printed if the test fails, providing more information with a test failure. (Typical strings are "bug2127" and "can fail when network is slow".)
known-failure if non-nil, affects what is printed when the test fails or succeeds. Thus a failure is reported as "Test failed: known failure: ..." and a success as "Expected test failure for [...] did not occur."
The value of condition-type must be a symbol naming a condition type (nil is not an acceptable value). The condition type is used to check against the signaled condition type. The test will fail if they do not match. Note: the default is simple-error so if this argument is unspecified, the test will fail if an error that is not a simple-error is signaled. If you want any error, specify the value of this argument error and the value of include-subtypes t.
include-subtypes, used with condition-type, can be used to match a condition to an entire subclass of the condition type hierarchy.
format-control and format-arguments can be used to check the error message itself. The comparison is done with equal.
test-no-error
Package: util.test
Arguments: form &key announce catch-breaks fail-info known-failure
Test that form does not signal an error. The order of evaluation of the arguments is keywords first, then form.
If announce is non-nil, then cause the error message to be printed.
If the catch-breaks is non-nil then consider a call to common-lisp:break an `error'.
The value of the fail-info argument should be nil or a string. If a string, it is printed if the test fails, providing more information with a test failure. (Typical strings are "bug2127" and "can fail when network is slow".)
known-failure if non-nil, affects what is printed when the test fails or succeeds. Thus a failure is reported as "Test failed: known failure: ..." and a success as "Expected test failure for [...] did not occur."
test-warning
Package: util.test
Arguments: form &key fail-info known-failure
Test that form signals a warning. The order of evaluation of the arguments is keywords first, then form.
The value of the fail-info argument should be nil or a string. If a string, it is printed if the test fails, providing more information with a test failure. (Typical strings are "bug2127" and "can fail when network is slow".)
known-failure if non-nil, affects what is printed when the test fails or succeeds. Thus a failure is reported as "Test failed: known failure: ..." and a success as "Expected test failure for [...] did not occur."
test-no-warning
Package: util.test
Arguments: form &key fail-info known-failure
Test that form does not signal a warning. The order of evaluation of the arguments is keywords first, then form.
The value of the fail-info argument should be nil or a string. If a string, it is printed if the test fails, providing more information with a test failure. (Typical strings are "bug2127" and "can fail when network is slow".)
known-failure if non-nil, affects what is printed when the test fails or succeeds. Thus a failure is reported as "Test failed: known failure: ..." and a success as "Expected test failure for [...] did not occur."
with-tests
Package: util.test
Arguments: (&key (name "unnamed") &body body
The body is a list of test forms. At the end of execution of the body, a report is generated which summarizes successes and failures. with-tests binds *test-successes*, *test-errors*, and *test-unexpected-failures* to 0 before the execution of body and the printing of the results.
name is printed with the results -- "Begin <name> test", so "Begin unnamed test" if unspecified.
3.0 Examples
The following are simple examples using the test harness. The test forms themselves are trivial, and the purpose is to indicate the behavior of the test harness macros.
user(1): (require :tester)
; Fasl loading .../tester.fasl
t
user(2): (use-package :util.test)
t
user(3): (test 1 1)
t
user(4): (test 1 2)
* * * UNEXPECTED TEST FAILURE * * *
Test failed: 2
wanted: 1
got: 2
nil
user(5): (defun foo (x) x)
foo
user(6): (test 1 (foo 1))
t
user(7): (test 1 (foo 2))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (foo 2)
wanted: 1
got: 2
nil
user(8): (setq *break-on-test-failures* t)
t
user(9): (test 1 (foo 2))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (foo 2)
wanted: 1
got: 2
Break: *break-on-test-failures* is non-nil.
Restart actions (select using :continue): 0: return from break. 1: Return to Top Level (an "abort" restart) 2: Abort #<process Initial Lisp Listener> [1c] user(10): :pop user(11): (setq *break-on-test-failures* nil) nil user(12): (test 1 (error "foo")) Error: foo
Restart actions (select using :continue):
0: Return to Top Level (an "abort" restart)
1: Abort #<process Initial Lisp Listener>
[1] user(13): :pop
user(14): (setq *error-protect-tests* t)
t
user(15): (test 1 (error "foo"))
Condition type: simple-error
Message: foo
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo")
Reason: an error (of type `simple-error') was detected.
nil
user(16): (setq *error-protect-tests* nil)
nil
user(17): *test-errors*
4
user(18): *test-successes*
2
user(19): (test 1 2 :known-failure t)
Test failed: known failure: 2
wanted: 1
got: 2
nil
user(20): (test 1 (foo 1) :known-failure t)
Expected test failure for (foo 1) did not occur.
nil
user(21): (test 1 (foo 1) :known-failure t :fail-info "This is bug666.")
Expected test failure for (foo 1) did not occur.
Additional info: This is bug666.
nil
user(22): (test-error (error "foo"))
t
user(23): (test-no-error (error "foo"))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo")
Reason: detected an unexpected error of type `simple-error'.
nil
user(24): (test-error (car '(10)))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (car '(10))
Reason: expected but did not detect an error of type `condition'.
nil
user(25): (test-no-error (car '(10)))
t
user(26): (test-warning (warn "foo"))
t
user(27): (test-no-warning (warn "foo"))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (warn "foo")
wanted: no warning
got: a warning
nil
user(28): (test-warning (car '(10)))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (car '(10))
wanted: a warning
got: no warning
nil
user(29): (test-no-warning (car '(10)))
t
user(30): (test-error (error "foo: ~a" 10))
t
user(31): (test-error (error "foo: ~a" 10) :format-control "foo: ~a")
t
user(32): (test-error (error "foo: ~a" 10) :format-control "foo: ~a"
:format-arguments '(10))
t
user(33): (test-error (error "foo: ~a" 10) :format-control "foo: ~a")
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: the format-control was incorrect.
wanted: "~1@<foo: ~a~:@>"
got: "~1@<foo: ~a~:@>"
nil
user(34): (test-error (error "foo: ~a" 10) :format-control "foo: ~a"
:format-arguments '(11))
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: the format-arguments were incorrect.
wanted: (10)
got: (11)
nil
user(35): (test-error (error "foo: ~a" 10) :condition-type 'condition
:include-subtypes t)
t
user(36): (test-error (error "foo: ~a" 10) :condition-type 'simple-break
:include-subtypes t)
* * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: detected an incorrect condition type.
wanted: simple-break
got: #<standard-class simple-error>
nil
user(37): (test-error (break "foo: ~a" 10) :condition-type 'simple-break
:include-subtypes t)
Break: foo: 10
[condition type: simple-break]
Restart actions (select using :continue): 0: return from break. 1: Return to Top Level (an "abort" restart) 2: Abort #<process Initial Lisp Listener> [1c] user(38): :pop user(39): (test-error (break "foo: ~a" 10) :catch-breaks t :condition-type 'simple-break :include-subtypes t) t
Index
Here are the macros and variables in this module.
*break-on-test-failures* [variable]
*error-protect-tests* [variable]
test [macro]
test-error [macro]
*test-errors* [variable]
test-no-error [macro]
test-no-warning [macro]
*test-successes* [variable]
*test-unexpected-failures* [variable]
test-warning [macro]
with-tests [macro]
Last updated on March 29, 2000 09:31:08 Pacific Standard Time
| Copyright © 2023 Franz Inc., All Rights Reserved | Privacy Statement | 
|
|
|
|
|
|