ToC DocOverview CGDoc RelNotes FAQ Index PermutedIndex
Allegro CL version 11.0

Miscellaneous Functionality

1.0 Introduction

This document describes functionality that does not naturally fit in any of the other overview documents.

2.0 The fasl reader/writer

The functions fasl-write and fasl-read provide a mechanism for writing Lisp data and subsequently reading it back into a Lisp image. It can handle many of the common Lisp data types. It can optionally detect circularity and structure sharing in the data and recreate the same topology up to eql-ness of components. The data is written in a binary file format similar to that used for compiled Lisp files, not in ASCII.

Among the advantages of fasl-read and fasl-write over standard Common Lisp read and print functions is that data does not have to be converted to its printed representation prior to being written (that conversion takes a significant amount of time for complex objects). The main disadvantage is that the files (unlike ASCII files with printed representations of Lisp objects) are not at all portable between versions of Lisp (or even between versions of Allegro CL on different platforms or between major releases of Allegro CL).

Not all Lisp objects may be written with fasl-write (and thus are not available to be read by fasl-read). The objects that cannot be written are CLOS objects, streams, stacks, or c-allocated data (cstructs). You can use fasl-open to open a file suitable for numerous fasl-write's. (fasl-read reads all of whatever file is specified to it.)

fasl-read reads the entire contents of its argument file. However, fasl-write, if its argument is a stream rather than a filename, writes data to the stream but does not close it. Thus you can open a stream and do multiple fasl-write's to it. The stream must have :element-type (unsigned-byte 8). fasl-open opens an appropriate stream, as shown in the example.

Here is a simple-minded example to show the correct syntax and give an idea of the effect:

(excl:fasl-write '(a b #1=#:c (#1#) #*1011 #(1 2 3) 
                  
1 1.1 1.1d0 1/2
                  
#c(1 1) #c(1.5 1.5) "abc")
     "test.fw")
(excl:fasl-read "test.fw")

(setq f (fasl-open "test2.fw")) 
(excl:fasl-write '(a b c) f) 
(excl:fasl-write '#1=(a b . #1#) f t) 
(excl:fasl-write '#(1 2 3 4.5) f)
(close f)

(excl:fasl-read "test2.fw")

Application note

fasl-read and fasl-write require the compiler so they are not available in application images without the compiler.

3.0 Miscellaneous extensions

The table describes those extensions to Common Lisp that do not naturally fit elsewhere in the documentation. We only provide brief information in the table. Please follow the link to the documentation page for a full description.

Name Arguments Notes
dribble-bug &optional file This function is an extension of the Common Lisp function dribble. dribble-bug called with the optional file calls several information functions whose output is then placed at the beginning of the dribble file. See also *dribble-bug-hooks*.
uncompile function-name

If the function function-name was compiled with the compile function (as opposed to having been in a file that was compiled with compile-file and subsequently loaded), then the function is `uncompiled,' i.e. its function definition is replaced by the original interpreted definition.

This function will only work when definitions are saved. See the description page for uncompile and the page for *save-function-lambda-expression* for details.
bignump object These functions, like similar ones in standard Common Lisp return t if object is of the type specified, and nil otherwise.
fixnump
ratiop
single-float-p
file-older-p file-1 file-2 If file-1 and file-2 both exist, and if file-1 is older than file-2, this function returns t. Otherwise, it returns nil.
if* test-form {then then-form+ | thenret} {elseif else-test-form {then else-then-form+ | thenret}}* [else else-form+] This extension to cl:if allows symbols like then, else, elseif, and thenret in the body allowing a complex number of cases and outcomes to be specified.
named-readtable name &optional errorp This function looks up a readtable by name. name must be a symbol, but it is coerced to a keyword (so readtables are named by keywords). setf may be used to associate a name to a readtable. The association can be broken by setting to nil. See also with-named-readtable.
prefixp prefix sequence This function returns non-nil if sequence starts with prefix, returning the index in sequence where prefix ends (so (prefixp "foo" "foobar") returns 3). While this can be done using regular-expression matching (see regexp.html, this function is easier to use for very simple cases.
gen-sym x Similar to gensym except that a symbol is accepted as the optional argument (as well as the standard string or integer), in which case the symbol-name is used. This is useful for programs that are intended for use in either ANSI or modern Lisps, where it is desired to print any gensyms without requiring escapes (see case.html).

3.1 Day and date calculation functions and their relation to time zones

Allegro CL has a number of functions to do calculations involving the day portion of a universal time. These functions, listed below with links to more complete documentation, do things like determining whether two universal times fall on the same day, or which is on a later day, or how many days pass between one universal time and another.

The day of a universal time depends on the time zone used to decode the time. Consider the universal time associated with 2:00 AM on the 27 of November, 2017, GMT:

(encode-universal-time 0 0 2 27 11 2017 0)
RETURNS 3720736800

Applying decode-universal-time to that value and specifying the GMT time zone (0) as the value of the optional time-zone argument, we find:

(decode-universal-time * 0)
RETURNS 0 0 2 27 11 2017 0 nil 0

The 7th returned value (0) indicate the day (Monday). But let us look at the same universal time but in time-zone 8 (western North America):

(decode-universal-time 3720736800 8)
RETURNS 0 0 18 26 11 2017 6 nil 8

Now the date is November 26 (not 27) and the day is Sunday (6) not Monday.

Consider the universal time 3720762000, seven hours later. Here is that time decoded, again with the GMT time zone and the Western American time zone:

(decode-universal-time 3720762000 0)
RETURNS 0 0 9 27 11 2017 0 nil 0
(decode-universal-time 3720762000 8)
RETURNS 0 0 1 27 11 2017 0 nil 8

In the GMT time zone, 3720762000 is 09:00 AM, still on Monday, November 27, 2017. But in the western America time zone, it is 01:00 AM, Monday, November 27, 2017, that is no longer Sunday, November 26.

Consider the function date=. It takes two universal times and returns true if they fall on the same date and nil otherwise. If the time zone is GMT (0), then (date= 3720736800 3720762000) returns t as both times are on Monday, November 27, 2017. But if the time zone is the western America time zone (8), (date= 3720736800 3720762000) returns nil as the times fall on different days.

Daylight saving time can also affect day and date calculations, as the following example shows:

cl-user(133): (setq ut1 (encode-universal-time 0 1 0 5 11 2017))
3718854060
cl-user(134): (setq ut2 (+ ut1 (* 24 60 60))) ;; adding 24 hours
3718940460
cl-user(135): (day-difference ut1 ut2) ;; but still the same day
0
cl-user(136): (decode-universal-time ut1) 
0 1 0 5 11 2017 6 t 8
                  ;; DST in effect (8th return value is T)
cl-user(137): (decode-universal-time ut2)
0 1 23 5 11 2017 6 nil 8
                  ;; DST not in effect (8th return value is NIL)
cl-user(138):

The time zone used by all the functions listed below is the system time zone taken from the machine running Lisp. Whether daylight saving time is in effect is also determined by the machine. You can see what time zone is set by the machine and get information about daylight saving time by evaluating the following form:

(decode-universal-time (get-universal-time))
RETURNS 52 51 14 29 11 2017 2 nil 8

The last return value (8, as this was run in Franz Inc. headquarters in Lafayette CA, USA) is the system time zone used by default by Lisp. The next to last value (nil) says daylight saving time is not in effect in Lafayette CA at that universal time.

So be aware that these function may return different results with the same inputs when run in different locations around the world. Currently, there is no way to specify the time zone that should be used by these functions. Thay all always use the system default time zone.

Name Arguments Notes
day-difference start end &key work-days start and end must be non-negative integers representing universal times. This function returns the number of days from the date of start to the date of end. The time of day encoded in start and end is ignored. If both start and end represent the same day, 0 is returned. work-days, if specified true, causes Saturdays and Sundays to be ignored. See the page for day-difference for numerous examples.
date= ut1 ut2 Returns true is the day/date of ut1 is the same as the day/date or ut2; returns nil otherwise.
date> ut1 ut2 Returns true is the day/date of ut1 later than the day/date or ut2; returns nil otherwise.
date>= ut1 ut2 Returns true is the day/date of ut1 is the same as or later than the day/date or ut2; returns nil otherwise.
date< ut1 ut2 Returns true is the day/date of ut1 is earlier than the day/date or ut2; returns nil otherwise.
date<= ut1 ut2 Returns true is the day/date of ut1 is the same as or earlier than the day/date or ut2; returns nil otherwise.
day-of-week ut Returns an integer signifying the day of the week of the universal time ut. 0 represents Monday, 1 Tuesday, and so on through 6 Sunday.
date+ ut number-of-days &key work-days Returns a universal time whose day/date is number of days later than the day/date of ut. If work-days is true, Saturdays and Sundays are not counted is determining the new day/date. Note that the resulting universal time may have different second, minute, and hour values that the argument universal time.
day-number ut Returns the (1-based) day number of the day/date associated with the universal time ut. January 1 has day-number 1, and December 31 has day-number 365 in non-leap years and 366 in leap years.
week-number ut &key (first-day-of-week :monday) Returns the week number of the week containing the day/date associated with the universal time ut. Weeks are considered to start on Monday unless first-day-of-week is specified to be :sunday. The first full (7-day) week of the year is week number 1. The partial (less than 7-day) week preceding week 1 is week 0. When January 1 falls on the first-day-of-week, there is no week 0.

There are also constants whose values are the number associated with a day of the week (0 for Monday, 1 for Tuesday, etc.) The constants are *decoded-time-monday*, *decoded-time-tuesday*, *decoded-time-wednesday*, *decoded-time-thursday*, *decoded-time-friday*, *decoded-time-saturday*, and *decoded-time-sunday*.

4.0 Creating and using pll files using cvdcvti/cvdcvt

A pll file can be used in association with a Lisp image. It contains constant code vectors and strings that can be shared among many Lisp objects. When an image uses a pll file and a function is compiled, the new codevector is compared to codevectors in the pll file. If a match is found, the match is used and no new codevector is allocated. Similarly, if a constant string is specified (with pure-string and a matching string appears in the pll file, no new string is allocated and the match is used. Strings in a pll file cannot be modified. Attempting to do so causes an error. (Neither can codevectors be modified but there is no user-visible way to modify codevectors as there is with strings.)

Strings and codevectors in a pll file are not also (after being garbage collected) in the Lisp heap. Thus if a string has been successfully purified, it will not be in the heap after a global gc. A total count of strings and codevectors is shown in the output of (room t).

Constant strings in pll files cannot be overwritten (constant strings stored in the heap can be overwitten although doing so is actually illegal). Strings naming Common Lisp symbols are usually stored in pll files, for example. This example shows in that case, the strings cannot be modified:

cl-user(1): (pll-file)
#P"/usr/fi/dcl.pll"
cl-user(2): (nstring-upcase (symbol-name 'car))
Error: Attempt to store into purespace address #x2d7865e8.
  [condition type: purespace-write-error]

Here we tried to upcase the string naming the symbol car (this is a modern Lisp where symbol names are lowercase -- see case.html). The change failed and signaled purespace-write-error, which is the condition specific to this issue. The next example shows that modifying a constant strings not in the pll file does not signal an error:

cl-user(88): (setq teststring "this-string-is-likely-not-in-the-pll-file")
"this-string-is-likely-not-in-the-pll-file"
cl-user(89): (dotimes (i 4) (gc))
nil
cl-user(90): (nstring-upcase teststring)
"THIS-STRING-IS-LIKELY-NOT-IN-THE-PLL-FILE"
cl-user(91): teststring
"THIS-STRING-IS-LIKELY-NOT-IN-THE-PLL-FILE"
cl-user(92):

The call to nstring-upcase is in fact illegal: you may not modify a constant, but it is an error that does not signal an error unless the string is located in the pll file.

pll files are created with the cvdcvti program (16-bit characters) or the cvdcvt program (8-bit characters) described next.

cvdcvt[i] Arguments: [-o outfilename] [-u] [file1 file2 ...]

Use cvdcvti for 16-bit character (also called international) Lisps and cvdcvt for 8-bit character Lisps. A .pll file, outfilename, is created holding all the unique code vectors and strings. If outfilename is omitted it defaults to code.blob.

If -u is specified, then no duplications of strings are done, otherwise for every string that has no lowercase characters in it and at least one uppercase character, a lowercase copy is added to the output file. This is the default and is useful for set-case-mode. If no files are specified, stdin is used. .str (string) files and .cvs (code vector) files are combined without redundancies; if two files of the same extension have identical objects, the object from the file specified first to cvdcvt is retained, and the latter object is removed from the output. This allows for files (.cvs files especially) to be arranged by code vectors in order of execution, to provide for locality of reference. Those .cvs files that were produced by training techniques should be placed first in order to have the desired effect.

As said in the description, pll files are built out of cvs files and str files. cvs files are created with sys:write-codevectors and can be created by sys:flush-codevectors. str files are created with sys:record-strings. See also sys:record-code-vectors.

The following functions can be used to associate a pll file with an image, to find out which pll file is used with an image, and to use strings in the pll file.

Name Arguments Notes
pll-file [none] Return the location of the current .pll file, or nil if there is no .pll file associated with this Lisp.
use-pll-file name &key (global-gc t) Associates the current Lisp with the pll file specified by name. It is an error to associate an image already using a pll file with another pll file.
pure-string x When not in the body of sys:record-strings, returns a pure-string if there is one identical to the argument and a heap-allocated string if there isn't. When in the body of sys:record-strings, also write the string to the str file being created.

5.0 MD5, SHA*, HMAC, RC4, and other message digest support

Allegro CL provides various message digest algorithms. Various algorithms are supported, including MD2, MD4, MD5, SHA1, SHA224, SHA256, SHA384, SHA512, and RMD160. Rather than have functions for each of these types, there are general functions for them, named digest-* and listed first. HMAC support is available for :md5, :sha1, :sha224, :sha256, :sha384, and :sha512. The functions hmac-init, hmac-update, hmac-string, and hmac-final implement these algorithms.

The MD*, SHA* and RMD160 functions are all are cryptographic hash functions. A hash function takes a long string (or message) of any length as input and produces a fixed-size output. This output is sometimes termed a message digest.

The keyed-hash algorithm HMAC is designed to work with algorithms MD5 and SHA* (for *=1, 224, 256, 384, and 512, to list what Allegro CL currently supports). Keyed hash algorithms work by utilizing an existing hash function (such as MD5 or SHA1) and using a secret "key" as part of the information to hash. If one party sends a message to another party and also includes an HMAC with the message, the receiver (if he/she shares the same secret key) can verify that the message hasn't been altered by running the message through the HMAC functions using the same key. If the two HMACs match, then the message is considered, with high probability, to be unaltered

RC4 is a stream cipher algorithm. It is used to encrypt streams of data.

Many of these function require SSL support (see Secuere Socket Layer (SSL) in socket.html).

The functions available are:

General digest functions

These functions are in the :digest module loaded with (require :digest).

MD5 functions

SHA1 functions

These functions are in the :sha1 module loaded with (require :sha1).

Related utility functions

HMAC functions for MD5 and SHA*

These functions are in the :hmac module loaded with (require :hmac).

RC4 functions

These functions are in the :rc4 module loaded with (require :rc4).

6.0 Base64 support

Allegro CL provides support for Base64 encoding within Lisp. Base64 encoding is a 64-bit representation scheme that uses the ASCII characters A-Z, a-z, 0-9, + and /. Since padding could be needed in converting multiples of 8-bits into base64, = characters are used, when necessary, as padding at the end of a converted string. Base64 encoding is described in the RFC2045 document (www.ietf.org/rfc/rfc2045.txt).

These functions provide the Base64 support in Allegro CL:

Here are some examples, first using integer-to-base64-string and base64-string-to-integer:

cl-user(2): (integer-to-base64-string #xfeedfacefeedface)
"/u36zv7t+s4="
cl-user(3): (base64-string-to-integer "/u36zv7t+s4=")
18369614222061337294
cl-user(4): (format t "~x" *)
feedfacefeedface
nil
cl-user(5):

And now using usb8-array-to-base64-string and base64-string-to-usb8-array:

;;  The encoding results may differ between Windows and 
;;  UNIX/Linux/Mac OS X (the transcript is from a
;;  UNIX machine).

cl-user(5): (setq a (string-to-octets
                     (setq s
                       "
(defun deep-thought ()
  (sleep (years2secs 7500000))
  42)
")
                     :external-format (crlf-base-ef :latin1)))
#(10 40 100 101 102 117 110 32 100 101 ...)
cl-user(6): (usb8-array-to-base64-string a)
"CihkZWZ1biBkZWVwLXRob3VnaHQgKCkKICAoc2xlZXAgKHllYXJz
MnNlY3MgNzUwMDAwMCkpCiAgNDIpCgA="
cl-user(7): (base64-string-to-usb8-array *)
#(10 40 100 101 102 117 110 32 100 101 ...)
cl-user(8): (setq a2 *)
#(10 40 100 101 102 117 110 32 100 101 ...)
cl-user(9): (equalp a a2)
t
cl-user(10): (octets-to-string a2 :external-format (crlf-base-ef :latin1))
"
(defun deep-thought ()
  (sleep (years2secs 7500000))
  42)
"
61
61
cl-user(11):

7.0 Support for encryption

Allegro CL provides implementations of some publicly available encryption algorithms: blowfish (see Support for Blowfish encryption) and rsa (see Support for rsa encryption). Please note that we make no claims about the actual security provided by these encryption schemes.

7.1 Support for Blowfish encryption

The Blowfish algorithm, described on this page (and links from it), is a high speed symmetric cryptographic algorithm (or cipher). The same key is used to encrypt and decrypt the data. Blowfish encrypts blocks of 64 bits (8 octets) at a time. The functions below can automatically pad out the data to encrypt to be a multiple of 8 octets. Blowfish was designed by Bruce Schneier, a leading authority on cryptography and author of the book Applied Cryptography. Schneier writes in his book published in 1996: "I know of no successful cryptanalysis against Blowfish."

Here are some examples of Blowfish encryption and decryption:

;; Example 1. string encrypting

cl-user(12): (blowfish-encrypt "my secret message" 
                        :key "my key")
#(57 27 110 242 191 19 182 150 1 5 ...)
24
cl-user(13): (blowfish-decrypt * :key "my key" :string t)
"my secret message"
cl-user(14): 

;; Example 2. (unsigned-byte 8) encrypting:

;; Here we allocate an (unsigned-byte 8) array with a size 
;; that is a multiple of 8 and fill it with data.  
;; We can do in-place encryption and decryption.
;; We specify no padding (since otherwise 8 bytes of padding would
;; have to be added and there's no room in this array for that):

;; Create our array:

cl-user(12): (setq aa (make-array 8 :element-type '(unsigned-byte 8) 
                        :initial-contents '(2 4 6 8 10 12 14 16)))
#(2 4 6 8 10 12 14 16)

;; Encrypt it in place

cl-user(13): (blowfish-encrypt aa :key "my key" 
               :in-place t :pad nil)
#(129 144 108 210 20 227 10 58)
8

;; Verify that it has been modified:

cl-user(14): aa
#(129 144 108 210 20 227 10 58)

;; Now decrypt in place. Notice how the arguments to 
;; blowfish-decrypt match those to blowfish-encrypt:

cl-user(15): (blowfish-decrypt aa :key "my key" 
                 :in-place t :pad nil)
#(2 4 6 8 10 12 14 16)
cl-user(16): 

;; And verify that the array is now back to normal:

cl-user(16): aa
#(2 4 6 8 10 12 14 16)
cl-user(17): 

;; Example 3. use of contexts

;; Create context which holds the key processed by 
;; blowfish to prepare it for encryption/decryption:

cl-user(21): (setq cc (blowfish-init "my key"))
#(141 90 172 196 250 88 140 57 179 211 ...)

;; Encrypt something using the context:

cl-user(22): (blowfish-encrypt "my message" :context cc)
#(75 202 37 143 4 243 181 205 211 126 ...)
16

;; And now decrypt it using the same context
;; to show the original string

cl-user(23): (blowfish-decrypt * :context cc :string t)
"my message"
cl-user(24):

It is a common practice to send Blowfish keys to intended recipients using their RSA public keys. See Support for rsa encryption for information on RSA encryption.

7.2 Support for rsa encryption

RSA is a public key cipher named after its inventors: Rivest, Shamir and Adleman. A public key cipher differs from a symmetric cipher like Blowfish (see Support for Blowfish encryption) in two important ways:

  1. There exist two keys: the Public key and the Private key.

  2. Different keys are used for encryption and decryption.

  3. One of the keys (the Public key) can be made public without making it possible to compute the other key (the Private key).

With RSA you can encrypt with the Public key and decrypt with the Private key or encrypt with the Private key and decrypt with the Public key. Typically one encrypts with the Public key to send a message to the person with the Private key.

RSA has never been proven to be secure. However the obvious way to crack the encryption involves factoring a very large number. There is no published way of factoring a large number that's better than a brute force attempt of trying all possible factors. Thus by making the key big enough you can be sure that it won't be possible to compute the factors by brute force search in a very long time. There may be other ways to crack RSA encryption that are simply not published yet.

One major downside to RSA is that it is roughly 1000 times slower to encrypt and decrypt than a symmetric cipher like Blowfish. As a result people usually use RSA as means of transmitting a key for a symmetric cipher. For example if Alice wants to send Bob a large document securely she'll first go to Bob's web site and copy down his Public RSA key. Then she'll use a random number generator to create a 64 bit blowfish key. She'll encrypt the blowfish key with Bob's Public key and send the result to Bob. Then she'll encrypt her document using Blowfish and the key she generated. Bob will decrypt the first message from Alice using his Private RSA key. That will give him the Blowfish key he'll need to decrypt the second message from Alice.

Because the public key is known to all you have to be careful to not encrypt small values with an RSA public key since that gives you very little security. For example, suppose you decide to encrypt a 4 digit security code using an RSA public key. A person willing to steal your code need only encrypt the values 0000 through 9999 and compare them to your encrypted value to determine what the value encrypted was. If you want to encrypt a 4 digit security code XXXX then it's best to encrypt instead YYYYYYYYXXXX where the Y's are digits chosen randomly.

An RSA key pair consists of three integers: a modulus, a private exponent and a public exponent. The only number that must be kept secret is the private exponent. The public exponent is usually one of a set of common small numbers. The Allegro RSA key generator always chooses 17 as the public exponent.

An RSA key is represented in Allegro as a vector of three values:

  1. t if this is the public key, nil if this is the private key. This value is to help you distinguish one key from the other and is not used in the encryption/decryption code.

  2. the modulus value (approximately 1024 bits long).

  3. the exponent value.

RSA is a block cipher: a sequence of octets is encrypted at once. The block size isn't fixed but is usually determined by the size of the modulus. In order to encrypt data whose length is not a multiple of the block size padding is done at the end of the value and information about the padding is added to the value. The format of this padding information is not standard among rsa encryption functions, thus you can't expect any function except rsa-decrypt to be able to decrypt a value encrypted with rsa-encrypt.

The functions associated with RSA encryption and decryption are:

Because RSA encryption is resource intensive compared to symetric encoders like Blowfish, it is a common practice to encode using Blowfish and send Blowfish keys to intended recipients using their RSA public keys. See Support for Blowfish encryption for information on Blowfish encryption.

Here are some examples of Blowfish encryption and decryption:

;; A call to generate-rsa-keys, such as the following, can take 
;; on the order of 10 minutes to complete. The example call
;; could have been made with ':verbose t' to get progress
;; information as it runs. 
;;
;; The return value is a list of the public and
;; private keys, both of which are vectors.  This list
;; is made the value of the variable 'keys'.


cl-user(12): (setq keys (generate-rsa-keys :verbose nil))
(#(t
   4696616306992156162791359909817969438301590857320651704912\
5099728659553054438846018512904176959283177314807123575693727\
9515543419344057970899365859403176313951068268266882944649562\
1008090347981854919956845970556254842289211552574616675107428\
9213609596618613446079618857135830766959762009927055865884710\
796501
   17)
 #(nil
   46966163069921561627913599098179694383015908573206517049125\
09972865955305443884601851290417695928317731480712357569372795\
15543419344057970899365859403176313951068268266882944649562100\
80903479818549199568459705562548422892115525746166751074289213\
609596618613446079618857135830766959762009927055865884710796501

   38678016645817756634752375727912689491895454119111249334573\
61154124904369189081436818709755749588026367101763117998307007\
77506345342165387799477766567321557421166007590816670669406236\
43773618649101682703783799537062586749082756929026484684965113\
373035085734095558864183533679809881862774879514342408924287321))



;; Here we encrypt with the public key and decrypt with the private key
;; we could have encrypted with the private key and decrypted with the
;; public key as well.

cl-user(13): (rsa-encrypt "my secret message" (car keys))
#(102 136 69 180 180 27 185 63 132 137 ...)


cl-user(14): (rsa-decrypt * (cadr keys) :string t)
"my secret message"
cl-user(15):

8.0 Support for gzip and bzip compression and decompression

The inflate and deflate modules allows you to compress data as it is written to files, and to open streams to files containing gzip compressed data and to uncompress the data while reading the file. To load the modules, evaluate (require :inflate) and (require :deflate). Symbols in the modules are in the util.zip package. (There are two modules because they were added at different times.)

The :deflate module requires that the libz library be available on your computer (and in the correct version). If you do not have that library or do not have the correct version, deflation will not work. Allegro CL functionality that uses deflation (such as AllegroServe, see aserve.html) will warn that they cannot compress data but will continue to work without data compression.

The inflation function util.zip:inflate can be applied to instances of the class util.zip:inflate-stream.

The various utility functions util.zip:skip-gzip-header, util.zip:skip-gzip-trailer, util.zip:skip-zlib-trailer, and util.zip:skip-zlib-trailer can be applied to an input stream to position the file position to the correct location for uncompressing and to avoid any trailers.

The deflation functions util.zip:deflate-target-stream, util.zip:deflate-stream-vector, and util.zip:deflate-stream-vector-combined can be applied to instances of the class util.zip:deflate-stream.

The value of the variable sys:*zlib-system-library* is the name of the gzip library used by the deflate module.

inflate-stream and related functions

inflate

function, package: util.zip

Arguments: input-stream output-stream

The compressed information from the input-stream is read and the uncompressed information is written to the output-stream.

Both streams must support (unsigned-byte 8) element reading and writing.

skip-gzip-header

function, package: util.zip

Arguments: input-stream

If the input stream is positioned on the header of a gzip'ed file then skip that header. input-stream is not an instance of the class util.zip:inflate-stream. It is simply an input stream (opened, for example, with open). The file position must be moved to the beginning of the compressed data before unziping, and only at that point should an instance of util.zip:inflate-stream be created (see util.zip:inflate-stream).

If the input stream is not positioned on a gzip header then nothing is done.

skip-gzip-trailer

function, package: util.zip

Arguments: input-stream

Skips past the next 8 bytes in input-stream. Note that gzip trailers have no byte-markers to identify them so care should be taken to only call this function after the final data block is read from the stream.

skip-zlib-header

function, package: util.zip

Arguments: input-stream

If input-stream is positioned at the header of a zlib'ed stream, then skip past it, returning the number of bytes read.

If input-stream is not positioned at a zlib header, return nil.

If input-stream appears to be positioned at a zlib header but turns out to not be, signal an error.

skip-zlib-trailer

function, package: util.zip

Arguments: input-stream

Skips past the next 4 bytes in input-stream. Note that zlib trailers have no byte-markers to identify them so care should be taken to only call this function after the final data block is read from the stream.

inflate-stream

Class, package: util.zip

The stream class for instances of files containing comressed gzip'ed data. Instances of this class are suitable as arguments to util.zip:inflate.

To create an instance of this class, do the following:

  1. Make a stream (using, e.g. open) with a data source which contains compressed data.

  2. If the this file may have a gzip header on it, apply util.zip:skip-gzip-header to the stream.

  3. Create an instance of util.zip:inflate-stream by evaluating:

    (make-instance 'inflate-stream :input-handle <stream created in 1>
       [:compression compress-spec])

    compress-spec can be one of the following three values:

    The call to make-instance will return a stream which can be read to recover the uncompressed data.

Closing the inflate-stream will also close the stream created in step 1.

deflate-stream and related functions

To write compressed data, you create a deflate-stream and at that time specify an ultimate taget, which is either a regular stream (perhaps open to a file) or a octet vector (of element type (unsigned-byte 8)). You then write data to the deflate-stream and that data is compressed and eventually written to the target. We say eventually because there is a lot of buffering so you do not see data in the target immediately. When you are done, you close the deflate-stream and that causes any remaining data to be written to the target.

Closing the deflate-stream does not close the target if it is a stream. You must close the target stream yourself.

You create deflate-streams with make-instance, as described next.

deflate-stream

Class, package: util.zip

The stream class for instances of deflation streams which accepts characters and bytes and causes them to be compressed and sent to a target.

You created a deflate-stream with make-instance. You must specify a target when you create a deflate-stream.

Here is a sample make-instance call:

(make-instance 'deflate-stream :target target-spec 
                               [:compression compress-spec])

target-spec can either another stream, or it can be a vector (that is an actual stream or an actual vector). The stream must be writable. The vector must have element-type (unsigned-byte 8).

compress-spec can be one of the following three values:

:gzip is the preferred format as the result can be uncompressed with the util.zip:inflate-stream (be sure to specify :skip-gzip-header t to the make-instance creating the inflate-stream or to call util.zip:skip-gzip-header after the input stream is opened). The :gzip format output can also be uncompressed with the gunzip program found on Unix.

A stream as the target-spec

If you pass a stream as the target-spec then as you write characters and bytes to the deflate-stream, the bytes resulting from deflation will be written to the given stream. There is a lot of buffering going on in this stream and the compression library. Therefore you may not see the results in your target-spec stream immediately.

When you close the deflate-stream the last bytes in all the buffers will be sent through deflation and the end of deflation record will be written to the target-spec stream.

Again, the target-spec stream will not be closed when the deflate-stream is closed. It is the callers responsibility to close the stream passed in as the target-spec.

The function util.zip:deflate-target-stream will return that target-spec stream used by the deflate-stream.

A vector as the target-spec

Passing a simple vector of type (unsigned-byte 8) as the target-spec is telling the deflate-stream that you wish to collect the deflation result in vectors in the lisp heap. The size of the vector passed is not important. Additional vectors will be created as necessary to hold data as it is written.

Once you have closed the deflate-stream after all data has been written to it, you can retrieve the result with util.zip:deflate-stream-vector or util.zip:deflate-stream-vector-combined, as described in the descriptions of those functions. The deflate-stream-vector-combined combines all results into a new vector if the target vector is not large enough. deflate-stream-vector returns the vectors created by writing to the deflate-stream, along with additional information.

deflate-target-stream

function, package: util.zip

Arguments: deflate-stream

Returns the stream which is the target of deflate-stream which must be a deflate-stream. Returns nil if deflate-stream has a vector as its target.

deflate-stream-vector

function, package: util.zip

Arguments: deflate-stream

deflate-stream must be a deflate-stream whose target is a vector. In that case, this function returns three values:

  1. The newest vector created so far by writing to deflate-stream (or the last such vector if deflate-stream is closed).

  2. The number of bytes of actual data in the newest vector.

  3. A list of previous vectors holding data in reverse order.

For example, if the three returns values are:

    v
    100
    (c b a)

then the deflated result is found by combining in this order:

    all of a
    all of b
    all of c
    the first 100 bytes of v

This function signals an error if deflate-stream has a stream as its target.

deflate-stream-vector-combined

function, package: util.zip

Arguments: deflate-stream

deflate-stream must be a deflate-stream whose target is a vector. In that case, this function returns two values:

  1. An octet vector.

  2. The number of bytes of actual data.

The octet vector is newly created if necessary (if the target vector specified when the deflate-stream was created is not large enough to hold the compressed data).

This function signals an error if deflate-stream has a stream as its target.

Example

Suppose we wish to create a compressed file foo.cl.gz from the following text:

;; file foo.cl begin
(in-package :user)

(defpackage :foo (:use :cl :excl))
(defun foo (y) (bar y))

;; file foo.cl end

In the following transcript, we use a deflate stream to create foo.cl.gz and then we inflate foo.cl.gz by opening the file, stripping the gzip header, creating an inflate-stream instance, reading the file line by line, closing the inflate-stream, and closing the file. We could also inflate the while file by calling inflate.

;; We load the modules and then write a compressed file.
;; Once it is written, we read it back uncompressing it.

cl-user(61) (require :deflate)
; [loading messages]
tcl-user(62) (require :inflate)
; [loading messages]
t
cl-user(63): (setq myfile (open "foo.cl.gz" :direction :output :if-exists :supersede))
#<file-simple-stream #P"foo.cl.gz" for output pos 0 @ #x100439f9f2>
cl-user(64): (setq *df* (make-instance 'deflate-stream :target myfile))
#<deflate-stream in 0 / out 0 @ #x10043ad922>
cl-user(65): (format *df* ";; file foo.cl begin~%~%")
nil
cl-user(66): (format *df* "(in-package :user)~%~%")
nil
cl-user(67): (format *df* "(defpackage :foo (:use :cl :excl))~%")
nil
cl-user(68): (format *df* "(defun foo (y) (bar y))~%~%")
nil
cl-user(69): (format *df* ";; file foo.cl end")
nil
cl-user(70): (close *df*)
#<deflate-stream in 120 / out 106 @ #x10043ad922>
cl-user(71): (close myfile)
t
cl-user(72): (setq s (open "foo.cl.gz" :direction :input))
#<file-simple-stream #P"foo.cl.gz" for input pos 0 @ #x1004400732>
cl-user(73): (util.zip:skip-gzip-header s)
10
cl-user(74): (setq is
              (make-instance 'util.zip:inflate-stream :input-handle s))
#<inflate-stream
  inflating #<file-simple-stream #P"foo.cl.gz" for input pos 10 @
              #x1004400732>ef :latin1-base, in: 0, inflated 0, used: 0 of
  @ #x1004417412>
cl-user(75): (read-line is nil s)
";; file foo.cl begin"
nil
cl-user(76): (read-line is nil s)
""
nil
cl-user(77): (read-line is nil s)
"(in-package :user)"
nil
cl-user(78): (read-line is nil s)
""
nil
cl-user(79): (read-line is nil s)
"(defpackage :foo (:use :cl :excl))"
nil
cl-user(80): (read-line is nil s)
"(defun foo (y) (bar y))"
nil
cl-user(81): (read-line is nil s)
""
nil
cl-user(82): (read-line is nil s)
";; file foo.cl end"
t
cl-user(83): (read-line is nil s)
#<file-simple-stream #P"foo.cl.gz" for input pos 98 @ #x1004400732>
t
cl-user(84): (close is) ;; this also closes s.

t
cl-user(85):

The source code to the gzip utility is included with the Allegro CL distribution, in [Allegro directory]/src/inflate.cl.

9.0 PAM support in Allegro Common Lisp

PAM stands for Pluggable Authentication Modules. It is a flexible mechanism for authenticating users. An Allegro CL module provides a Lisp wrapper around the PAM API on Linux, Solaris, and some other unixlike operating systems. We do not discuss PAM in detail here. See www.kernel.org/pub/linux/libs/pam/FAQ.

PAM is supported on the following platforms:

PAM is not supported in Allegro CL on the following platforms:

The PAM API is is loaded by evaluating (require :pam). Symbols naming functionality are in the util.pam package. Depending on your system configuration, your program may be required to run with 'root' privileges to successfully make use of PAM.

There is one class and several operators defined in the pam module. They are:

pam

Class, package: util.pam

The class of pam objects. A pam object is created by pam-start.

pam-start

function, package: util.pam

Arguments: service-name user &key conversation data

This function provides a wrapper around the pam_start(3) PAM library function. service-name should be a string naming the desired PAM service. user may be nil or a string.

If conversation is specified, it should be a function (or a symbol naming a function) which will perform the PAM "conversation", when necessary. data is optional user-defined data that will be passed to the conversation function. If conversation is nil (the default), a default conversation function will be used. For more information on conversation functions, see PAM conversation functions.

This function returns a pam object on success. You will pass this object to other functions and methods. If pam-start fails, an error is signalled.

pam-end

generic function, package: util.pam

Arguments: pam &optional status

The default method is analogous to the pam_end(3) PAM library function. pam must be a pam object returned by pam-start. If status is specified, it should be an integer. See the pam_end(3) description (in PAM documentation not supplied here) for details on the use of status. If status is nil, excl-osi:*pam-success* will be used.

This method returns t on success, otherwise it signals an error.

with-pam

macro, package: util.pam

Arguments: (var &rest rest) &rest body

with-pam is a convenience macro which evaluates body with var bound to the result of calling pam-start with the arguments specified in rest. pam-end will be called when body terminates, either normally or abnormally.

Sample use 

(util.pam:with-pam (pam "login" "jimmy")
  (format t "This is the body~%"))

pam-authenticate

generic function, package: util.pam

Arguments: pam &key flags password

This default method is analogous to the pam_authenticate(3) PAM library function. pam must be a pam object returned by pam-start. If password is specified, it should be a string. It will be used when needed if the default conversation (see pam-start) is used. If flags is specified, it should be an integer.

On success, this function returns t.

If the call is not successful, the function returns two values, nil and a status value. The status value will be a keyword or an integer. Possible status value keywords are:

If pam_authenticate(3) returns an unrecognized status code, it will be returned without being converted to a keyword.

set-pam-fail-delay

generic function, package: util.pam

Arguments: pam microseconds

This method is used to request a delay of at least the specified number of microseconds (which must be an integer) before returning from an unsuccessful pam-authenticate call. Setting a delay slows down attempts to rapidly try different passwords for an account.

If multiple calls to set-pam-fail-delay are made, the largest requested delay will be used. On some systems, it is possible that the PAM modules themselves may request delays, so you might notice a delay longer than one you requested (and in particular, you might notice a delay even though you hadn't called set-pam-fail-delay).

The actual delay will is computed pseudorandomly and may differ by as much as 25% above or below the maximum requested value.

Regardless of the success or failure of a pam-authenticate call, the delay is set back to 0 before returning from pam-authenticate. This means that you should generally call set-pam-fail-delay before each call to pam-authenticate.

9.1 PAM conversation functions

The default conversation function will display prompts and request input from *terminal-io*. If the password argument is supplied to pam-authenticate, it is probable that no interaction with *terminal-io* will occur at all. However, if your system configuration or application has different requirements, you can provide your own conversation function.

The conversation function will be called by the PAM API when it needs to collect information to move the authentication process along.

If you supply your own conversation function, it should accept two required arguments, and one keyword argument:

Arguments messages data &key password

messages will be a list of pam-message structures. data will be the same value that was supplied to pam-start. password will be the password that was passed to pam-authenticate (and may possibly be nil).

The conversation function should return a list of pam-response structures. The list must have the same length as the messages list. The first entry in the list should be the response that corresponds to the first message. The second entry should correspond to the second message, and so forth. See pam_conv(3) for details.

The pam-message structure has two slots, style and message. The style slot will be one of the following keywords (:prompt-echo-off, :prompt-echo-on, :error, :text) or an integer (indicating an unrecognized style). message will be a string which may be used to prompt the user.

The pam-response structure has two slots, response and code. response should be a string with the data requested by the corresponding message. code should be an integer (the default is 0). Again, see pam_conv(3) for details.

10.0 asdf support in Allegro CL

The popular system definition facility, asdf, is included with Allegro CL. Evaluate (require :asdf) to load it into a running Lisp. See [Allegro directory]/code/asdf.readme for more information, [Allegro directory]/code/asdf.license for the license, and [Allegro directory]/src/asdf.lisp for the source code. asdf documentation can be found on the web at http://constantly.at/lisp/asdf/.

11.0 tar support in Allegro CL

Allegro CL provides support for extracting contents from tar files. To use this facility, evaluate

(require :tar)

The tar functions are named by symbols exported from the util.tar package.

The tar functions take streams (rather than pathnames) as arguments. util.tar:list-tar lists the contents of the tar file. util.tar:extract-tar extracts the contents into a specified directory.

Here are example forms using the tar functions:

(with-open-file (s "foo.tgz")
  (util.tar:list-tar s :gzip t))
(with-open-file (s "foo.tgz")
  (util.tar:extract-tar s :gzip t :directory "tmp/"))

list-tar

function, package: util.tar

Arguments: stream &key gzip

List, to *terminal-io*, the contents of stream, which should be a stream opened to a tar file. If stream is compressed with gzip compression, specify a non-nil value for gzip. bzip2 compression is not supported.

extract-tar

function, package: util.tar

Arguments: stream &key gzip directory verbose

Extract, to directory, the contents of stream, which should be a stream opened to a tar file. If stream is compressed with gzip compression, specify a non-nil value for gzip. bzip2 compression is not supported.

directory defaults to nil, which means extract to the current directory (as returned by current-directory).

If verbose is specified non-nil, information about what is being done will be printed.

12.0 Coverage analysis in Allegro CL

When testing a program, the coverage is a measure of how much of the source code has actually been tested: have all branches of a conditional been taken, have all defined functions been called, have all error handlers been triggered, and so on (coverage is described in Wikipedia here).

Allegro CL has a macro, with-coverage, which executes code and when done, prints information about how well a specified list of functions are excercised. The type of coverage, following the description in the Wikipedia article linked to above, is statement coverage, where "lines of code" is understood to mean Lisp S-expressions.

See the with-coverage page for further details and an example. Note that source file recording must be on for the coverage tool to work.

13.0 Checking format forms during compilation

Allegro CL has a facility for checking format forms during compilation. What is checked is whether there seem to be sufficient arguments in the form for the format control string to be processed without error when the form is evaluated at run time. If enabled, the facility will warn when it detects that there are two few arguments, note when there are too many arguments, and note when the control string is too complex for analysis. (Having too many argument is not an error and is indeed explicitly permitted by the ANS. However, if it is not what is intended, it is useful to know that it has happened.)

The analysis will occur when the comp:verify-format-argument-count-switch is non-nil and not 0. This switch differs from most compiler switches in that its value can be 0, 1, 2, 3, or 4 as well as nil (equivalent to 0) and t (equivalent to 2).

Here is the behavior for the various values of the switch:

Statistics are only kept if the value of the variable *format-arg-count-stats* is a list of six fixnums (representing all, missing args, args equal in number to needed, args more than needed, too complex, and syntax error). These numbers are incremented appropriately as format forms are analysed. Warnings and compiler-notes are printed if the switch value calls for them. The function format-arg-count-stats prints the collected statistics.

14.0 Heap walkers in Allegro CL

A heap walker is a tool which examines the Lisp heap and gathers information about it. The Lisp heap is where most Lisp objects are stored. It comprises the old space and new spaces described in the garbage collection document gc.html.

The two built-in tools which walk the heap are get-objects and get-references. Each returns a heapwalk vector. This is a simple vector with element type t, whose size depends on the number of objects found. The first element of a heapwalk vector is the integer number of objects of interest found). These objects are stored in the vector elements after the first. The actual size of the vector is a little bit bigger than strictly necessary to allow for including objects created while the heap is being examined. Note that the heapwalk vector itself is an object in the heap. It is persumably not an object of interest (it did not even exist when get-objects or get-references was called) so its existence can skew the results slightly but unavoidably.

The function get-objects takes a code as its required argument and has keyword agguments old and new. The keyword arguments default to t. If specified nil, that particular space will not be searched. The code is the type code of a lisp object.

Type codes are shown in room output and also by the function print-type-counts. (Neither lists all possible type counts, just those for which at least one object of the particular type exists.) Here is a (slightly truncated) output of (room t):

cl-user(2): (room t)
area area  address(bytes)        cons         other bytes
  #  type                   8 bytes each
                             (free:used)      (free:used)
     Top #x20d7a000
     New #x20994000(4087808)   916:15225     843936:3035024
     New #x205ae000(4087808)    -----            -----
   0 Old #x20000aa0(5952864)   711:78771    2098888:3209704
Root pages: 126
  Lisp heap:    #x20000000  pos: #x20d7a000 resrve: #x20fa0000
Aclmalloc heap: #xa0000000  pos: #xa0048000 resrve: #xa00fa000
 Pure space:    #x1f8ec000  end: #x1ffff888

code   type                                 items     bytes 
126: (simple-array (unsigned-byte 16))       10767   2153400 31.4%
112: (simple-array t)                         8891   1445976 21.1%
  1: cons                                    93996    751968 11.0%
  7: symbol                                  20360    651520  9.5%
  8: function                                 9681    602864  8.8%
133: sv-vector                               20549    340840  5.0%
120: (simple-array fixnum)                     259    270272  3.9%
119: (simple-array code)                       367    192064  2.8%
117: (simple-array character)                 2396    148960  2.2%
125: (simple-array (unsigned-byte 8))           19     98720  1.4%
 12: standard-instance                        3900     62400  0.9%
  9: closure                                  2897     50432  0.7%
 15: structure                                1159     47856  0.7%
127: (simple-array (unsigned-byte 32))          11     12744  0.2%
[...]

 total bytes = 6856032
aclmalloc arena:
[...]

Now we call get-objects on type code 127, which is (simple-array (unsigned-byte 32)). There are eleven such objects, according to the list printed by room (code 127 is the last item before the list is truncated).

cl-user(3): (get-objects 127)
#(11 #(255 65535 16777215 4294967295) #(32 8224 2105376 538976288)
  #(3960924350 2165561044 562617442 2270225120 1264129478 758582028 172407450
    2782512936 595962478 1609675396 ...)
  #(0 2567483615)
  #(546914304 4087808 916 15225 529448960 536869000 553254912 536870912 843936
    3035024 ...)
  #(3960924350 2165561044 562617442 2270225120 1264129478 758582028 172407450
    2782512936 595962478 1609675396 ...)
  #(3960924350 2165561044 562617442 2270225120 1264129478 758582028 172407450
    2782512936 595962478 1609675396 ...)
  #(200235464 1375865383 2472949741 3729159793 443451277 421802134 4188904507
    2175392005 408067652 1254986169 ...)
  #(3960924350 2165561044 562617442 2270225120 1264129478 758582028 172407450
    2782512936 595962478 1609675396 ...)
  ...)
cl-user(4): (length *)
31
cl-user(5):

The result is a heapwalk vector. The first element is 11, the number of objects of the type of interest. Then those eleven objects. Then some extra elements (19 extra in this case, for a total vector size of 31).

Once you have objects of a particular type, you can find references to that object with get-references, which returns a heapwalk vector filled with objects in the Lisp heap or in lispstatic space which point directly to the object of interest. Note that compound structures might not point to the object directly; for example, if a list has an element within it, the first call to get-references will return only the cons cell whose car is that object; you would have to repeat the get-references on that cons cell to find the cons whose cdr points to it, and so on backward through the list until you find a recognizable structure, symbol, or function that points to the head of the list. The stack can be included in the search if the include-stacks keyword argument to get-references is specified true.

Heapwalker vectors themselves point to lots of objects but are not what you are looking for when you search for references. Therefore, heapwalk vectors are marked and are not included in get-references output.

When using get-references, note the following:

15.0 The JSON-RPC API

The JSON-RPC module depends on the ST-JSON open source module. A compiled version of the module, the file st-json.fasl is included with the Allegro CL distribution and can be loaded with (require :st-json). You can download the sources and documentation from github at https://github.com/marijnh/ST-JSON. ST-JSON is a 'Common Lisp library for encoding and decoding JSON values (as specified on json.org)' (quoting from the ST-JSON home page). In what follows, we assume that module has been downloaded and loaded in Allegro CL. The symbols in the ST-JSON module (useful for manipulation JSON objects) are in the st-json package.

JSON-RPC is lightweight remote procedure call protocol similar to XML-RPC. (See xml-rpc.html for information on the XMP RPC interface in Allegro CL.) The JSON-RPC module in Allegro CL provides an API within Lisp to JSON-RPC.

Package and module loading

Symbols naming functionality in the module are in the net.json.rpc package. You load the module with the following require form:

   (require :json-rpc)

The JSON-RPC module is initialized with the following function.

init-json-rpc

function, package: net.json.rpc

Arguments: &key client server converter methods global

This function initializes various JSON-RPC features. The various keyword arguments are described in later sections as they apply to client or server operations.

The converter keyword must be omitted, nil, :default, or a function of one argument. If omitted or nil, there is no effect.

When a function is specified, it must be a function that transforms a string (denoting a JSON method or member name) to a corresponding Lisp symbol, or Lisp symbol to a corresponding JSON name string. This function is used in all cases where derived-symbol or derived-string is specified.

The value :default specifies a built-in function with the following behavior. This is the initial setting as well.

Convert a string to a symbol or a symbol to a string using simple case and hyphenation ruled:

       "fooBarBaz"  <-->  foo-bar-baz

This function will try to signal a warning when troublesome cases are found.

For servers

A non-nil server keyword argument specifies a default transport protocol for JSON-RPC servers. If the global argument is nil, the only effect is to perform any initialization required for the transport protocol (such as loading a required fasl or library). If the global argument is non-nil (the default) then any running JSON-RPC servers are stopped.

When the methods keyword argument is non-nil, all current JSON-RPC method definitions are deleted, and any desired methods must be defined again by evaluating or loading the defining forms.

For clients:

A non-nil client argument specifies a default destination for client calls. If the global argument is nil, the only effect is to perform any initialization required for the transport protocol (such as loading a required fasl or library). If the global argument is non-nil, the value of [*json-rpc-destination*(#s_json-rpc-destination_s-varxxx) is updated.

15.1 JSON-RPC server operators and variables

def-json-rpc-method

macro, package: net.json.rpc

Arguments: name bv &body body

Defines a Lisp function that can be dispatched as the destination of a JSON-RPC message to the server. The arguments are:

name: the value can be

options: options may only be specified using the list value for name. The options can name a set or sets in which to include the method. If no set is named, the method is part of all sets.

The bv argument: the value should be a list of one of the following forms:

([var]... [&rest tailvar] [&whole listvar])

The "params" member in the JSON message must be a JSON array. Each var specifies a required element in the array, and is bound in body to the corresponding array element.

If &rest is present, additional arguments are allowed in the array, and the list is bound to this tailvar. By-position methods are dispatched by matching all methods without &rest first, then all methods with &rest. All methods are matched in order of decreasing number of required args.

(&key [label]... [&allow-other-keys] [&whole wholevar])

The "params" member in the JSON message must be a JSON object. Here:

      label --> (var member-name)
        --> symbol == (symbol derived-string)
        --> string == (derived-symbol string)

Each var is a keyword argument bound in body to the named member of the object. If &allow-other-keys is present, additional members are ignored; they are accessed by the member name from the pointer bound to wholevar.

By-name methods are dispatched by matching methods without &allow-other-keys first.

In each group, the methods are ordered by decreasing number of required members. The member names are sorted lexicographically and the signatures are sorted lexicographically within each set of equal length.

The dispatch strategy may cause some methods to be unreachable. Some unreachable methods may be detected at compile time, others may not.

body returns one value, a JSON object or a Lisp object that is converted to JSON by default rules, or the body signals an error that is returned as a JSON error object.

The same JSON method name may apply to several actual methods with distinct signatures; the Lisp names of these methods must be distinct (therefore only one can be a derived name).

*json-rpc-redefinition-action*

Constant, package: net.json.rpc

This variable specifies the behavior when def-json-rpc-method redefines a JSON-RPC method or a one of the Lisp functions associated with the method. The value can be:

The initial value is :warn.

start-json-rpc-server

function, package: net.json.rpc

Arguments: &key transport export receive send &allow-other-keys

Start a JSON-RPC server, or enhance an existing server with JSON-RPC features.

The transport argument specifies how JSON-RPC messages will arrive. The format of the argument depends on the nature of the transport.

The inital implementation supports a simple stream or a web server running AllegroServe. The default for this argument can be specified with init-json-rpc. Additional arguments are interpreted by specific transport implementations.

The receive argument must be a list that specifies the JSON-RPC versions accepted by the server. The first entry in the list is the default reply version. The initial value is (:2.0 :1.0 :batch). The entry :batch specifies that batch requests will be honored.

The send argument specifies the version of reply messages. The value :same specifies that the reply message will be the same version as the request message.

The export argument is a string or symbol or a list of strings and/or symbols. A symbol denotes a Lisp function defined with def-json-rpc-method, or a set of methods specified with the :rpc-set option; if a symbol denotes both a set and a function, the set definition prevails. A string denotes a JSON-RPC method (in that case all Lisp names are included).

This function returns an object that can be used as the argument to stop-json-rpc-server.

Simple stream transport:

The transport argument must be :stream or a list (:stream stream-instance). If the argument is :stream, then a stream-instance must be specified with a :stream keyword argument. The stream-instance must be subclass of dual-channel-simple-stream.

AllegroServe transport:

The transport argument must be :aserve or a list (:aserve . aserve-start-options).

Additional transport protocols may be defined in the future.

stop-json-rpc-server

function, package:

Arguments: &key server

Stop a server. The server argument should be the return value of the call to start-json-rpc-server that started the server.

See the description of init-json-rpc for information on initializing JSON-RPC for servers. A non-nil server keyword argument to that function specifies a default transport protocol for JSON-RPC servers. If the global argument is nil, the only effect is to perform any initialization required for the transport protocol (such as loading a required fasl or library). If the global argument is non-nil (the default) then any running JSON-RPC servers are stopped.

When the methods keyword argument to that function is non-nil, all current JSON-RPC method definitions are deleted, and any desired methods must be defined again by evaluating or loading the defining forms.

15.2 JSON-RPC client operators and variables

*json-rpc-destination*

variable, package: net.json.rpc

A destination where JSON-RPC call messages can be sent, such as a URL.

*json-rpc-call-version*

Variable, package: net.json.rpc

The default version for method calls from a client. The initial value is :2.0.

call-json-rpc-method

function, package: net.json.rpc

Arguments: method-name json-parameters &key notification id destination version &allow-other-keys

Call the server at destination with a JSON-RPC message where method-name is the method member and json-parameters is the parameters member.

Default destination is *json-rpc-destination*. Default version is *json-rpc-call-version*.

The id argument is used only in the scope of a with-json-rpc-batch expression. It specifies an id value that can be used in the call to json-rpc-batch-result.

Additional keywords are interpreted or ignored by methods specific to a particular destination.

Within the call body of a with-json-rpc-batch, the returned value is the unique JSON-RPC call id. In other contexts, the returned value is the JSON-RPC result. When notification is non-nil, no value is returned in either case.

def-json-rpc-call

macro, package: net.json.rpc

Arguments: name bv &key destination version notification id

This macro defines a Lisp function that will make a JSON-RPC call to a JSON-RPC method.

name: the value can be

The bv argument: the value should be a list of one of the following forms:

([var]... [&rest rvar])

The Lisp function expects arguments that get sent as a single JSON array. The argument names are ignored.

(&key label...  [&optional label...])

The Lisp function expects keyword arguments that are sent as members of a JSON object. The labels before &optional specify required keyword args. The trailing labels specify optional keyword args. (We ask the reader to excuse this deviant usage of the &optional marker)

      label --> (keyword-name member-name)
        --> symbol == (symbol derived-string)
        --> string == (derived-symbol string)

with-json-rpc-batch

macro, package: net.json.rpc

Arguments: bv options callbody &rest body

Call several JSON-RPC methods as a single batch call. The options are passed to methods specific to the destination.

The variables in bv are bound and visible in callbody and in body.

The callbody expression is evaluated first for its side-effects, and then any values are discarded. Any JSON-RPC calls during this evaluation add to the content of a batch; the calls themselves do not return a JSON result, instead, they return the unique JSON-RPC id of the call in the batch. The calls in the batch are sent when the progn exits.

The expressions in body are evaluated as a progn after the result of the batch call have arrived. Within body, the function json-rpc-batch-result can be called to extract the result of a specific call.

json-rpc-batch-result

function, package: net.json.rpc

Arguments: id

When called in the body of a with-json-rpc-batch expression, this function returns the result of the request with the specified id. The id argument can be the unique id returned in the callbody of the with-json-rpc-batch expression, or the local id specified in the JSON-RPC call.

When init-json-rpc is called with a non-nil client argument, it specifies a default destination for client calls. If the global argument is nil, the only effect is to perform any initialization required for the transport protocol (such as loading a required fasl or library). If the global argument is non-nil, the value of *json-rpc-destination* is updated.

15.3 JSON-RPC built-in stream transport

Application program must supply a stream to the server or the client. The stream must be a dual-channel-simple-stream instance such as a socket or a pipe stream.

In a call to start-json-rpc-server, the value of the transport keyword argument must be :stream or a list (:stream stream). An additional stream keyword argument is accepted (so the stream to be used may be part of a list value of transport or separately specified as the value of stream argument). The stream must be a dual-channel-simple-stream instance.

In a call to call-json-rpc-method, the value of the destination argument must be a stream.

15.4 JSON-RPC built-in allegroserve transport

To use the JSON-RPC module with AllegroServe, you must load the :json-rpc-aserve module:

(require :json-rpc-aserve)

In calls to start-json-rpc-server, the transport argument must be either :aserve or the dotted list (:aserve . start-arguments) or a net.aserve:wserver instance.

Additional accepted keyword arguments include:

:url "/json-rpc"   
:content-type "text/html"
:publish additional-publish-args

In a call to call-json-rpc-method, the value of the destination argument must be a uri string or a net.uri:uri instance.

15.5 JSON-RPC: how to add a new transport method

The steps for adding a new transport method are:

  1. Choose a new keyword to identify the method.

  2. Define a subclass of json-rpc-server.

  3. Push the entry (keyword class-name) onto *json-rpc-transports*.

  4. When the method is loaded, maybe setq *json-rpc-transport* to the keyword (or a list starting with the keyword). This is used as the default transport argument to start-json-rpc-server.

  5. Define a method for start-json-rpc-server-implementation specialized on the new server class, to initialize the server and instance. The server calls json-rpc-dispatch when it has parsed a complete JSON-RPC request. The result is nil, or a fully formed JSON-RPC reply message. If the result is nil, the message was a notification or an array consisting entirely of notifications; in that case the server does nothing. If the result is not nil, the server must send the reply message to the client; the ST-JSON package includes several methods to serialize a JSON object for transmission if the transport method uses characters.

  6. Define a method for json-rpc-stop-transport.

  7. Define a distinct type for client destination. Reserved types are string, uri, and stream.

  8. Define a method for json-rpc-send-message specialized on that type.

As an example, this very trivial implementation assumes the client and server are in the same address space:


(defclass local-server (json-rpc-server) ())
(push (list :local 'local-server) *json-rpc-transports*)

(defmethod start-json-rpc-server-implementation 
     ((server local-server) &key &allow-other-keys)
   ())

(defmethod json-rpc-send-message ((server local-server) message notification)
   (declare (ignore notification))
   (json-rpc-dispatch server message)

Here are the descriptions of some of the needed functionality.

json-rpc-server

Class, package: net.json.rpc

This class used in defining a new transport method. This class must be a superclass of any new transport server.

*json-rpc-transports*

Variable, package: net.json.rpc

This variable is used in defining a new transport method. The value of this variable is a list of the form

((transport-keyword name-of-server-class) ... )

Additional entry formats may be defined in the future.

*json-rpc-transport*

Variable, package: net.json.rpc

This variable is used in defining a new transport method. The value of this variable is the default transoprt method for JSON-RPC calls. This must be a transport name keyword or a list beginning with such a keyword. The tail of the list contains additional transport-specific arguments.

start-json-rpc-server-implementation

generic-function, package: net.json.rpc

Arguments: server &rest options &allow-other-keys

This method must be implemented when defining a new transport method. Keyword arguments provide transport-specific inital values. This method is called after the server instance is created and initialized at the json-rpc-server level.

json-rpc-stop-transport

generic-function, package: net.json.rpc

Arguments: server

This method must be implemented when defining a new transport method. This method is called before any generic json-rpc-server shutdown operations are performed.

json-rpc-send-message

generic-function, package: net.json.rpc

Arguments: destination message notification &rest options &key &allow-other-keys

This method, specialized on a new destination type, must be implemented when defining a new transport method. The message argument is a jso instance formatted for the specified JSON-RPC version. The notification argument is t for a notification or for a batch consiting entirely of notifications.

The options argument will contain all the options specified in the call to call-json-rpc-method.

json-rpc-dispatch

function, package: net.json.rpc

Arguments: server message

This function is called from the body of a start-json-rpc-server-implementation method for a new transport method. The server argument is the server argument passed to this method, and the message argument is a parsed JSON-RPC message received by the server.

This function returns nil if the message was a notification or an array consisting entierly of notifications. Otherwise, this function returns a JSON-RPC reply object that contains the result(s) of the message.

15.6 JSON-RPC: examples

To call a method with positional args to add two numbers:

;; Server def:

(def-json-rpc-method "adder" (x y) (+ x y))

;; Client call:

(call-json-rpc-method "adder" (list 2 5)) 
;; ==> 7

The client can also define a Lisp function that makes this call

(def-json-rpc-call (json-add "adder") (x y))

(json-add 2 5) 
;;  ==> 7

To call a json-rpc method with named args:

;; Server def:

(def-json-rpc-method "combineMembers" (&key (p1 "part1") (p2 "part2"))
    (jso "combined" (append p1 p2)))

;; Client call:

(setq r (call-json-rpc-method "combineMembers"
     (jso "part1" (list 2 4) "part2" (list 6 8))))
;;  ==> jso instance
(getjso "combined" r) 
;;  ==> (2 4 6 8)

;; or

(def-json-rpc-call (jappend "combine-members")
        (&key (m1 "part1") (m2 "part2")))

(setq r (jappend :m1 (list 2 4) :m2 (list 6 8)))
(getjso "combined" r) 
;; ==> (2 4 6 8)

15.7 JSON-RPC: functionality index

16.0 Websocket API

The websocket module implements the websocket protocol specified in RFC 6455. This document describes the Lisp functionality that support this specification; it is not a websocket tutorial by any measure. Many websocket tutorials are available on the web; the Mozilla Firefox tutorials at https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API is one starting point.

This module implements both a server and a client API. Extensions are not implemented at this time, but protocol and extension negotiation is performed in the initial handshake, and extension data are parsed and transmitted correctly in websocket message payloads.

The module is included in a running Lisp with the expression

(require :websocket)

The module requires the AllegroServe component and includes the necessary calls to require. Most of the documented symbols are in the net.aserve package. Symbols specific to the client API are in the net.aserve.client package.

16.1 Websocket threading behavior

On the server side, all websocket application code runs in one thread (that is, in a single Allegro CL process), which is the aserve worker thread associated with the current websocket connection. The application program may initiate other threads but these are all the responsibility of the application.

Messages are processed sequentially in the order in which they arrive through the socket. When a message arrives, the corresponding on-function (see Websocket event handlers) is called. The next message is seen and handled after the function returns. All the send functions simply make entries in a send queue; the send queue is processed whenever an on-function returns, and when there is no input immediately available at the connection. The message queue may also be handled when websocket-join is called.

If application code starts other threads, these run in parallel and must interract with the main server thread accordingly. The function websocket-join provides one built-in means of synchronizing behavior with the main thread. Other standard synchronization tools are gates, locks, barriers, and semaphores.

If responsiveness to messages is important in an aplication, it should not perform time consuming operations in the on-functions since that will delay the handling of subsequent messages and the sending of messages in the output queue.

On the client side, the behavior is managed by the return argument to open-websocket. When the return argument is nil, the behavior is very similar to the server behavior. The thread calling open-websocket becomes the main thread where all operations takes place, unless some other threads are started by the application.

If open-websocket is called with a return argument of t, the main websocket thread is a new thread started by the call and the calling thread continues as a parallel application thread.

16.2 Websocket server functionality

The main server operator is publish-websocket.

publish-websocket

function, package: net.aserve

Arguments: path host port content-type format server authorizer timeout hook headers remove on-open on-message on-close on-error on-ping on-pong on-idle on-fragment (class 'net.aserve:websocket-message-entity) contract-class plist protocols extensions origins debug ply close-timeout fragment-size ping-interval

This function enables a websocket service in an AllegroServe web server. It has many keyword arguments, which are divided into three groups:

Group 1: Arguments that are passed unchanged to the AllegroServe publish function

See the description of the AllegroServe publish function in aserve.html for details on these arguments.

Group 2: This group of arguments specify event handlers for messages

The value of these arguments must be nil or an appropriate function which accepts the specified number of arguments. See Websocket event handlers for further information. The names of these arguments are: on-open, on-message, on-close, on-error, on-ping, on-pong, on-idle, and on-fragment.

Group 3: Other arguments specific to this function

websocket-published-plist

generic-function, package: net.aserve

Arguments: contract

This generic function can be specialized on the contract argument. A method is predefined on the class websocket-message-server-contract. That method returns the value of the plist slot initialized by the plist argument to publish-websocket. This value is typically used with getf and (setf getf) to fetch and modify values specific to an application. Any changes are seen by all connections to this endpoint.

16.3 Websocket client functionality

Symbols naming websocket client functionality are in the net.aserve.client package.

open-websocket

function, package: net.aserve.client

Arguments: url & key cookies proxy proxy-basic-authorization basic-authorization digest-authorization query user-agent ssl on-open on-message on-close on-error on-ping on-pong on-idle on-fragment headers class ply protocols extensions (wait *default-open-websocket-wait*) plist close-timeout fragment-size ping-interval (return t) (retry *default-open-websocket-retry*) (error-p *default-open-websocket-error-p*)

This function opens a websocket connection to a server to enable sending and receiving websocket messages. The function operates in two modes depending on the return argument.

If the value of the return argument is true (which is the default value if omitted) the function returns as soon as the websocket connection is established; the returned value is a websocket-contract instance that is used subsequently to specify operations on the websocket; the actual websocket messages are handled in a separate thread (Lisp process) created for that purpose.

If the value of the return argument is nil, the function returns only after the websocket connection is closed; all message handling takes place in the thread that called open-websocket. Two values are returned, the numeric close code and the string included in the close message if any.

If the websocket connection cannot be established, the function returns immediately regardless of the return argument. The result is controlled by the error-p argument. If true (the default value), an error is signalled. If the error-p argument is nil, two values are returned, nil and a second value describing the reason for the failed connection.

The required argument to this function is url. This argument is a string containing the URL of the desired websocket server endpoint. The URL must specify the websocket scheme "ws" or "wss".

The keyword arguments to this function are divided into four groups:

  1. Arguments passed unchanged to the aserve function make-http-client-request. See here.

  2. Event handling arguments. See here.

  3. Arguments specific to this function. See here.

  4. Arguments that are the same as arguments to publish-websocket. See here.

Group 1: arguments passed to make-http-client-request

This first argument group, which includes the following arguments, are passed unchanged to the AllegroServe function net.aserve-client:make-http-client-request:

Group 2: arguments which specify event handlers

The second group of arguments are event handlers. The value of these arguments must be nil or an appropriate function which accepts the specified number of arguments. See Websocket event handlers for further information. The names of these arguments are: on-open, on-message, on-close, on-error, on-ping, on-pong, on-idle, and on-fragment.

Group 3: argument specific to this function

Group 4: arguments identical to publish-websocket arguments

These final arguments are the same as the identically named arguments to publish-websocket. See that description for details. For each argument, there is a link to the description in the publish-websocket documentation.

close-websocket

generic-function, package: net.aserve.client

Arguments: contract &key code message wait

This generic function is only defined for client applications. In most cases it must be called by the application to close and cleanup the websocket and underlying connection. When open-websocket is called with :return nil, these steps are automatically performed before returning to the caller; consequently, an application running in this mode should not call close-websocket. In general, any code running in an on-message function also must not call close-websocket, and should instead call websocket-close to signal the start of the steps to finally close a websocket.

This generic function can be specialized on the contract argument. A method is predefined on the class websocket-message-client-contract. This method begins and possibly waits for the completion of a closing handshake on the websocket connection specified by contract.

The keyword arguments are:

On successful completion, the method returns two values: the status code sent by the server and a string containing the message text if any.

16.4 Websocket event handlers

Event handlers are application functions (written by the application programmer) called by a websocket implementation when specified messages arrive at the endpoint. The functions are specified by the corresponding keyword arguments to publish-websocket and open-websocket.

The first argument is in all cases the instance of websocket-message-contract specific to the current websocket connection.

on-message

function, package: net.aserve

Arguments: contract data ext

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-message) or the associated function object should be passed as the value of the on-message keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when an endpoint receives a complete message. When the message is a text message, the data and ext arguments are nil or strings.

When the message is a binary message, the data and ext arguments are nil or arrays of octets.

on-fragment

function, package: net.aserve

Arguments: contract message-contract in-message ftype appdata ext

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-fragment) or the associated function object should be passed as the value of the on-fragment keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

When this function is specified, it is called for every message fragment.

If the function returns nil, it must return nil for every fragment in a single message, and the fragments are collected and handled as if no function was specified. The function is used simply to note the passing of fragments. The on-message function is called after the final fragment has been collected.

If the function returns a non-nil value, it must return a non-nil value for every fragment in a single message. The function is responsible for collecting and handling the content of the message. The on-message function is not called for that message.

If the on-fragment function returns an inconsistent sequence of values, the results are unspecified.

The in-message argument is one of the keywords :text or :binary.

The ftype argument is one of the keywords :only, :first, :continuation, or :final.

The appdata and ext arguments are always nil or arrays of (unsigned-byte 8) elements. Note that when the ftype is :text, these arrays may begin or end with incomplete UTF-8 multi-octet characters.

on-close

function, package: net.aserve

Arguments: contract code data-string

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-close) or the associated function object should be passed as the value of the on-close keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when a close message is received.

on-ping

function, package: net.aserve

Arguments: contract data-usb8

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-ping) or the associated function object should be passed as the value of the on-ping keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when a ping message is received and is called before the required pong response is sent, and its behavior does not affect the standard response.

on-pong

function, package: net.aserve

Arguments: contract data-usb8

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-pong) or the associated function object should be passed as the value of the on-pong keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when a pong message is received.

on-error

function, package: net.aserve

Arguments: contract code message

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-error) or the associated function object should be passed as the value of the on-error keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when an endpoint detects an error that causes an attempt to send an abnormal termination message. The message may or may not actually be sent.

on-open

function, package: net.aserve

Arguments: contract

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-open) or the associated function object should be passed as the value of the on-open keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when a connection is first opened after a successful handshake.

In a client application where open-websocket is called with ply value nil or 0, this is the where the application would normally begin operation on the client side (unless it expected an unsolicited server message instead).

on-idle

function, package: net.aserve

Arguments: contract

This function is not already defined. The application writer must define a function which accepts the arguments listed and returns values as specified in the function description below. The symbol naming the function (which need not be on-idle) or the associated function object should be passed as the value of the on-idle keyword argument to publish-websocket and/or open-websocket (different functions can be specified for each).

This function is called when the websocket handler is setup to handle receiving and sending in the same thread. The function is called when there is no incoming message and the send queue is empty.

If the function returns nil, the handler enters an operating system wait state waiting for input from the other endpoint. If the function return a non-nil value, the handler polls for input or queued messages to send.

16.5 Websocket general functionality

Symbols naming general websocket functionality are in the net.aserve package.

websocket-send

generic-function, package: net.aserve

Arguments: contract message &key ext fragment-size

There is a predefined method on this generic function specialized on (websocket-message-contract t):

That method sends a websocket message to the other endpoint and returns the keyword :text or :binary depending on the message type.

The contract argument must be an open websocket connection instance.

If the message argument is a string, the message type is :text. If the message argument is an array of octets, the message type is :binary. Any other type causes an error to be signaled.

The ext argument when specified must be the same type as the message argument. The data is included in the message payload and will be presented as extension data to the receiver. The current implementation in Allegro CL does not support any named extensions.

The fragment-size argument if specified overrides the fragment-size default for the connection. Messages above this size will be sent in fragments of approximately the specified size. If the ext argument is specified, any fragment-size specification is ignored; the message is always sent in a single payload.

websocket-ping

generic-function, package: net.aserve

Arguments: contract &key message

The predefined method specialized on websocket-message-contract does the following:

The method sends a ping message to the other endpoint. The message argument must be a string or nil.

websocket-pong

generic-function, package: net.aserve

Arguments: contract &key message

The predefined method specialized on websocket-message-contract does the following:

The method sends a pong message to the other endpoint. The message argument must be a string or nil.

websocket-close

generic-function, package: net.aserve

Arguments: contract &key code message

This generic function is separate and distinct from the generic function close-websocket, which is intended for client applications only. This generic functions works in both client and server code.

The predefined method specialized on (websocket-message-contract t) does the following:

This method sends a close message to the other endpoint. If a code is specified it should be a valid websocket close code. The message argument must be a string or nil.

Once the close message is sent, subsequent send calls are ignored.

websocket-state

generic-function, package: net.aserve

Arguments: contract &optional state wait err-if-never

The predefined method specialized on websocket-message-contract does the following:

If the state argument is not specified, or nil, the other arguments are ignored, the state of the websocket contract is examined, and one the following values is returned:

When the state argument is specified and non-nil, This function returns the state argument if the current state is consistent with the specified state, or return nil if not.

If the wait argument is non-nil, verify also that the desired state can possibly be reached; if not, the behavior is specified by the err-if-never argument. If that argument is true, signal an error of type websocket-unreachable-state; otherwise return nil.

If the wait argument is zero, perform the test and return immediately.

If the wait argument is a positive integer, wait for at most that many seconds or until the state is reached. If the state becomes unreachable, take the specified unreachable state action.

If the wait argument is any other value, wait indefinitely, or until the state becomes unreachable.

The state argument may be one of the keywords:

websocket-protocol

generic-function, package: net.aserve

Arguments: contract

The predefined method specialized on websocket-message-contract does the following:

Returns a symbol or a string naming the protocol negotiated for the connection idenified by contract. Returns nil if the negotiated result is the unmodified websocket protocol.

websocket-extensions

generic-function, package: net.aserve

Arguments: contract

The predefined method specialized on websocket-message-contract does the following:

this method returns a list of strings or symbols naming the extensions negotiated for this connection.

websocket-send-fragment

generic-function, package: net.aserve

Arguments: contract fq &key begin final data ext

The predefined method specialized on (websocket-message-contract t) does the following:

The method allows an application to exert more control over the fragmentation of a message. Each call prepares a separate message fragment. Any number of fragments can be prepared before any sending takes place. As soon as the first fragment is sent, no other messages will be sent (except for ping or pong) until the final fragment is sent.

The fq argument must be nil for the first call of a fragmented message. The value returned is an opaque handle used as the fq argument in subsequent calls.

The keyword arguments have the following effect:

When fq is nil, return new fq, required in subsequent calls.

The data and ext arguments are as in websocket-send. Note that this function allows an application to scatter extension data among fragments.

This method may be called with a nil value for data and begin specified as t.

websocket-contract-plist

generic-function, package: net.aserve

Arguments: contract

This generic function can be specialized on the contract argument. A method is predefined on the class websocket-contract. That method returns the value of the plist slot in the contract instance.

In server code this slot is initialized to nil. In client code it is initialized with the plist argument to open-websocket.

This value is typically used with getf and (setf getf) to fetch and modify values specific to an application. Any changes are seen only by one connection instance.

websocket-join

generic-function, package: net.aserve

Arguments: contract handle &key wait

This function waits until until a message has been sent or until some time passes. It returns t if the message has been sent or nil otherwise. Note that sending a message does not necessarily mean that it has been received.

The handle argument must be the value returned by a previous send call with the :track argument.

If the wait argument is nil, negative or zero, the function returns immediately after testing whether the message was sent or not. If the wait arguement is a positive number, wait at most that many seconds. If it is any other value, wait indefinitely.

If the websocket reaches the :not-sending state at any time, the function returns nil as well.

websocket-close-state

generic-function, package: net.aserve

Arguments: contract

If the websocket is closed, return a list of two elements, the close code and the close message string. Otherwise return nil.

16.6 Websocket classes

Websocket classes are named by symbols in the net.aserve package.

websocket-unreachable-state

Class, package: net.aserve

A condition class, a subclass of simple-error, this condition is signaled if websocket-state is called but the requested state becomes unreachable.

websocket-connect-error

Class, package: net.aserve

A condition class, a subclass of simple-error, this condition is signaled if open-websocket establishes a connection to a server but the connection is terminated because of protocol mismatch or error. There is no reason to retry such a connection unless some calling parameter is changed.

websocket-connect-failure

Class, package: net.aserve

A condition class, a subclass of simple-error, this condition is signaled if the connection attempt during a call to open-websocket fails for some network or socket related reason. It may be reasonable to retry the operation in the event of this kind of failure.

websocket-entity

Class, package: net.aserve

The superclass of any websocket implementation for AllegroServe. Currently the only subclass is websocket-message-entity.

websocket-message-entity

Class, package: net.aserve

This class implements the server part of the websocket protocol specified in RFC 6455.

websocket-contract

Class, package: net.aserve

The superclass on any any websocket contract classes. Currently the only subclass is websocket-message-contract.

websocket-message-contract

Class, package: net.aserve

An instance of this superclass is the opaque handle for all websocket operations in both server and client applications. This class implements the websocket protocol specified in RFC 6455.

websocket-message-client-contract

Class, package: net.aserve

The subclass of websocket-message-contract used by client code.

websocket-message-server-contract

Class, package: net.aserve

The subclass of websocket-message-contract used by server code.

16.7 Websocket variables

*default-open-websocket-wait*

Variable, package: net.aserve

Default value for the wait argument to open-websocket. The initial value is t.

*default-open-websocket-retry*

Variable, package: net.aserve

Default value for the retry argument to open-websocket. The initial value is nil.

*default-open-websocket-error-p*

Variable, package: net.aserve

Default value for the error-p argument to open-websocket. The initial value is t.

16.8 Websocket API index

All operators and classes associated with the websocket API are listed below.

17.0 The multiprocessing logging (mplog) facility

In a multi-threaded application it can be difficult to trace the sequence of events leading to an unexpected outcome. This is especially true in an SMP lisp. Allegro CL provides a very low-level log of multiprocessing events, including arrival of interrupts, starting and exiting of threads, opening and closing gates, garbage collections and socket events, that can sometimes be helpful in these situations.

Logging is not enabled by default. You start (and stop) logging with the mplog:recording function. The log is maintained in an internal memory buffer of fixed and limited size, which can hold approximately one thousand entries. When it fills up, new entries push the oldest ones out.

You often want to examine a log while debugging a misbehaving program, but because the running Lisp is stopped in the debugger, you cannot execute the calls which return log information. A feature of the facility allows you to store the log in a shared memory buffer so a separate Lisp image can be used to examine it. See mplog:share.

On Windows only, it is possible to construct a log from information available in debuggers. See Constructing mplog files from the debugger on Windows for more information.

Package and module information

The mplog functionality is in the :mplog module. To load it, evaluate

(require :mplog)

Operators are named by exported symbols in the :mplog package except the function sys:mplog is named by a symbol in the system package.

mplog operators

recording

function, package: mplog

Arguments: on-or-off &optional include-process-events

on-or-off can be :prof, another non-nil value, or nil. If non-nil but not :prof, enables mplogging. When :prof all events in SMP lisps will be ignored except ones dealing with the profiler (the value :prof is just like any other non-nil value in non-SMP Lisps). When nil, mplogging is disabled.

If include-process-events is specified true, high-level process events like creation and termination, are recorded. If nil (the default) such events are not recorded.

As said in the introduction above, the mplog is stored in an internal buffer of fixed size, holding about 1000 entries. Older entries are dropped when the buffer fills up and new entries are added. Entries can be viewed with mplog:show. On Linux, it is possible for one running Lisp to view the entries for another, a useful feature when an image is stopped for debugging and so cannot itself call mplog:show: see mplog:share.

show

function, package: mplog

Arguments: &key read write format

This function formats the data in an mplog. The keyword arguments are

share

function, package: mplog

Arguments: mode &optional filename

This function is only available on Linux systems. It allows a running Lisp to put it's mplog buffer in shared memory so a second Lisp can read it for formatting. This is useful when debugging a lisp application that hangs and does not respond to the console.

mode is either :write, :read, or :close. The lisp that is being debugged executes (mplog:share :write filename) before enabling recording. The lisp that will be used to format the result executes (mplog:share :read filename) before calling (mplog:show :shared). The lisp displaying the shared buffer does not itself put data in the buffer, and should call (mplog:share :close) when it is done.

mplog

function, package: system

Arguments: code &optional data

The function (named by a symbol in the system package, not the mplog package, note) places an entry in the mplog. This can be useful for syncing the data in the formatted output to program actions. The function returns nil.

code must be an integer in the range #xf000 (61440) to #xffff (65535) inclusive.

data can be anything, even a Lisp value, but what is actually written is the tagged address of the object (its LispVal) at the time of the call and that value will not be maintained by the garbage collector. For this reason it's best to limit this to fixnum values (which will display as 4 (32-bit) or 8 (64-bit) times the values passed in) or addresses of non-moving items.

;; Suppose mplog recording is on. Then we execute

cl-user(17): (sys::mplog #xf1f1 20)
nil

;; Then in the log we have

cl-user(18): (mplog:show)
index tcb        id    name
[...]
2138786: Initial Lisp Listener(7fec6954d700) (#xf1f1 #xa0) ???
[...]

;; #xf1f1 is the code we specified and #xa0 is 160, 8 times the data
;; value, which was the fixnum 20. (#xa0 is the value with the 
;; positive fixnum tag. This example was run in a 64-bit Lisp.)

mplog example 

;; In a running Lisp image, load the mplog module:
(require :mplog)

;; Start mp logging:
(mplog:recording t)

;; Show the current log:
(mplog:show)

;; Stop logging
(mplog:recording nil)

;; To share date with another running Lisp, load
;; the module and start logging:
(require :mplog)
(mplog:recording t)

;; Share the data:
(mplog:share :write "myfile")

;; In a second running Lisp, load the module and link to
;; the shared data:
(require :mplog)
(mplog:share :read "myfile")

;; Still in the second Lisp, show the data
(mplog:show :shared)

;; When done, in the second Lisp stop sharing:
(mplog:share :close)

17.1 Constructing mplog files from the debugger on Windows

mplog:share is only available on Linux machines. On Windows, it is possible to construct mplog results when debugging a running Lisp from information available in either the msdev or gdb debuggers.

To do that you create a text file holding data pulled from the debugger display using cut-and-paste or any other technique that captures the data as text. Optionally edit that file with a text editor to supply identifying names for the different threads. Then evaluate (mplog:show :read fname) or (mplog:show :read fname :format :gdb). See mplog:show.

To get the data that should be copied in the MSDEV debugger, get a memory-display window showing the address and 5 32-bit entries per line, all in hex. Display the region at acl_mplog. Use copy-paste to copy the whole buffer, as is, into an editor. This will be the main body of a file for readlog to process. The rest of the file consists of (mplog:name-thread ...) entries and a marker symbol. Here's how a typical file might look:

(mplog:name-thread 1 #x08c #x52008002 "Initial Lisp Listener")
(mplog:name-thread 2 #x3e4 #x52008802 "portmapper")
(mplog:name-thread 3 #x738 #x54008002 "GWHandler")
(mplog:name-thread 4 #x4c0 #x54008802 "mountd")
(mplog:name-thread 5 #x44c #x54009002 "nfsd")
(mplog:name-thread 6 #x538 #x54009802 "open file reaper")
(mplog:name-thread 7 #x5e0 #x5400a002 "attr cache reaper")
(mplog:name-thread 8 #x708 #x5400a802 "dir cache reaper")
(mplog:name-thread - #x744 - "lisp console")
(mplog:name-thread - #x1b8 - "lisp timer")
(mplog:name-thread - #x3d4 - "Socket Control")
mplog
638A9720  00003E80  0000D6BB  000004C0  00000008  00000004  
638A9734  00003E81  0000D6BB  0000044C  00000200  00000000  
638A9748  00003E82  0000D6BB  000004C0  00000080  00000000  
638A975C  00003E83  0000D6BB  0000044C  00000204  00000000  
638A9770  00003E84  0000D6BB  0000044C  00000202  00000000

with the hex-dump lines continuing to the end of the file.

To get the info for the name-thread entries, display acl_thread_control in the MSDEV watch window, then expand it and scroll down to the registry array. Expand that and you'll see the hex addresses of all the allocated threads. Use the index displayed there with each address as the first number in the name-thread form. The second number is the thread id. The third number is the address of the thread control block itself; the string is the thread name. These last three can be extracted from the memory display since you have the address of the control block in the watch window. The name-thread entries with "-" in the index and control block slots represent non-lisp threads. Get the thread id's from the MSDEV debug/threads panel, and put in whatever you want for the thread names.

The symbol mplog following the name-thread entries is the marker that tells readlog the rest of the file is the log dump.

Now you can (mplog:show :read "xxx"), where xxx is the name of the file with all this stuff in it, and see a human-readable display of the log info pour out onto standard-output.

Note: mplog:show expects the buffer data to be ordered per the usual wrap-around regime. It finds the split point and cuts the deck there to put it in order. It then checks that the sequence numbers of the entries are kosher and complains if they aren't. This would most likely be due to not grabbing all (and only) the mplog buffer data in the cut-and-paste operation.

18.0 Conditionalizing assert calls

assert is a powerful mechanism for making run-time checks for logical assertions by the programmer, but these asseertions come at a high cost, with no run-time way to disable them - disabling is generally done with conditional compilation, which means that in order to disable an assertion the code in question must be recompiled.

The assert-enable feature allows for very fast decision making about the use of assertions at run-time, rather than at compile-time.

The basic structure of assert-enable is a vector of named switches which can be registered (some user-defined) and which can also be enabled or disabled dynamically at run-time.

An assert-enable is used by placing an assert-enabled-p form into a predicate position, for example:

   (when (excl:assert-enabled-p :excl)
     <do-an-excl-related test>)

No other assert-enables are affected, nor do they affect this assert-enable. Each assert-enable is set/enabled or unset/disabled individually.

18.1 The assert-enable interface

There is an internal vector which lists the assert-enable names. It is returned by the list-assert-enables function and is initially:

cl-user(3): (list-assert-enables)
(:excl :cg :db.agraph :generic-checks :reserved4 :reserved5)

The assert-enable names called reservedN are reserved for Allegro CL internal use and should not be used or even referred to by users.

The :generic-checks assert-enable

The :generic-checks assert-enable is somewhat different from the others. It is (by default) set when Allegro CL starts up (all others are initially unset) and it causes all defstruct accessors and svref calls that have not been turned into single-instruction accesses to be checked for their correct lowest-level type (e.g. regardless of whether structure type is being checked or not). We recommend this assert-enable not be turned off unless

Note: If the application it services is to be loaded into a Lisp by other Lisp users or developers, the global nature of the switch will cause defstruct and array accesses to be unsafe.

The status of this global switch will show up in the output of print-system-state.
Also, a warning will be issued if this switch is off after the loading of a file, or just before dumplisp is called.

Adding new assert-enable names

User can add new assert-enable names:

cl-user(4): (register-assert-enable :nolo 6) ;; The name should be a keyword.
:nolo                            ;; The index (2nd arg) should be 
                                 ;; the current vector length.

Now the list has seven elements:

cl-user(5): (list-assert-enables)
(:excl :cg :db.agraph :generic-checks :reserved4 :reserved5 :nolo)

Initially assert-enable names, except for :generic-checks are not set:

cl-user(6): (list-assert-enables-set) ;; The reserved names are usually
(:generic-checks)                     ;; not set but might be.

We create and compile a function making use of our :nolo assert-enable name:

cl-user(7): (defun foo (arg)
              (when (assert-enabled-p :nolo)
                (assert (floatp arg)))
              (1+ arg))
foo
cl-user(8): (compile foo)
foo
nil
nil

If the :nolo name is set, the assert form will be evaluated. If it unset, it will not be. As we said it is initially unset, so a non-float argument will not cause an assertion error:

cl-user(9): (foo 2)
3

Now we set it and try (foo 2) again:

cl-user(10): (set-assert-enable :nolo t)
t
cl-user(11): (foo 2)
Error: the assertion (floatp arg) failed.
  [condition type: simple-error]

Restart actions (select using :continue):
 0: retry assertion.
 1: Return to Top Level (an "abort" restart).
 2: Abort entirely from this (lisp) process.
[1] cl-user(12):

The following timings show the potential effect of unnecessary assertions.

cl-user(60): (time (dotimes (i 1000000) (foo 1.0)))
; cpu time (non-gc) 0.065107 sec user, 0.000889 sec system
; cpu time (gc)     0.031025 sec user, 0.000062 sec system
; cpu time (total)  0.096132 sec user, 0.000951 sec system
; real time  0.097088 sec (99.99%)
; space allocation:
;  0 cons cells, 32,000,000 other bytes, 0 static bytes
; Page Faults: major: 0 (gc: 2), minor: 2 (gc: 2)
nil
cl-user(61): (set-assert-enable :nolo nil)
nil
cl-user(62): (time (dotimes (i 1000000) (foo 1.0)))
; cpu time (non-gc) 0.059274 sec user, 0.000000 sec system
; cpu time (gc)     0.029191 sec user, 0.000000 sec system
; cpu time (total)  0.088465 sec user, 0.000000 sec system
; real time  0.088470 sec (99.99%)
; space allocation:
;  0 cons cells, 32,000,000 other bytes, 0 static bytes
; Page Faults: major: 0 (gc: 1), minor: 1 (gc: 1)
nil
cl-user(63): (/ 0.0971 0.884)
1.0984162
cl-user(64):

As the last form shows, even these simple assertions can result in a near 10% slow down.

Further, you can turn assertions back on with (set-assert-enable :nolo t) (which you can have evaluated with a command-line argument) in case your app has errors that might be caused by improper values which can be caught by assertions.

Users should not set or unset reserved enable-assertion names without instruction from Franz Inc., nor should reserved names be used for user assertions.

18.2 Enable-assertion functionality

Here are the operators that implement enable-asssertion. All are functions.

Name Arguments Notes
list-assert-enables nil Returns the current list of names of assert-enables. The first 6 are reserved for internal use.
list-assert-enables-set nil Returns the current list of names of set assert-enables.
set-assert-enable name value Set the enable-assert with name to value. nil means disable. Any non-nil value means enable.
register-assert-enable name index name is a keyword representing a new assert-enable. index must be an unused index.
unregister-assert-enable name index name is a keyword representing a current assert-enable. index must be the index of the assert-enable. Remove the name from the list.
assert-enabled-p name name is a keyword representing a current assert-enable. Returns true if name names a registered and set assert-enable, and nil if it is resitered but unset. If name is unregistered, this function signals an error. Assert-enables are set and unset with set-assert-enable.

19.0 Creating user-defined hash-table implementations

As of 11.0, hash-tables have been restructured to (among other things) allow user-defined hash-table implementations. This is done using the def-hash-table-implementation macro. In order to create a hash-table implementation, an understanding of the basic structure of a hash-table is needed. The :acl implementation is a pre-defined implementation that encompasses all functionality offered by Allegro CL. Many of the components of the :acl implementation can be elided when building a new implementation, and can be either filled in automatically or can be copied from another implementation.

The structure of a hash table follows these patterns:

  1. The hash-table. A hash-table is an object of two major slots: descriptor and instance. These slots constitute the actions and data, respectively, of the hash-table. Also present in a hash-table is a hash field, which is filled in with a mostly-unique integer for each hash-table created, so that hash-tables can themselves be used as keys in other hash-tables without having to rely on object addresses to provide good hash-code generation for eq, eql, or equal hash-tables.

  2. The descriptor vector: A descriptor vector is a simple-vector which has at least two sections: the identifier section and the dispatch section. Users may choose to add other sections, but the first two are required. A make-hash-table call with an explicit implementation argument will build the descriptor vector according to the implementation specifications. The sections and slots of a descriptor vector are described here.

  3. The hash-table instance. A hash-table instance is usually a vector-like object that has its own type code, but it can be just a vector or even a list, if the implementation will not do rehashes, and its data representation can be any structure or format, if none of the descriptor-vector's functions inherit any behaviors from the :acl implementation. The normal instance from the :acl implementation contains various slots for the hash-table's data, which are enumerated here.

The three-part structure of a hash-table makes it ideal for multiprocessing, and especially smp processing, because the the hash-table object provides identity and doesn't change, the descriptor vector component provides the personality of the hash-table (which also doesn't normally change), and the instance component provides protection from race conditions by turning potential race conditions into data races. For example, if one thread is iterating over a hash-table and another is rehashing the same hash-table at the same time, the iteration by the first thread isn't bothered by the changes the second thread is making, because they are both working on separate instances (the rehashing thread creates a new instance and doesn't install it until it is finished with the rehash), so the first thread is able to iterate completely over the hash-table because it is only working on the instance (and not the hash-table itself). In this situation the iterating thread can be thought of as having finished the iteration before the rehashing thread started. Similar considerations are made for (setf gethash), whose morphing from race condition to data race is a bit more complex, but the concept is similar (i.e. that it doesn't matter when two threads operate on a shared hash-table, as long as they do the right thing and leave the hash-table unbroken). These data races could turn into race conditions if the two threads operating on the same hash-tables interact and expect hashing operations to be performed in a particular order, but that is an issue for higher level programming, and likely involves locks. See Race conditions vs data races for more info on this distinction.

19.1 Descriptor vector sections

Instance Creation

Basic

Less Frequent

Optimizations

Rehash support

19.2 Instance slots for :acl implementation

As a reference, and perhaps as a guide as to what components might be important for an efficient hash-table implementation, the slots which the :acl implementation uses are listed below, in the order in which they appear in the instance.

The self slot

The self slot contains the actual hash-table, which allows the garbage collector and other tools to refer back to the hash-table if necessary.

The keys slot

The keys slot holds the hash-table's key vector. Empty, deleted, and reserved keys are special values that don't represent actual hash-table keys.

The values slot

If values are present in the hash-table, this slot holds a vector of those values, where each value that is present in the hash-table has the same index as the non-empty/deleted/reserved key entry in the keys vector.

The entry-count slot

This slot contains a fixnum which represents a count of key/value pairs in the hash-table. It will be smaller than the sizes of the keys vector and the values vector (if present).

The deleted-count slot

Contains the number of deleted entries in the hash-table. The deleted-count is used when (setf gethash) wants to determine if the hash-table needs to be rehashed due to fullness; due to SMP restrictions a deleted entry cannot be replaced with a new entry, so the deleted count is only reset to zero when the hash-table is rehashed. For this reason the deleted count is added to the entry-count to compare with the rehash threshold, to determine the number of truly empty entries the hash-table still has available.

The rehash-control slot

This slot is nil for normal operation, and during rehashing holds a struct that guides the rehash process and resolves conflicts between multiple threads competing for access to the same hash-table.

The rehash-size slot

This slot holds a fixnum (for a simple increment) or single-float (for a percentage) that indicates how much to grow the hash-table because it is full.

The rehash-threshold slot

This slot holds a fixnum which is the threshold for the hash-table being considered full and in need of a rehash-for-size (i.e. a rehash due to lack of space in the hash-table).

The hash-tick slot (optional)

If the hash-table test is eq or eql, this slot is present and serves to indicate when the addresses of its keys might have changed, thus requiring a rehash-for-address (i.e. a rehash due to keys moving). The garbage collector increments a tick for every scavenge, and when any access on an eq or eql hash-table is performed, the hash-tick is compared with the current garbage collector tick; if they are not the same, then a garbage collection has taken place and some of the keys may have moved, thus possiblly indicating the need for a rehash operation on the hash-table. Note that the hash-tick is not the only indicator that a hash-table is in need of a rehash: some objects used for keys can have hash-codes generated for them that do not depend on their address, so if the only keys currently in the hash-table are this kind of object, the hash-table is not considered dirty (i.e. in need of a rehash), and so a rehash isn't needed even if the hash-tick doesn't match the current gc tick.

20.0 Concise-printing

Concise-printing is a mechanism for printing a lisp form all on one line with a limited output width. The interface for concise printing is not exported or documented, but it is conceptually easy to understand and manipulate. It was first used in the inspector and has since been expanded to include other tools.

It is used primarily in situations where human interaction causes the extra processing power that is needed to not be an issue, mostly in debugging or informational situations like print-function-meta-info, validate-lisp-source, source-context, and others.

The concise printing of a Lisp expression is a trial-and-error printing of that expression until it fits within an excl::limited-string-output-simple-stream (also undocumented/unexported, but see the table for its placement in the simple-streams portion of the stream hierarchy). The length of this stream's string is usually 500, and the limit to the output is the minimum of 500 and either *print-right-margin* (if not nil) or 72 if it is nil. The printing of the expression is repeated for decreasing values of *print-length* and *print-level* (using an undocumented algorithm) until the expression is able to print within the limit.

21.0 Position information

Whenever compile-file or load work on Lisp source, there is always a desire to know where in the source file something is happening. Such position information consists of a character location of a lisp object within a file, usually delineated by a start character (inclusive) and an end character (exclusive) of that object. Position information is always zero-based; the first character in a file is at position 0.

Preserving file positions

Ideally, position information is always available as a file is being read. However, because characters may each contain more than one octet (8-bit byte) and some characters are combinations of other characters, for example #\newline character is a single character on most Unix machines, but is instead a combination of #\return/#\linefeed on Windows. So a file-character-position within a file may diverge from the file-position (which in Allegro CL is defined to be the number octets from the beginning of the file). Also, file-character-position is not always successful at returning a character position while reads are occurring in the source-file. If the reading starts at file-position 0 (which it usually does) and only character operations are performed on the file, character positions will continue to be avaiilable. But if any non-character operation is performed on the stream (e.g. (read-byte stream) or (file-position stream new-position)), the running character position of the stream will be set to nil, thus disabling file-character-position unless and until a (file-position stream 0) is done.

Often, parsing a source file requires read-ahead, and unread-char can be used to go back after a delimiting character is read. But in some situations, where perhaps a Sharpsign macro charcter might read ahead two or more characters instead of one, unread-char can't work in this situation, and the file-position must be saved at the beginning before the read-ahead, and it must be then restored in order to try again from the same file position.

Unfortunately, the restoring of the file-position also disabled the character-position information, so file-character-position will return nil from that point forward.

A new mechanism allows the saving and restoration of both the (octet) file-position and the character-position in files, so that the file-character-position operation can be continued and accurate. If a section of code is wrapped by a with-saved-file-positions then the excursion away from the current file and character positions can be saved and then restored with a call to restore-file-positions just before returning from the form. At any time within the with-saved-file-positions form, saved-byte-file-position or saved-char-file-position can be used to look at the saved value of the character or byte (octet) positions that will be restored.

Examples

mlisp8 example

;;; Start by creating a file with utf-8 characters and looking at it:
;;;
cl-user(1): (with-open-file (s "tmp/tmpfile" :direction :output :if-exists :supersede)
  (write-string "1234567890" s)
  (terpri s))
nil
cl-user(2): (shell "cat tmp/tmpfile")
1234567890
0

;;; Now we read some characters, and then make a temporary excursion beyond where
;;; we eventually want to come back to:
;;;
cl-user(3): (with-open-file (s "tmp/tmpfile")
    (dotimes (i 3)
      (read-char s))
    (format t "after 3, octet: ~s, char: ~s~%" (file-position s) (file-character-position s))
    (with-saved-file-positions (id s)
      (dotimes (i 3)
    (read-char s))
      (format t "inside, after 3 more, octet: ~s, char: ~s~%" (file-position s) (file-character-position s))
      (restore-file-positions id))
    (format t "back out, octet: ~s, char: ~s~%" (file-position s) (file-character-position s))
    (read-char s)
    (format t "after 1 more, octet: ~s, char: ~s~%" (file-position s) (file-character-position s)))
after 3, octet: 3, char: 3
inside, after 3 more, octet: 6, char: 6
back out, octet: 3, char: 3
after 1 more, octet: 4, char: 4
nil

;;; Note above that the character-position was never compromised, even
;;; though we went more than an unread-char would have allowed.
;;;
cl-user(4): (delete-file "tmp/tmpfile")
t
cl-user(5):

mlisp (ics) example

;;; Start by creating a file with fat characters and looking at it:
;;;
cl-user(1): (with-open-file (s "tmp/tmpfile" :direction :output :if-exists :supersede :external-format :fat)
  (write-string "1234567890" s)
  (terpri s))
; Fast loading /net/ice/acl/duane/110beta/src/code/ef-fat.fasl
; Fast loading /net/ice/acl/duane/110beta/src/code/efft-fat-base.fasl
nil
cl-user(2): (shell "cat tmp/tmpfile")
^@1^@2^@3^@4^@5^@6^@7^@8^@9^@0^@
0

;;; Now we read some characters, and then make a temporary excursion beyond where
;;; we eventually want to come back to:
;;;
cl-user(3): (with-open-file (s "tmp/tmpfile" :external-format :fat)
    (dotimes (i 3)
      (read-char s))
    (format t "after 3, octet: ~s, char: ~s~%" (file-position s) (file-character-position s))
    (with-saved-file-positions (id s)
      (dotimes (i 3)
    (read-char s))
      (format t "inside, after 3 more, octet: ~s, char: ~s~%" (file-position s) (file-character-position s))
      (restore-file-positions id))
    (format t "back out, octet: ~s, char: ~s~%" (file-position s) (file-character-position s))
    (read-char s)
    (format t "after 1 more, octet: ~s, char: ~s~%" (file-position s) (file-character-position s)))
after 3, octet: 6, char: 3
inside, after 3 more, octet: 12, char: 6
back out, octet: 6, char: 3
after 1 more, octet: 8, char: 4
nil

;;; Note above that the character-position was never compromised, even
;;; though we went more than an unread-char would have allowed.
;;;
cl-user(4): (delete-file "tmp/tmpfile")
t
cl-user(5):

Copyright (c) Franz Inc. Lafayette, CA., USA. All rights reserved.

ToC DocOverview CGDoc RelNotes FAQ Index PermutedIndex
Allegro CL version 11.0