Allegro CL Case Modes

Allegro CL provides various options for starting Lisp, including a choice between "Modern" and "ANSI" mode Lisps. See Allegro CL Executables in startup.html for details of the startup options.

This document describes the issues of case sensitivity and modern and ANSI modes.

2.0 Allegro CL Modern Mode

The Modern/ANSI mode choice refers to the Lisp image's case mode. As described elsewhere in this document, a case mode is a combination of two things: (a) the Lisp symbol reader's default behavior, and (b) which case (upper case or lower case) is used to store pre-defined Lisp symbols which name functions, variables, classes, etc. The case mode being used has a big effect on how source files and data files are read and how their data may be represented internally or output by Lisp. Thus, it is important to understand how the case mode choice affects one's Lisp programming. But first, a couple of comments about compatibility.

2.1 Modern mode background

The ANSI Common Lisp specification states that all pre-defined Common Lisp functions, variables, classes, etc. are represented as symbols whose names are stored in all upper case. Thus, for example, suppose you wish to evaluate the following Lisp source expression:

When Lisp reads this expression, either by loading a source file, or from a user typing it to a Lisp Listener, the symbol reader internally converts all symbol names to their upper case equivalents. (As an aside, please note that case of some non-English letters can actually depend on a user's locale. Thus, it is possible to have the Lisp reader's behavior depend on the environment in which it is being run. Other than mentioning it here, we disregard this problem in the rest of this document.)

When Lisp evaluates this expression, it finds that the first element, MEMBER, matches the pre-defined Lisp symbol, MEMBER, whose function when applied to the arguments as described, returns the following:

As can be seen by the examples below, the ANSI Lisp symbol reader is case-insensitive. In other words, each of the following expressions, when read in ANSI mode, returns the same result:

One of the features of such a case-insensitive reader is that it allows programmers to write source code using mixed case knowing that Lisp will canonicalize (and thus ignore the case variants of) the symbol names. Some programmers may choose to adopt a coding style where they indicate predefined Lisp symbols in upper case:

One can have many different such case-variant styles even within a single file. Whether it is good or not to have such styles is another matter. However, as far as Lisp is concerned, all (unescaped) symbols are converted to upper case as they are read.

2.2 ANSI mode problems

As long as Lisp programs only require that symbol names be in upper case, there would be no problem using the default ANSI specified Lisp reader. However, difficulties occur when interfacing to non-Lisp programming systems which are increasingly using lower case case-sensitive identifiers. The case-insensitive nature of the Common Lisp symbol reader creates hurdles for programmers wishing to interface to such systems.

For example, if a user wants to establish a foreign function associated with the Microsoft Windows API function CreateDIBitmap() (which is used to Create a Device Independent Bitmap), the default Lisp reader action is to turn the name into upper case. Thus, one may have source code which reads as follows:

If, however, an error occurs during the execution of this call, the Lisp stack backtrace displayed by the debugger is hard to read due to missing word boundary information in the function's name:

Processing XML data is another example which demonstrates Lisp's power as well as the ANSI-mode hurdle with case-sensitive data. Suppose we have employee information in XML format as shown in the following fragment:

A natural way to represent such elements in Lisp is with keyword-style lists, and this is what Franz' open-source xml-parser returns. XML is intrinsically case sensitive, and the DTD (Document Type Definition) mandates the case of all tag and attribute names. Thus, the above expression can be represented with case-preserving tags, but escapes are needed when the list is printed in ANSI-mode:

A reason to choose symbols for tags is that we can then use powerful Lisp tools such as destructuring-bind to further process the parsed structure. Element attributes in XML consist of key/value pairs, but the attributes attached to any node are unordered, and it can be tedious to extract some set of attributes from a node. This pattern is so very similar to Lisp keywords that exploiting Lisp lambda-list binding is an obvious mechanism for extracting and collecting attributes and their values. In ANSI mode, however, Lisp code which deals with case-sensitive symbol tags needs a lot of escaping, making the code both hard to read and hard to write.

 ;; An XML phone directory processor
 (case (car element)
   ((|CompanyOfficer| |Manager| |Employee| |Consultant|)
    (destructuring-bind
        ((tag &key ((|OfficePhone| office-phone))
                   ((|HomePhone| home-phone))
                   ((|SSNumber| ss-number))
              &allow-other-keys)
         &rest content)
        element
      (my-formatted-output content tag office-phone
                           home-phone ss-number)
      ...))
   (|Customer| ...)
   ...)

The above code works in either ANSI mode or Modern mode Lisp, but we'll see below how this code can be rewritten more felicitously in Modern mode.

Many Lisp programmers have chosen to avoid the symbol case-insensitivity problem by using strings instead of symbols to represent externally named objects. Strings are always read with their case preserved. Unfortunately, by using strings, a Lisp programmer gives up using Lisp variables for case-sensitive external identifiers. Furthermore, one gives up other features unique to symbols such as fast eq comparisons or being able to store the identifiers in separate package name spaces.

Note that ANSI Common Lisp does provide some support for case mode choice, using readtable-case and various global variables. See readtable-case for details.

2.3 Modern Mode

Modern programming environments almost universally use case sensitive lower case identifiers. Modern mode in Lisp thus eases interfacing to such environments. It also simplifies the Lisp reader.

As opposed to ANSI Mode, symbols in Modern mode are simply treated as WYSIWYG (What You See Is What You Get). In other words, symbols are simply read with case preserved. Thus, following up on an example above, the call to (CreateDIBitmap ...) shows up as (CreateDIBitmap ...) when displayed in a stack backtrace, or trace output, etc. Further examples show how in a Modern mode Lisp session, one gets straightforward results:

Thus, in Modern mode, one is easily able to use symbols as names for externally defined case-sensitive identifiers (e.g., foreign functions).

As a further example, the Lisp / XML fragment from above can be greatly simplified by using unescaped symbols to represent XML tags, and by using the corresponding case-sensitive variable names directly in Lisp code:

;; An XML phone directory processor
(case (car element)
  ((CompanyOfficer Manager Employee Consultant)
   (with-xml-element (tag (OfficePhone HomePhone SSNumber)
              &rest content)
       element
     (my-formatted-output content tag
              OfficePhone HomePhone SSNumber)))
  (Customer ...)
  ...)

(defmacro with-xml-element ((tag (&rest attributes)
                 &rest contents)
                element
                &body body)
  `(destructuring-bind
       ((,tag
     &key ,@(mapcar #'(lambda (x)
                (list (list x x)))
            attributes)
     &allow-other-keys)
    ,@contents)
       ,element
     ,@body))

As for readtable-case, there is little need for it at all in Modern mode. In fact, the original implementation of Modern mode, which predates readtable-case and even the full ANSI Common Lisp specification itself by several years, ignored readtable-case settings because ignoring them allowed greater ease moving between ANSI mode and Modern mode by allowing the same readtable-case settings in both modes. Responding to user feedback, however, Franz Inc. reconsidered that decision. Allegro CL now provides full readtable-case functionality in Modern mode. The default readtable-case setting for Modern mode is :preserve.

2.4 A note on support for ANSI mode

Franz Inc. is committed to providing fully ANSI-compliant Common Lisp development and runtime environments. With each release, Franz Inc. includes a fully supported ANSI-compliant Common Lisp executable. As Franz Inc. introduces new features that extend the ANSI Common Lisp specification, priority is given to ensure not only that ANSI-compliant Common Lisp code will run, but also that such new features are as backwards compatible as possible with prior Franz Inc. releases.

Modern mode is provided separately from the backwards compatible ANSI executable(s) because Modern mode directly contradicts the ANSI specification in two very specific user-visible ways which can break ANSI-compliant Common Lisp programs. If you have Allegro CL installed, you may find, for example, files named alisp.dxl and mlisp.dxl. These correspond to the base lisp images for ANSI mode and Modern mode respectively. (Please see startup.html for information on starting Allegro CL images.) Windows users may find "ANSI" and "Modern" as Allegro CL start menu items. Other than the case mode, however, the images are equivalent. Users unable to switch to Modern mode can simply stick to using the ANSI mode versions.

In order to increase Common Lisp's effectiveness with modern day, externally accessible, case-sensitive programming environments, Franz Inc. is bringing its Modern mode to the fore as a way to encourage Lisp users and developers to consider new ways of thinking about how Lisp symbols can be used. In Modern mode, Lisp symbols can easily be used not only for Lisp function/variable/object names, but also for external names such as classes/functions in Java/C++, or data elements in XML or CORBA. Note that it is generally easy to write and maintain Lisp code which runs in either ANSI or Modern mode.

3.0 Case implementation in Allegro CL

There are two parameters that determine the reader's actions: case preference and case sensitivity. The preferred case is either upper or lower, and refers to the case of the characters in the print names of all of the standard symbols, such as car and cdr. Case sensitivity can be either sensitive or insensitive to case. Case-sensitive means that the Lisp reader does not modify the case of any characters it reads. Case-insensitive means that characters that are not of the preferred case are converted to the preferred case.

Thus, there are four possible values for the combination of case preference and case sensitivity. However, only three are supported in Allegro CL.

So, Allegro Common Lisp can operate in three different case modes, two of which, modern (case-sensitive lower) and ANSI (case-insensitive upper) are the more important and treated in detail in this document.

4.0 readtable-case

The ANSI specification does, in fact, address limited case-sensitivity by providing readtable-case functionality. The function readtable-case accesses (and with setf sets) the readtable-case of a readtable object. Franz Inc. fully supports this readtable-case functionality in both ANSI and Modern modes.

While the ANSI readtable-case solution may be suitable for programmers wishing to use ANSI mode yet still requiring a case-sensitive reader, this functionality is somewhat counterintuitive due to ANSI limitations. Specifically, the only readtable-case setting which can effectively be used that doesn't require all source code to be converted to upper case and which still provides case-sensitivity is the :invert setting. The way this setting works is by having the Lisp symbol reader turn symbol names with all lower case characters into all upper case and vice versa.

For those not already aware of the interaction between *print-case* and the :invert readtable-case, ANSI mode Common Lisp can create some surprising results. For example, a symbol whose name is "ZEBRA" is printed in all lower case even when *print-case* is specified to be :upcase. Also, the symbol whose name is "Zebra" is printed as 'Zebra' (i.e., capitalized) even if *print-case* is either :upcase or :downcase. While these results are understandable, given how the reader and printer must operate with the ANSI specified case mode, they are not intuitive. On the other hand, the Allegro CL case modes, the Modern mode in particular, provide a direct way of dealing with reading and printing case-sensitive symbols.

For each case mode, the initial case settings for readtable-case are summarized in the following table:

mode	initial symbol-name storage	readtables cases
`:case-insensitive-upper` (ANSI)	uppercase	`:upcase`
`:case-sensitive-lower` (Modern)	lowercase	`:preserve`
`:case-insensitive-lower`	lowercase	`:downcase`

A call to set-case-mode resets to the corresponding value in the table above the readtable-case of the Lisp standard readtable as well as all existing readtables except those whose readtable-case has been explicitly set. In other words, the default situation is that a readtable's readtable-case will be updated by set-case-mode. If you have used setf to modify a readtable's readtable-case to one of the ANSI-specified settings, then that readtable's readtable-case is not changed by subsequent calls to set-case-mode unless/until that readtable-case is set to :default. Specifically, for any readtable rt, evaluating

sets rt's readtable-case to the value according to the current case mode as specified by the above table. Furthermore, unless the rt's readtable-case is later set to a value other than :default, rt's readtable-case is automatically adjusted by future calls to set-case-mode.

5.0 ANSI/Modern code portability

In general, Common Lisp programs which don't depend on case-sensitivity, and which are written using lower case for pre-defined Lisp names will work either in Modern mode or in ANSI mode. Code which works in both ANSI mode and Modern mode is called case-portable code.

There are a few areas that programmers must look for when developing code that is to be case-portable, that is compatible between both Modern and ANSI modes (and portable to other conforming Common Lisp implementations).

Specifically, places where literal upper case strings are used as string designators can portably be replaced by symbols. For example, the following form works only in ANSI mode:

If the string designator is changed to be a symbol, then the form will work in both ANSI and Modern modes (and is portable to other conforming Common Lisp implementations):

Some functions may deal with symbol-names directly in other ways. Again, these can also be made compatible between ANSI and Modern modes by using symbols (often keywords) as string designators. For example, the following form works only in ANSI mode:

(defun my-new-symbol (arg)
  (intern (concatenate 'string
        "MY-NEW-SYMBOL-"
        (format nil "~a" arg))))

If written as follows, however, the function works exactly the same way in ANSI mode, but also can be used in Modern mode:

(defun my-new-symbol (arg)
  (intern (concatenate 'string
        (symbol-name :my-new-symbol-)
        (format nil "~a" arg))))

Some symbols naming operators in the winapi and winapi-dev packages, which are documented only indirectly in cgide.html, have mixed-case symbol names like CreateWindowEx. Unlike in releases prior to 9.0, these names do not generally have single-case equivalents in an ANSI Lisp unless specifically documented. This change was made to ease portability between ANSI and Modern code.

That package is the only one in Allegro CL where mixed-case symbols name operators.

5.1 Package names

Packages are named by strings and are looked up in a case sensitive manner. In Modern mode the standard package names are all in lower case and in ANSI mode they are in upper case. Thus a good way to refer to package is not by a string e.g. "common-lisp-user" but by a symbol such as :common-lisp-user or #:common-lisp-user. Putting the symbol in the keyword package or using an uninterned-symbol keeps from cluttering the current package (and maybe causing symbol clashes).

Symbols can be used nearly everywhere a package name is needed. The functions that look for package names use the symbol-name of a symbol passed as an argument. By doing

you end up calling in-package with "ole" in Modern mode or "OLE" in ANSI mode, and thus the package is found regardless of the mode.

A global solution to matching package names is to set the variable ignore-package-name-case. When the value of that variable is true, package name to package lookup is done in a case insensitive manner.

5.2 Package definitions

5.3 Creating symbols

(defun constructor (struct) 
  (intern (format nil &quot;MAKE-~a&quot; struct)))

This assumes that this code will run in ANSI mode. Writing it as follows allows the case of the make- symbol and the struct argument to be determined by the case of the symbols' symbol-names, which will be correct for the case mode:

(defun constructor (struct) 
  (intern (format nil &quot;~a~a&quot; 
            (symbol-name :make-) (string struct))))

5.4 Testing symbols

When reading and testing against symbol names you'll want the code to work regardless of the case of the print names. This can be accomplished by using case-insensitive predicates (such as equalp). Another possibility is to use (string :foo) rather than "foo" or "FOO so that the correct string for the current case mode is used at runtime.

6.0 Importing ANSI/Modern-mode code into a Modern/ANSI-mode Lisp

Import can mean two things: (1) compiling and loading source files, and (2) loading fasl (compiled) files. Obviously, if you have the sources, you can make modifications when you encounter difficulties. When you only have fasl files, modifications are harder to impossible but some potential problems go away.

Please note that importation may not be possible. Particularly if the suggestions in ANSI/Modern code portability are not followed, the sources must be modified and likely the fasl files simply will not work (they may load but the code will likely error when run).

The macro in-case-mode is designed to facilitate loading files compiled in one mode into the other mode. (It does not deal with the case-insensitive, lowercase preferred mode.) See the description of that macro for further information.

6.1 Importing ANSI mode code into Modern mode

Generally ANSI mode code can be made compatible by looking for and fixing the few items we've mentioned above in ANSI/Modern code portability. However you may encounter code written in a style such as this:

(DeFun Foo (x y)
      (Cond ((PlusP x) (Sqrt y)) 
            (t y)))

The author of this code is taking advantage of the fact that in ANSI mode the case of symbols in the source code doesn't matter. In order to port this to Modern mode you could go through and lowercase every symbol naming an official Lisp form. The author of the code is unlikely to want the code back after such a transformation, as he must see some value to his style of capitalization. A way to make this code readable into Modern mode Lisp without modifying it is to switch the Lisp to :case-insensitive-lower mode. In this mode, the strange symbol names will be downcased. After reading in this file the mode can be switched back to :case-sensitive-lower mode.

Note that when you switch to :case-insensitive-lower mode the value of *print-case* determines how symbols will be printed. The initial value of *print-case* is :upcase. Thus unless you change *print-case* you'll find that when you switch to :case-insensitive-lower, symbols will start printing in upper case even though their print names are in lower case. If this annoys you, you can set *print-case* to :downcase.

6.2 Importing Modern mode code into ANSI mode

This often works so long as mixed-case symbol names are not used, nor are two distinct symbols in the same package named with string-equal names. If you have compiled files only see in-case-mode.