Environments support in Allegro Common Lisp

This document contains the following sections:

1.0 Environments introduction

An environment in Common Lisp is a Lisp object that contains, in some fashion, a set of bindings and also information about operators, variables, symbols, and so on. This information can be useful in various ways, particularly during evaluation, compilation and macro expansion. The macroexpand operator takes an optional environment argument, whose value is either nil or an environment object.

The actual nature of an environment object is implementation dependent. In standard Common Lisp, there is no way for programmers to access or modify environment objects. In the proposed Common Lisp standard specified by Guy Steele et. al. in the second edition of Common Lisp: the Language, an interface to environment objects was proposed (in Section 8.5, pp 207-214), but the ANSI J13 committee decided not to include this in the ANS.

Allegro CL now has implemented much of the CLtL-2 environments proposal, with some differences that we describe. We recommend that users read that section of CLtL-2, although this document is complete in itself.

2.0 General information about environments in Allegro CL

Environments can be thought of as being specified in CLtL-2, Section 8.5, pp 207-214, with a number of differences. In the following points, we describe the differences between our implementation and the CLtL-2 specification.

Environments

• Environments are immutable, as stated in CLtL2, with one exception: The locative cons returned by the *-information functions (block-information, declaration-information, function-information, tag-information, and variable-information) are mutable, and usually contain values or structures.
• Environment objects are divided into several kinds, although all have the same type/class. These kinds are identified by the augmentable-environment-kind slot: :evaluation, :interpreter, :compiler (previously :compilation), :compilation (previously :compilation-walking), and :macros-only. These kinds are described in detail in Section 2.1 Kinds of environments. Note that :compilation assists in distinguishing between the compilation process, which wants to expand macros, and the walking process, which does not. :macros-only allows the creation of an environment which is appropriate for a macrolet lexical closure.
• The function namespace might have one or more of four different bindings for a name. In order for the compiler to know all of the possibilities, up to three calls to function-information are needed: one with special-operators argument true (which might return :special-operator), one call with a :compiler or :compilation environment (which might return :compiler-macro) and one with a :evaluation environment in order to see if there is a real functional definition (:function or :macro).
• The environment base objects (which all related environments share) contain property tables for storing environment-oriented properties (accessible with ce-get). Both ce-get and ce-putprop are now exported.

Interface Functions

• All interface functions are exported from the system package.
• CLtL-2 called for three namespaces: variable, function, and declaration. Allegro CL implements two additional ones: block and tag. These have block-information and tag-information accessors, respectively, and corresponding keyword arguments to augment-environment.
• The second return value for all *-information functions except for declaration-information has been moved to the fourth value (since it is seldom used) and instead the second value returned is usually a locative cons (a cons whose car and/or cdr may be used as a mutable value). This locative cons is used to provide the local binding of the value (for :evaluation environments) or local compiler structure information (for the two compilation environments). To reduce consing at critical points in the environments implementation, there are a few exceptions where the second value is not a cons representing a locative:
  1. When the first return value from variable-information is :constant, then the second value returned is an actual value, and not a value housed within a locative. This ensures that the value is not changed by replacing it in the (non-existent) locative.
  2. When the first return value from variable-information is :special, then the second value is nil, because the value is dynamic and can be best accessed via symbol-value.
  3. When the first return value from function-information is :special-operator, then the second value is nil, because the value of a special operator is opaque to the programmer (companion-macros are mandated by the spec in order for a non-compiler to "see" into special-operators, and so a functional value doesn't make sense).
• When the first return value from function-information is either :function or :macro, and the definition is in fact dynamic and thus accessible via fdefinition or macro-function, respectively, then the second returned value is nil, unless the third argument to function-information is non-nil, indicating that consing definitions and declarations is ok.
• All of variable-information, function-information, and declaration-information accept a null environment argument, which may mean to look in the global environment. The global environment is the environment of the running lisp, possibly shadowed by the *compilation-unit-environment*.
• augment-environment allows some very limited augmentation of the global environment (either the null lexical environment or compile-file-environments) when only one name-to-value mapping is being added, or when only declarations are being added. Rules for this augmentation are not solid yet, and in certain cases may require use of the :reuse argument described below.
• Both variable-information and function-information may receive an optional third argument, a boolean which requests a building of all declarations out of previous declarations of the same name, including the global environment. This argument being non-nil also results in a non-nil second return value (the definition housed in a locative), although these return values are likely to be consed on the fly. This switch is added so that the interpreter, which almost never looks at declarations, doesn't need to cons as much for no good reason.
• function-information receives an optional fourth argument controlling information returned for special-operators. See Section 2.1 Kinds of environments or the description of function-information for more details.
• function-information might return as its first value :special-operator, not :special-form.
• augment-environment accepts the additional keywords special-operator, block, tag, compiler-macro, constant, and flavor-iv (i.e. flavor instance variable), which contribute to the function, block, tag, compiler-macro, variable, and variable namespaces, respectively.
• augment-environment accepts a single name for each keyword except :declare, in order to reduce consing. In this case, the locative (if present) is sought in the :locative argument (see below)
• augment-environment accepts the additional keyword argument reuse. It is not necessary to create a new augmentable-environment object whenever adding to the environment, but it is only useful to cons a new environment object when entering a new contour (both in the compiler and in an interpreter). When reuse is nil (the default) a new environment object is consed, i.e. the environment object returned is not eq to the one given. But when reuse is non-nil, then the old environment object is returned, and any additions to the environment are added at the same level, as if they had been all added in the same call to augment-environment which created this environment object.
• augment-environment accepts the additional keyword argument locative. Usually, the locative is nil or a cons cell, and can be used efficiently when only one name is being added to the environment. When a :constant is being augmented, the locative argument is the actual value of the constant. The locative argument becomes the value which is returned as the second value from the *-information functions. For augmentation with many names at a time, a locative can be specified for each name, where instead of a list of names for each keyword, the list may be an alist, each element of which specifies the name in the car and the locative in the cdr. The car of a non-nil locative cons is always mutable, unless it represents a :constant value. See also the function system:constant-value.
• define-declaration is implemented, but the lambda-list called for in its definition in CLtL-2 is ambiguous as to whether it applies to some kind of syntax for the declaration specifiers, or the arglist of the two-argument function this macro is supposed to define. Therefore the lambda-list is chosen to specify the syntax of the declaration, and the arglist for functionality of the declaration is given either implicitly or via a lambda form. The lambda-list for define-declaration is different from that specified in CLtL-2, with additional required arguments prop and kind.

  The functions that are described as being defined by the macro are in fact implemented, and work as specified, except that it may return as its first value one of :function, :variable, :both (meaning :function and :variable), or :declare.

• ensure-portable-walking-environment: this function returns an environment suitable for portable code walkers to use entirely within the ANSI Specification of Common Lisp.

2.1 Kinds of environments

The kind slot of an augmentable-environment object determines what behaviors the accessors have on it. Usually, accessors will return similar information, but for several reasons including performance and the fact that namespaces are not pure mappings, the kind does play a part, along with optional arguments, in returning different levels of information.

An environment object can be "modified" to become a different kind simply by performing a null augmentation on the original environment and by storing into the kind slot:

(setq e2 (sys:augment-environment e1))
(setf (sys::augmentable-environment-kind e2) :evaluation)

which results in an evaluation environment in e2, which has exactly the same information as e1 (which might be any kind of environment, but which in practice is probably a :compiler environment).

The environment kinds are:

• :interpreter: for a lisp with an interpreter, such as Allegro CL, an interpreter environment can form the basis and implementation for the interpreter. Accesses on it will tend to generate no declaration information (with the special-case of the special declaration), and global values will be left up to the caller to retrieve, rather than to cons up a locative for each call. The by-words for an interpreter environment are speed and cons-free operation.
• :compiler: a compiler environment is the normal environment which the compiler uses to establish contours and contexts for the compilation process. The by-word for a compilation environment is as much information as possible. (Prior to version 8.0, :compiler environments were called :compilation environments.)

  A compiler environment might also have a global component, suitable for storing definitions temporarily during a compile-file If so, it is generally stored into *compilation-unit-environment*, and any definitions it contains will shadow any global definitions stored directly in the lisp. When the *compilation-unit-environment* is removed, then the shadowing is stopped, and the original definitions appear again.

• :evaluation: an evaluation environment is a compilation environment posing as an interpreter environment. It has many of the same characteristics of a compilation environment, and in fact is created with the augment-environment/setf technique described above.
• :compilation: a compilation environment is also similar to a compiler environment, except that macros and compiler-macros can recognize one and macroexpand differently. Note: it is a goal to eventually remove this kind of environment; the distinction should not be as useful as it currently is. (Prior to version 8.0, :compilation environments were called :compilation-walking environments.)
• :macros-only: this environment kind serves a special-purpose when making a lexical-closure for a macrolet. Because macrolet makes macro-functions in its own lexical environment, but because referencing a local variable within this environment is undefined, it is necessary that only macro definitions be copied when the lexical-closure is created.

Special Handling of function-information returns

If one considers a namespace to be a one-to-one mapping of a name to a binding, then the function namespace is not a pure namespace in Common Lisp; consider that a name can simultaneously name both a special-operator and either a macro or a compiler-macro, or it can name a macro or function and a compiler-macro simultaneously. Of course, any lexical definition created for that name (such as an flet, labels, or macrolet) will shadow all of these potential combinations, but if no such shadowing occurs, there is a necessity for function-information to be able to make the distinctions between the various combinations of definition that are possible.

If the fourth argument (the special-operators argument) to function-information is true, and if the name is a non-shadowed special-operator, then :special-operator is returned, even if it has a macro or a compiler-macro definition as well. If the argument is nil, then for a special-operator which also has a compiler-macro, :compiler-macro is returned only for :compiler and :compilation environments (otherwise :special-operator is returned), and for a special-operator which also has a macro definition, :macro is returned only for :interpreter and :evaluation environments (otherwise :special-operator is returned).

We do not define what occurs if a special-operator has both a macro and a compiler-macro definition, because Allegro CL has none of these situations. There should be a normalized behavior for such a situation.

If a name defines a compiler-macro as well as either a macro or a function, then which is returned depends on the environment kind: a :compiler or :compilation environment will cause the :compiler-macro to be returned, and an :interpreter or an :evaluation environment will result in the :function or :macro being returned.

3.0 Environments functionality

The following functions and variables are defined in our environments implementation:

• augment-environment
• block-information
• ce-get
• ce-putprop
• declaration-information
• define-declaration
• function-information
• make-compile-file-environment
• tag-information
• variable-information
• *compile-file-environment* (now deprecated, use *compilation-unit-environment* instead)
• *compilation-unit-environment* (8.0 replacement for *compile-file-environment*)
• *interpreter-environment*
• *evaluation-environment*

4.0 Environments in Allegro CL

The optimize qualities (safety, space, speed, debug) are no longer repesented by variables in the excl package (e.g. excl::.speed.) Also, the compilation-speed quality is added to the set, because it is specified by ANS. These 5 qualities are now accessed by (sys:declaration-information 'optimize <env>) which returns an alist which will always include at least one of each of these qualities. This return value is constant -- never modify its contents. Also, there may be more than one entry for a quality; the first one encountered is the correct value for the specified environment.

The optimize declaration is defined by

(sys:define-declaration optimize (declaration env)
  .optimize.
  :declare
  <body>)

This means, for debugging purposes, that sys::.optimize. is used as the holder of the global optimization quality list.

The declaration-information function will return this list, possibly shadowed by local lexical optimize declarations, depending on the environment object passed to it.

Example 1

cl-user(1): (sys:declaration-information 'optimize nil)
((safety 1) (space 1) (speed 1) (compilation-speed 1) (debug 2))
cl-user(2): 

Example 2

cl-user(1): (setq e1 (sys::make-augmentable-environment-boa :compilation))
#<Augmentable compilation environment @ #x71b0941a>
cl-user(2): (setq e2 (sys:augment-environment e1 :declare '((optimize speed (safety 0)))))
#<Augmentable compilation environment 1 @ #x71b0992a>
cl-user(3): (sys:declaration-information 'optimize e1)
((safety 1) (space 1) (speed 1) (compilation-speed 1) (debug 2))
cl-user(4): (sys:declaration-information 'optimize e2)
((speed 3) (safety 0) (safety 1) (space 1) (speed 1)
 (compilation-speed 1) (debug 2))
cl-user(5): 

Inline, notinline declarations

This is an example of a mutually exclusive declaration; one can declare the same symbol binding both inline and notinline, but not at the same time. Therefore, these two declarations are combined into a single declaration class called inline.

The entire definition for these two declarations is

(sys:define-declaration (inline inline notinline) (declaration env)
  .inline.
  :function)

The first inline in the namespace is the class, and each of the two further names in the namespace are the declaration instances. The kind of declaration is :function, and the body is null and thus takes on the default action.

Thus, function information for a function name that has been declared either inline or notinline will include as its third returned value an entry of the form (inline <value>), where value is one of inline, notinline, or nil (but this might usually be left off).

Example 1

cl-user(1): (setq e1 (sys::make-augmentable-environment-boa :compilation))
#<Augmentable compilation environment @ #x71b0941a>
cl-user(2): (setq e2 (sys:augment-environment e1 :function 'foo
                       :locative (list #'(lambda (x) (1+ x)))
                       :declare '((notinline foo))))
#<Augmentable compilation environment 1 @ #x71b09c62>
cl-user(3): (setq e3 (sys:augment-environment e2 :declare '((inline foo))))
#<Augmentable compilation environment 1 1 @ #x71b09eba>
cl-user(4): (sys:function-information 'foo e1)
nil
cl-user(5): (sys:function-information 'foo e2)
:function
(#<Interpreted Function (unnamed) @ #x71b09c32>)
((inline notinline))
t
cl-user(6): (sys:function-information 'foo e3)
:function
(#<Interpreted Function (unnamed) @ #x71b09c32>)
((inline inline) (inline notinline))
t
cl-user(7): 

Example 2

cl-user(1): (sys:function-information 'bar nil)
nil
cl-user(2): (sys:augment-environment nil :declare '((notinline bar)) :reuse t)
nil
cl-user(3): (sys:function-information 'bar nil)
nil
cl-user(4): (sys:function-information 'bar nil t)
:free
(nil)
((inline notinline))
cl-user(5): (setq e1 (sys::make-augmentable-environment-boa :compilation))
#<Augmentable compilation environment @ #x71b09bd2>
cl-user(6): (setq e2 (sys:augment-environment e1 :declare '((inline bar))))
#<Augmentable compilation environment 1 @ #x71b09fba>
cl-user(7): (sys:function-information 'bar e1)
nil
cl-user(8): (sys:function-information 'bar e1 t)
:free
(nil)
((inline notinline))
cl-user(9): (sys:function-information 'bar e2 t)
:free
(nil)
((inline notinline) (inline inline) (inline notinline))
nil
cl-user(10): 

Ignore, ignorable, excl::ignore-if-unused declarations

Like inline and notinline, these declarations are mutually exclusive. excl::ignore-if-unused is the archaic form for ignorable. The declaration class is called ignore, and will have a value of ignore, ignorable, or excl::ignore-if-unused (though the archaic form only shows up in a compilerless lisp, where it is never used anyway).

The initial definition of these declarations is:

(sys:define-declaration (ignore ignore ignorable excl::ignore-if-unused)
    (declaration env)
  .ignore.
  :variable)

However, when the compiler is loaded, it overwrites this definiton with one which will

• do some error checking
• fold excl::ignore-if-unused into ignorable
• annotate the value in the locative (which is currently a varrec structure)

It doesn't make sense to ignore a variable which hasn't been bound, but no error is generated when such a situation occurs.

Example 1

cl-user(1): (setq e1 (sys::make-augmentable-environment-boa :compilation))
#<Augmentable compilation environment @ #x71b0941a>
cl-user(2): (setq e2 (sys:augment-environment e1 :variable 'foo
                        :locative (list (comp::make-varrec-boa 'foo))
                        :declare '((ignore foo))))
#<Augmentable compilation environment 1 @ #x71b09ab2>
cl-user(3): (multiple-value-list (sys:variable-information 'foo e2))
(:lexical
 (#<compiler::varrec foo DynamicExtent: maybe Used: ignore
    unknown-type>)
 ((ignore ignore)) t)
cl-user(4): (comp::varrec-used (caadr *))
ignore
cl-user(5): 

Example of maintenance of definitions in compile-file-environment, as opposed to the global-environment:

Note that the two definitions of foo given here are maintained in different locations; one in the global environment, and one in the lexical environment:

cl-user(1): (fboundp 'foo)
nil
cl-user(2): (sys:augment-environment nil :function 'foo
              :locative (list (compile nil (lambda (x y)
                                             (+ x y)))))
nil
cl-user(3): (sys:function-information 'foo)
:function
nil
nil
cl-user(4): (fdefinition 'foo)
#<Function (:anonymous-lambda 9) @ #x71b0ae42>
cl-user(5): (funcall * 10 20)
30
cl-user(6): (setq e1 (sys:make-compile-file-environment))
#<Augmentable compilation environment @ #x71b0e7b2>
cl-user(7): (setq e2 (sys:augment-environment e1 :function 'foo
                     :locative (list (compile nil (lambda (x y)
                                                    (- x y))))))
#<Augmentable compilation environment 1 @ #x71b11d62>
cl-user(8): (sys:function-information 'foo)
:function
nil
nil
cl-user(9): (fdefinition 'foo)
#<Function (:anonymous-lambda 9) @ #x71b0ae42>
cl-user(10): (sys:function-information 'foo e2)
:function
(#<Function (:anonymous-lambda 10) @ #x71b10bda>)
nil
t
cl-user(11): (multiple-value-bind (ignore locative)
                 (sys:function-information 'foo e2)
               (funcall (car locative) 10 20))
-10
cl-user(12): 

Copyright (c) 1998-2019, Franz Inc. Oakland, CA., USA. All rights reserved.
This page was not revised from the 8.2 page.
Created 2012.5.30.