ToC DocOverview CGDoc RelNotes FAQ Index PermutedIndex
Allegro CL version 11.0

Building Images

Though there are many links to build-lisp-image in this document, essentially all relevant information about build-lisp-image is contained in this document.

The function build-lisp-image can be used to create a new image (dxl) file. This will be a fresh image, fresh in the sense that it inherits little from the running image (only values of certain global variables used as argument defaults). Typical reasons for building images with build-lisp-image include:

build-lisp-image is a lisp function, so it is called from a running Lisp. We distinguish between the image which calls build-lisp-image, the running image, from the image being created, the new image.

build-lisp-image spawns a separate process. On Windows, a "Create image" window displays information about the build process (you may have to expose this window). On Unix, information is printed to the listener. You will see many prompts: do not attempt to type to them.


1.0 Comparison with excl:dumplisp

Images can also be created with dumplisp, as described in the documentation for that function and in the document dumplisp.html. An image created by excl:dumplisp is essentially a copy of the currently running image, with all loaded functionality included and all values of variables preserved. In contrast, build-lisp-image spawns a separate process which builds a new image out of constituent parts. The currently running image supplies only default values for certain arguments. The new image does not capture loaded or newly-defined functionality of the running image (thus, if the function user::foo is defined in the currently running image, it will not be defined in the new image created by build-lisp-image).

This is why build-lisp-image can be used to create a new image with new patches. Patches in the currently running image are not captured by the newly created image. Instead (as described below), patch files are built into the newly created image but the patches in the current image are not involved.

Note that build-lisp-image calls dumplisp in the spawned process. All arguments accepted by dumplisp are also accepted by build-lisp-image except the name argument (which is accepted but ignored since the required argument to build-lisp-image specifies the new image name).


2.0 Comparison with excl:generate-application

generate-application itself calls build-lisp-image. generate-application is designed to produce a directory of files suitable for shipping to another machine or site (note that you must be licensed to distribute software built in Allegro CL -- contact your Franz Inc. account manager if you are unsure of licensing terms). In contrast, build-lisp-image builds a single image file only. See the description of generate-application and the document delivery.html.


3.0 The template of a call to build-lisp-image

The "BUILD" module must be loaded for build-lisp-image to work (that is, "BUILD" should be on the modules list). It will be loaded automatically when build-lisp-image is called. You can force loading of the module by evaluating

(require :build)

build-lisp-image takes one required and many keyword arguments. The required argument must be a string naming a file, with or without path information. If no path information is specified, the image file will be placed in the current directory (as returned by current-directory). If relative path information is supplied, it will be resolved relative to the current directory. The image file name must have an extension (type). If it does not, it will not be found when Lisp is started. The standard extension (type) of an image file is .dxl but any extension will do. Note that on Windows machines, .dxl is registered and associated with Allegro CL, so double-clicking on a dxl file (in the Windows Explorer, say) executes mlisp.exe with -I [that file].dxl as arguments. When mlisp.exe is called without an image specified with -I, it looks for an image (dxl) file with the same name (i.e. mlisp) and in the same directory. You can copy mlisp.exe to [anything else].exe if you want it to find a different image file named [anything else].dxl automatically.

Here is the template for a call to build-lisp-image:

(excl:build-lisp-image image-file
                       ...keyword/value pairs... 
                       ...dumplisp-keyword/value pairs...)

3.1 The character size in the resulting image

Allegro CL supports two character sizes: 16-bit and 8-bit. Thus images and executables either use 16-bit characters or 8-bit characters. No image or executable supports characters of both sizes. See iacl.html for information on character sizes in Allegro CL.

build-lisp-image creates a 16-bit character or an 8-bit character image as the new image is created using a 16-bit executable (mlisp and alisp on UNIX, mlisp.exe and alisp.exe on Windows) or an 8-bit executable (mlisp8 and alisp8 on UNIX, mlisp8.exe and alisp8.exe on Windows). The build-executable keyword argument to build-lisp-image specifies the executable that will be used to create the new image. The value of that argument defaults to the executable used to start the running image.

Therefore, if you want the character size of the new image to be the same as the character size in the running image, you need not specify a value for the build-executable keyword argument. If you want a different character size, specify an appropriate executable as the value of that argument.

For example, if you are running a 16-bit character size image and you want to build an 8-bit character size image, call build-lisp-image like this:

(build-lisp-image "my-image.dxl"
   :build-executable "mlisp8" ;; On Windows "mlisp8.exe"
   ;; < other keyword arguments and values >
  )

(That call also works if you are running an 8-bit executable already, of course.)

For example, if you are running an 8-bit character size image and you want to build a 16-bit character size image, call build-lisp-image like this:

(build-lisp-image "my-image.dxl"
   :build-executable "mlisp"  ;; On Windows "mlisp.exe"
   ;; < other keyword arguments and values >
  )

(That call also works if you are running a 16-bit executable already, of course.)

Note: Unless you have created your own executable (which is uncommon, but see main.html, all 16-bit character size executables are identical and all 8-bit character size executables are identical. Therefore, any executable of the correct character size may be specified.


4.0 Arguments to build-lisp-image 1 - defaults inherited from the running image

Certain defaults for keyword arguments to build-lisp-image are inherited from the currently running image. It is important to understand that these inheritances are the only effect that values in the currently running image have on the new image being built.


5.0 Arguments to build-lisp-image 2 - defaults not inherited from the running image

Most of the remaining keyword arguments are listed in the next table. The keyword arguments to dumplisp are also acceptable to build-lisp-image. None of these arguments inherit values from the currently running image.

Uninherited Arguments Note 1: Finding available memory addresses

To successfully allocate the heap, you will need to move the starting address of the Lisp heap to a location large enough to support a contiguous address range specified by the heap size you chose. If you evaluate the following forms in Allegro CL, a memory map of the current state of virtual memory on your machine will be printed to the file filename.ext (except on the IBM RS/6000 where the output file will say only that the information is not available). Note that you may use any filename and extension.

(ff:def-foreign-call memory_status_dump ((filename (* :char)))
    :arg-checking nil :strings-convert t)

(memory_status_dump "filename.ext")

The output differs for different platforms, but in all (except the RS/6000) cases, a set of address ranges is provided showing what ranges are allocated.

The value of the filename may be 0, causing the information to be printed to the terminal (on Windows, if the filename argument is 0, then the Console is used as output). You may get a warning if you specify 0 as the filename argument. It can be ignored.

Permissions on Windows are shown in the last four characters in the output:

You will need to locate a large chunk of free memory and specify to build-lisp-image a starting address that will support your heap size.

Uninherited Arguments Note 2: :close-oldspace argument details

The default value of the close-oldspace keyword arguments in nil. If its value is specified true, the following is evaluated just before the image being built is created (newspace is the value of the :newspace keyword argument described above, oldspace is the value of the :oldspace keyword argument also described above):

(sys:resize-areas
            :old 0
            :new (- (/ *newspace* 2) (* 1025 50))
            :global-gc t
            :tenure t)
(sys:resize-areas :old *oldspace* :sift-old-areas nil)
(setf (sys:gsgc-parameter :open-old-area-fence) -1)

See:

As described in gc.html, closed old areas are never gc'ed, so objects in them and objects they point to are never considered garbage. The idea is that the image being created is an application, and the application machinery is being loaded. That machinery will never become garbage, so it saves time if the garbage collector never looks at it. Data to be used by the application is loaded after the image is created. That data will be tenured to the open old areas, and thus will be garbage collected. This allows faster global gc's to clear out data sets when they are no longer needed.

Uninherited Arguments Note 3: specifying starts and sizes for heaps and old and new space

The arguments aclmalloc-heap-start, aclmalloc-heap-size, initial-newspace, initial-oldspace, lisp-heap-start, lisp-heap-size, newspace, and oldspace all take a number of bytes as a value. You can express this number as a positive integer or as a string which is a decimal or hex integer optionally followed by the letter k, m, or g (case does not matter). k means multiple the integer by 1024 (i.e. kilobytes), m by 10241024 (i.e. megabytes), and g by 10241024*1024 (i.e. gigabytes). Hex values should be prefixed by "0x" or #x". So 1048576, "1048576", "1024K", "0x400k" and "#x400K" are all the same as "1m"; "168577466368", 168577466368, "164626432k", "0x9d00000K", and "160768m" are all the same a "157g". We recommend using letter modifiers for large values as sometimes very large integers can overflow when read.

Table Note 4: The value of :splash-from-file

The value of the :splash-from-file argument, if non-nil, must be a string naming a BMP file. There are other image file formats (jpg, png, etc.) but the Windows system code (called by Allegro CL) which displays the spash screen is hardwired for BMP files only. Non-BMP image file can be converted to BMP image files in Common Graphics (see load-pixmap and save-pixmap. Other programs outside of Allegro CL can also make that conversion.


6.0 Arguments to build-lisp-image 3: defaults taken from environment variables

build-lisp-image will take certain defaults from the values of environment variables if these are set when build-lisp-image is called. The following table shows the arguments and the name of the associated environment variable.

All these arguments do not inherit values from the running Lisp image (the one which calls build-lisp-image). See Arguments to build-lisp-image 2 - defaults not inherited from the running image above for more information on these argument and their default values.

If an environment variable has a value, it overrides a specified value in the call to build-lisp-image. This allows having different values then called for in scripts without modifying the script.

Argument Associated Environment Variable
:aclmalloc-heap-size ACL_BUILD_ACLMALLOC_HEAP_SIZE
:aclmalloc-heap-start ACL_BUILD_ACLMALLOC_HEAP_START
:initial-newspace ACL_BUILD_INITIAL_NEWSPACE
:initial-oldspace ACL_BUILD_INITIAL_OLDSPACE
:lisp-heap-size ACL_BUILD_LISP_HEAP_SIZE
:lisp-heap-start ACL_BUILD_LISP_HEAP_START
:newspace ACL_BUILD_NEWSPACE
:oldspace ACL_BUILD_OLDSPACE

7.0 Debugging an image build problem or failure

Starting in Allegro CL 6.2, the method for debugging a failed or problematic image build has been simplified, and unified for all platforms. (In earlier releases, behavior was different for Windows and UNIX.) As part of the changes, the debug-on-error, exit-after-image-build, and wait keyword arguments to build-lisp-image (and therefore to generate-application) have been removed. An error is signaled if they are specified. In their place is a new keyword argument build-debug. In this section, we discuss that argument and the general issue of debugging builds.

build-lisp-image spawns another Lisp to build the desired image (that is one reason why the new image does not inherit from the calling image). If there is a problem, it is this spawned image that one wants to debug. Debugging the image in which you called build-lisp-image or generate-application is not useful because the problem is not manifested in that image.

The problem is usually caused by a Lisp error being signaled. If the spawned image exits at that point, it will exit with a non-zero status. As we describe just below, specifying the build-debug keyword argument as :interactive causes the spawned image not to exit and allows interactive debugging.

It is possible that a warning will be signaled during the build. Warnings will not typically cause the build to fail, but are legitimate causes for concern. If you want to debug a warning, call build-lisp-image or generate-application specifying :pre-load-form '(setf *break-on-signals* t). This will cause an error when the warning is signaled.

The choices for debugging are:

Here are examples using the various values of build-debug. The file foo.cl includes the form at the top-level (setq x y), but y does not have a value.

It is possible that the spawned image will fail without signaling a Lisp error. In that case, it will also exit with a non-zero status, but interactive debugging is not, of course, possible. However, this is rare. If it happens, please save any messages that are printed and try the build again (to ensure that it was not a transient problem that caused the failure). If the retry fails, send a bug report to support@franz.com.

The call that invokes the spawned process and input to it

To get maximal information about the invocation of the spawned process caused by your call to build-lisp-image (or generate-application add the arguments:

:build-input "input.txt" :verbose t :build-output "output.txt"

"input.txt" and "output.txt" are filenames -- any filename with any valid path will do, of course. Here is what the build-lisp-image form looks like:

(excl:build-lisp-image <image-file> 
                       :build-input "input.txt" :verbose t 
                       :build-output "output.txt" 
                       <your arguments>)

This causes the actual command starting the Lisp that builds the new dxl (along with other stuff) to be printed to the listener where the build-lisp-image command was issued.

Generated (by the :build-input argument) is the file "input.txt" which contains the Lisp forms passed to the spawned Lisp. It looks roughly like:

(excl:set-case-mode :case-sensitive-lower) (common-lisp:force-output) 
(common-lisp:progn (common-lisp:setq excl::*store-documentation* common-lisp:nil) 
(common-lisp:setq excl:*record-source-file-info* common-lisp:nil) 
(common-lisp:setq excl:*load-local-names-info* common-lisp:nil) 
[...]

Generated (by the :build-output argument) is the file "output.txt" which contains essentially a dribble output from the spawned process.


8.0 Use of custom.cl

The file sys:custom.cl in the Allegro directory is loaded into the new image at the end of the building process but just before the files specified by :lisp-files are loaded. If you are running generate-application, sys:custom.cl is loaded before the files specified by the (required) input-files argument are loaded.

sys:custom.cl, as delivered, contains various things, either commented out or marked with #+ignore. For example, certain forms setting values to the defaults used in previous versions of Allegro CL are provided, preceded by #+ignore so they will not be read unless the #+ignore is removed. (See the section Features present or missing from *features* in Allegro CL in implementation.html for information on #+ignore.)


9.0 Building an image to include patches

During the build, all patch files in [Allegro Directory]/update relevant to the products included in the image (as coded by the filenames) are loaded into the image during the build.


10.0 Minimal top levels

When building a lisp image, specifying nil for include-tpl will cause a greatly reduced top-level functionality to be built into the lisp. The sole purpose of this minimal top-level is to reduce the space used by the full top-level. Applications which do not require a top-level or which provide their own will often specify include-tpl nil. The minimal top level described here will be available in such images.

The entire text of the minimal top level functionality is given below. Note that this code is loaded only if include-tpl is nil. The top-level code when include-tpl is true is quite different.

(defpackage :top-level
  (:nicknames :tpl)
  (:use :common-lisp :excl)
  (:size 20)
  (:import-from :excl excl::read-eval-print-loop)

  ;; These are the function handlers for the top-level commands.
  ;; They are user visible.
  (:export #:*read-eval-print-loop*     ; user-defined read-eval-print-loop
           #:*read*                     ; the top-level reader
           #:*eval*                     ; the top-level evaler
           #:*print*                    ; the top-level printer
           ))

(provide :tpl-user)

(in-package :top-level)

;; simple default tpl handlers:
(defvar *read-eval-print-loop* 'default-read-eval-print-loop)
(setq *read* 'read)
(setq *eval* 'eval)
(setq *print* 'print)

(declaim (special *break-level*))
(setq *break-level* 0)

(defun start-interactive-top-level (*terminal-io*
                                    function args
                                    &key initial-bindings
                                    &aux vars vals)

  (declare (:discard-source-file-info))
  (setf (getf (excl::stream-property-list *terminal-io*) 'initial-listener)
    sys::*current-process*)

  ;; Compute the list of special variables and bindings for progv.
  (dolist (b initial-bindings)
    (unless (member (car b) vars :test #'eq)
      (push (car b) vars)
      (push (eval (cdr b)) vals)))
  (progv vars vals
    (setq vars nil vals nil)            ;free up space
    (apply function args)))

(defun top-level-read-eval-print-loop ()
  (declare (:discard-source-file-info))
  (loop
    (setq *evalhook* nil *applyhook* nil)
    (catch ':top-level-reset (read-eval-print-loop :level 0))
    ))

(defun read-eval-print-loop (&key &allow-other-keys)
  (declare (:discard-source-file-info))
  (let (pop-type cval1 cval2)
    (loop
      (multiple-value-setq (pop-type cval1 cval2)
        (catch 'top-level-break-loop
          (funcall *read-eval-print-loop*)))
      ;; If we get here and pop-type is not null, then a throw
      ;; to 'top-level-break-loop was done (by a different toplevel)
      (case pop-type
        ((:pop :debug-pop)
         (when (plusp cval1)
           (excl::funcall-in-package :debug-pop :debugger
                                     nil (1- cval1) (1- cval2))))
        (error "user toplevel can't handle this pop type: ~s" pop-type)))))


(defun default-read-eval-print-loop ()
  (loop
    ;; print the prompt
    (fresh-line *terminal-io*)
    (princ "// " *terminal-io*)
    (force-output *terminal-io*)
    (let* ((exp (funcall *read*))
           (res (funcall *eval* exp)))
      (funcall *print* res *terminal-io*))))

10.1 Using the default minimal top-level

The minimal top-level is set up by default to issue a "//" prompt. It only accepts lisp evaluable expressions as "commands", and does not interpret any other top-level commands.

Example on a sparc:

$ mlisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl70/src/libacl70pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 7.0
Copyright (C) 1985-2004, Franz Inc., Oakland, CA, USA.  All Rights Reserved.
// (room)
area  address(bytes)        cons        symbols        other bytes
                        8 bytes each  24 bytes each
                        (free:used)   (free:used)      (free:used)
Top #x81dc000
New #x8180000(376832)     918:3158      254:0         246536:37944
New #x8124000(376832)      -----         -----            -----
Old #x8000c40(1192896)    781:15523     135:5628      518896:397344
Root pages: 34
Lisp heap limit: 67108864


NIL 
// (exit)
; Exiting Lisp
$

10.2 Requiring the normal top-level in a minimal top-level lisp

If the power of the normal top-level is needed after a (non-runtime) minimal top-level lisp is built, :toplevel can be required. However, simply requiring :toplevel is not enough to start the regular top-level listener; instead, the listener must be invoked recursively, either by an error or by any command (such as inspect) that starts a new listener level. At that time, top-level commands (such as :zoom, etc) can be invoked.

Note however that when :toplevel is required, the read-eval-print-loop is reset, and so a :reset command will make it appear as if the Lisp had always had a normal top-level.

Example 1:

;; In this example, an error can be debugged after the fact by requiring
;; the normal top-level.

$ mlisp -I umsloadxcomp.dxl -qq
Loading /release/duane/acl70/src/libacl70pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 7.0
Copyright (C) 1985-2004, Franz Inc., Oakland, CA, USA.  All Rights Reserved.
// (require :toplevel)
; Fast loading /acl70/src/code/toplevel.fasl
;   Fast loading /acl70/src/code/frame.fasl
;     Fast loading /acl70/src/code/r/rframe.fasl

T 
// a
Error: Attempt to take the value of the unbound variable `A'.
  [condition type: UNBOUND-VARIABLE]

Restart actions (select using :continue):
 0: Try evaluating A again.
 1: Use :A instead.
 2: Set the symbol-value of A and use its value.
 3: Use a value without setting A.
[1] USER(1): :zo
; Autoloading for TOP-LEVEL::ZOOM-COMMAND:
; Fast loading /acl70/src/code/tpl-debug.fasl
; Autoloading for package "DEBUGGER":
;   Fast loading /acl70/src/code/debug.fasl
Evaluation stack:

 ->(EXCL::INTERNAL-INVOKE-DEBUGGER "Error" #<UNBOUND-VARIABLE @ #x81b9f8a> ...)
   (ERROR #<UNBOUND-VARIABLE @ #x81b9f8a>)
   (SYS::..CONTEXT-SAVING-RUNTIME-OPERATION)
   (EVAL A)
   (TPL:TOP-LEVEL-READ-EVAL-PRINT-LOOP)
   (TPL:START-INTERACTIVE-TOP-LEVEL #<BIDIRECTIONAL-TERMINAL-STREAM [initial terminal io] fd 0/1 @ #x80439ba>
                                    #<Function TOP-LEVEL-READ-EVAL-PRINT-LOOP @ #x804d22a> ...)

(to see any ghost frames, the disassembler must be loaded)
[1] USER(2): :res
USER(1): 

Example 2:

;; Note in this example a variable is inspected after setting it,
;; to indicate the state of the lisp before the new top-level is
;; pulled into the lisp.  Note also that we wrap a progn which will
;; return a final nil value, so as not to see a huge printout due to
;; the lack of *print-level*/*print-length* controls.

$ mlisp -I umsloadxcomp.dxl -qq
Loading /acl70/src/libacl70pf23.so.
Mapping umsloadxcomp.dxl...done.
Mapping umclxcomp.pll.
Allegro CL 7.0
Copyright (C) 1985-2004, Franz Inc., Oakland, CA, USA.  All Rights Reserved.
// (progn (setq x (excl::get-objects 7)) nil)

NIL 
// (require :toplevel)
; Fast loading /acl70/src/code/toplevel.fasl
;   Fast loading /acl70/src/code/frame.fasl
;     Fast loading /acl70/src/code/r/rframe.fasl

T 
// (inspect x)
; Autoloading for INSPECT:
; Fast loading /acl70/src/code/inspect.fasl
A simple T vector (5649) @ #x81a506a
   0-> fixnum 5629 [#x000057f4]
   1-> The symbol CL:NIL
   2-> The symbol T
   3-> The symbol EXCL::ER-WNAERR
   4-> The symbol EXCL::ER-GENERAL-ERROR-HANDLER-ZERO
   5-> The symbol EXCL::ER-GENERAL-ERROR-HANDLER-ONE
   6-> The symbol EVAL
   7-> The symbol EXCL::INTERPRETED-FUNCALL
   8-> The symbol EXCL::+_2OP
   9-> The symbol EXCL::GC-AFTER
  10-> The symbol EXCL::*WITHOUT-INTERRUPTS*
  11-> The symbol EQUAL
  12-> The symbol *PACKAGE*
  13-> The symbol *LISP-PACKAGE*
  14-> The symbol *KEYWORD-PACKAGE*
  15-> The symbol EXCL::INTERN*
  16-> The symbol EXCL::FASL-FIND-PACKAGE
  17-> The symbol EXCL::CONVERT-TO-INTERNAL-FSPEC
  18-> The symbol *COMPILER-PACKAGE*
  19-> The symbol *SYSTEM-PACKAGE*
  20-> The symbol EXCL::CONVERT-TO-EXTERNAL-FSPEC
  21-> The symbol SYS::LISP-BREAKPOINT
  22-> The symbol EXCL::HANDLE-PENDING-SIGNAL
  23-> The symbol EXCL::SET-FUNCTION
  24-> The symbol EXCL::.INV-MACRO-FUNCTION
   ...
 5648-> The symbol NIL
[1i] USER(1): 

10.3 Top-level variables

The following variables are maintained or used by the minimal top-level:

If *read-eval-print-loop* is set to a value other than tpl::default-read-eval-print-loop, then the three read/eval/print variables are not used. Normally, this variable should not be set unless it is desired to remove all possible user interaction with lisp. If a replacement top-level is supplied, it is recommended that all possible errors be handled explicitly with handlers.

The top-level variables in the table with links are also discussed in Top-level variables in top-level.html.


Copyright (c) 2023, Franz Inc. Lafayette, CA., USA. All rights reserved.

ToC DocOverview CGDoc RelNotes FAQ Index PermutedIndex
Allegro CL version 11.0