Dumplisp

This document contains the following sections:

1.0 Introduction to excl:dumplisp, a tool for saving an image

The function dumplisp writes an image file from the current running Lisp image. It has many options and many uses. Unlike the two other functions that create new images, build-lisp-image and generate-application, dumplisp preserves much of the environment of the running image. Both build-lisp-image and generate-application create new images from external constituent parts that inherit nothing from the running image.

If you are building images for general use (for example, an image incorporating an application), we recommend that you use build-lisp-image or generate-application rather than dumplisp as those functions give more control over the resulting images and unnecessary functionality which may be in your running image will not be included.

A new dumplisp feature on Windows only (added by a post-9.0 patch) allows for what is essentially a pseudo-fork operation on Windows and is useful for very large images. See Section 21.0 Virtual dumplisp: for very large images and pseudo-forks on Windows for more details.

2.0 Finding additional files

One advantage of the small executable/large image model is that all image files use the same executable, so there needs only be one executable file on a machine (or on a network). This means that the location of that executable can be used as a starting location which can be a reference point for finding other necessary files.

The location of the executable is taken as the reference point for finding files so no specific location must be supplied when Lisp starts up. (You can specify a different location with the -H argument if you wish. However, that should not be necessary.)

3.0 Uses of excl:dumplisp

You might use dumplisp for the following purposes (as well as others not listed):

• Saving certain parts of the state of a running image. dumplisp cannot save the entire state. Thus processes cannot be saved (neither their state nor even the fact that they were running) nor can OS connections such as open files and databases, but other things, like variable values, function definitions, macro definitions etc. are all saved.
• Customizing an image. You may dump an image after, for example, loading a large application (which you intend to use yourself). Then you do not have to spend time loading the application each time you start Lisp. You can also dump an image after defining functions and setting variables to your liking, but simple things like that are usually better done with initialization files (like ~/.clinit.cl). See Section 7.0 Creating a customized image below.
• Preparing an application for delivery. We generally recommend using excl:generate-application for this purpose but dumplisp can be used as well. The dumped image may be intended for delivery to others (perhaps at different sites or running on different networks). See delivery.htm where delivery issues are discussed in greater detail. Note that you must have the correct license before you can ship an image of any sort away from your site. Please contact the correct administrator at your site or your Franz Inc. account manager if you have questions about what you are licensed to do with dumped images.

4.0 excl:dumplisp will fail under some conditions

In order for dumplisp to work, it must examine the currently running image. For this reason, dumplisp will fail if:

• The :name argument to dumplisp names the currently running image. Thus, if the current image is /usr/mydir/cl.dxl and you specify the value of name to be "/usr/mydir/cl.dxl", dumplisp will get confused between the image file it is examining and the image file it is writing and fail.
• The image that is currently running is removed or moved (with mv).
• Lisp was started with a relative pathname (./cl, for example) and chdir or one of its top-level command counterparts :pushd, :popd, and :cd has since changed the current working directory. Since relative pathnames are resolved with respect to the current directory, the image will not be found with respect to the changed directory. If you change back to the original working directory, dumplisp should work.

4.1 dumplisp error: too many regions

dumplisp will fail if there are too many oldspaces. The limit depends on a complex formula, but is around 2000. You can see how many oldspaces there are by calling (room t). You can consolidate oldspaces with sys:resize-areas. The error message when there are too many is Error: Too many regions. Consolidate oldspace with sys:resize-areas and retry if you get this error.

5.0 [allegro directory]/src/aclstart.cl is the source code for startup

Allegro CL does a number of things when it starts up before it prints the first prompt and is ready to accept input. Programmers may want to know what it does and in what order, so that programmer modifications and customizations can be done at the right time. Allegro CL calls the function excl::start-lisp-execution on startup. The source for that function and for several associated functions (that process command-line arguments and read initialization files, e.g.) can be found in the file sys:;src;aclstart.cl, equivalently [Allegro directory]/src/aclstart.cl. See also startup.htm where startup is described in English rather than Lisp code.

6.0 Creating an application

Please see delivery.htm for a detailed discussion of creating images for delivery.

7.0 Creating a customized image

A typical customization is loading a complicated program and thus saving the time to load whenever you start Lisp (but quicker startup should be balanced against additional disk space usage).

8.0 When the dumped image starts 1: values of global variables

When a Lisp Listener (which presents the prompt to users and accepts and processes typed input) starts up, bindings are set up for many Common Lisp and Allegro CL-specific global variables. The bindings come from alists, one of which is the value of *cl-default-special-bindings*. Here is part of *cl-default-special-bindings*:

;; From excl:*default-cl-special-bindings* (a few values only)
;; (*PRINT-LENGTH*) is equivalent to (*PRINT-LENGTH* . NIL).
(*PRINT-LENGTH*) (*PRINT-LEVEL*) (*PRINT-RADIX*) (*PRINT-BASE* . 10)
(*PRINT-PRETTY* . T) (*PRINT-ESCAPE* . T)

Note that each variable in the portion of the alist shown is associated with a specific value (nil for *print-length*, *print-level* and *print-radix*, 10 for *print-base*, and t for *print-pretty* and *print-escape*). When a Lisp listener is started in the dumped image, *print-length* in that listener will be nil regardless of its value when dumplisp was called, as this example shows:

;; We bring up Allegro CL and note that *print-length* has initial value
NIL.
USER(1): *print-length*
NIL
;; We set the value of *print-length* to 20:
USER(2): (setq *print-length* 20)
20
USER(3): *print-length*
20
;; and we dump the image:
USER(4): (excl:dumplisp :name "my-image")
USER(5): (exit)
;; Now we start my-image:
==============================================================
Starting image `my-image'
[...]
;; Note that *print-length* is NIL, not 20:
USER(1): *print-length*
NIL
USER(2):

The listener in the restarted image gets the value of *print-length* from the *cl-default-special-bindings* alist, where *print-length* is associated explicitly with nil. The value in the parent image is not relevant. In order to ensure that *print-length* (or any variable whose name appears on either alist) has the desired value in the listener when the dumped image is started, you must modify the appropriate alist. The easiest way to do this is with tpl:setq-default. This macro is like setq: its first argument should be a symbol and is not evaluated; its second argument should be a value and is evaluated. It modifies the alist appropriately (see the definition for a complete description) but does not affect the current value. Thus:

;; We bring up Allegro CL and note that *print-length* has initial value
NIL.
USER(1): *print-length*
NIL
;; We use TPL:SETQ-DEFAULT to modify the EXCL:*CL-DEFAULT-SPECIAL-BINDINGS*
;; entry referring to *PRINT-LENGTH* so new bindings will be to 20, not NIL:
USER(2): (tpl:setq-default *print-length* 20)
20
USER(3): *print-length* ;; its value in the current listener is unchanged
NIL
;; and we dump the image:
USER(4): (excl:dumplisp :name "my-image")
USER(5): (exit)
;; Now we start my-image:
==============================================================
Starting image `my-image'
[...]
;; Note that *print-length* is 20, not NIL:
USER(1): *print-length*
20
USER(2):

Recall that changes to the reader (such as setting a macro-character) are stored in *readtable* and that variable appears on *cl-default-special-bindings*.

This same issue affects setting values from an initialization file. A discussion very similar to this one appears in startup.htm.

9.0 When the dumped image starts 2: command-line arguments

Command-line arguments are processed as usual or ignored as the ignore-command-line-arguments keyword argument to dumplisp is nil (the default) or t. The emacs-lisp interface uses command-line arguments to start up. See below under the heading Section 13.0 The emacs-lisp interface and dumped images for information on starting the interface when command-line arguments are ignored.

Command-line arguments are available, of course, with sys:command-line-arguments and related functions. They are just ignored by Allegro CL's startup routine when ignore-command-line-arguments is true.

10.0 When the dumped image starts 3: reading init files

There are various initialization files that might be read: ~/.clinit.cl, [working-directory]/.clinit.cl (if [working-directory] is different than ~), ~/ and [working-directory]/clinit.cl, and sys:siteinit.cl. These files will be looked for if *read-init-files* is t when the image is dumped (just [working-directory]/.clinit.cl and clinit.cl and sys:siteinit.cl, if *read-init-files* is :no-home). This variable should be set (not bound) prior to the call to dumplisp. Setting this variable after Lisp has started up only affects its value in dumped images. It is not used by Lisp after startup has completed.

Command-line arguments can suppress the reading of some or all initialization files (see startup.htm). Since command-line arguments are processed before initialization files are read, the value of this variable can be changed using them -- e.g. specify `-e (setq excl:*read-init-files* t)' on the command-line -- assuming command-line arguments are not ignored.

11.0 When the dumped image starts 4: restart actions

The variable *restart-actions*, if true, should be a list of functions of no arguments (function names or function objects are acceptable). These functions are called in order, after the command-line arguments are processed and the init files are read. The purpose of this variable is to do system initializations. Do not set this variable or you may eliminate values put on the list by Allegro CL and its associated products (CLIM, for example, uses this variable). Instead, use push or pushnew to add items if desired.

Programmers are in fact discouraged from using this variable. *restart-init-function* and *restart-app-function* (both described under the next heading) are better suited to application code while *restart-actions* is best left to Allegro CL and its associated products. Both *restart-init-function* and *restart-app-function* are processed at the end of the startup procedure. *restart-actions* is processed early in the procedure, too early for certain actions (like starting CLIM) to succeed.

12.0 When the dumped image starts 5: the two restart functions

At the end of the startup procedure, Allegro CL examines the variable *restart-init-function*. If it is true, its value is assumed to be a function of no arguments (either a function name or a function object) and that function is called. The function typically will return. It is designed for late initializations (perhaps based on information supplied by initialization files), possibly cleanup of state from the pre-dumplisp image, application tasks like printing an information message or an application banner, reading an application-specific initialization file, processing application command-line arguments, etc.

When the *restart-init-function* returns, Allegro CL examines *restart-app-function*. If it is nil, a standard Lisp listener is started. If it is true, its value is assumed to be a function of no arguments (either a function name or a function object) and that function is called. The function should not return (the behavior if it does return is undefined.) *restart-app-function* is designed to provide an application top-level when the application does not want to use the standard Lisp listener as a top-level.

There are two restart functions specifically to divide the two purposes of a restart function in earlier versions of Allegro CL: application-specific initializations and providing an application top-level. Note that, a listener is started if *restart-app-function* is nil but not when *restart-app-function* returns (indeed, the behavior if *restart-app-function* returns is undefined). Note that *restart-app-function* can start its own Lisp listener by evaluating the following form:

(tpl:start-interactive-top-level *terminal-io*
   #'tpl:top-level-read-eval-print-loop nil)

See the definitions of tpl:start-interactive-top-level and tpl:top-level-read-eval-print-loop.

13.0 The emacs-lisp interface and dumped images

This issue is also discussed in eli.htm. The Emacs-Lisp interface provides a close coupling of Allegro CL and Emacs. What is important with respect to dumped images is how the interface is started. It can only be started by invoking Allegro CL within Emacs with the function fi:common-lisp. Images invoked in that way cause the Emacs-Lisp interface to be started with the command line arguments "-e (excl::new-start-emacs-lisp-interface t)" (plus any additional command line arguments you specify). This will be ineffective if processing of those command line arguments is suppressed because the image was dumped with the ignore-command-line-arguments argument to dumplisp specified true.

If you create an image which ignores command-line arguments and you want the emacs-lisp interface started, you must take responsibility for evaluating (excl::new-start-emacs-lisp-interface t). This could be done as part of the *restart-init-function*.

There is one minor complication: you may want to handle the case where you are starting the image in some fashion other than calling fi:common-lisp within Emacs (for example, starting Lisp in a shell). In that case, calling excl::new-start-emacs-lisp-interface will not, of course, do anything effective. Calling the function is not an error. However, it does print a cryptic string to the listener, which you may want to avoid. The solution to that problem is, instead of unconditionally calling excl::new-start-emacs-lisp-interface, to look yourself for "-e (excl::new-start-emacs-lisp-interface t)" among the command line arguments and only call excl::new-start-emacs-lisp-interface if it appears. The following code (which actually simply looks for the first -e argument and evaluates its companion argument) will do that:

(let ((e-arg (member "-e" 
                     (system:command-line-arguments :application nil)
                     :test #'string=)))
  (when e-arg
    (eval (with-standard-io-syntax 
            (read-from-string (cadr e-arg))))))

14.0 Dumping a prestoized image

In earlier releases, there were issues with dumping an image using the Allegro Presto feature and with some functions partially loaded. In 7.0, the Allegro Presto facility has been removed so interaction between dumplisp and Allegro Presto is no longer an issue. See The Allegro Presto facility has been removed in loading.htm for further information.

15.0 Standalone image: not supported

Because Allegro CL now requires an executable file and an image file, standalone images (images that depend on no other files) are no longer possible.

16.0 How the dumped image finds loaded library files

Foreign loading is done on Unix by `loading' .so or .sl files with dlopen() or an equivalent and on Windows by loading .dll files. Dumped images must find these files when it starts up.

The pathnames of loaded .so files are stored. So long as those pathnames are valid when the image is restarted, things should work correctly. However, to guarantee that the dumped image finds the .so files when it restarts, even if run on a different machine or network, you can use one of the following methods.

1. Use pathnames that will be valid when the dumped image restarts (and, as we describe, logical pathnames can be made valid in the dumped image); or
2. Unload the .so files prior to dumplisp and reload them as part of the startup procedure (perhaps as an action taken by *restart-init-function*). We describe both methods in some detail.

Using logical pathnames and making them valid in the dumped image. Lisp stores the pathnames of all loaded .so files as they are specified to load (or :ld). It uses these stored pathnames when a dumped image is restarted to find the files and reload them. We recommend these steps for loading .so files. Because logical pathnames are used and because a translations file is specified (~/myhosts), all that is needed is to ensure that file contains the correct translation for the logical host.

1. Specify the needed files to be loaded with logical pathnames.
2. Modify logical-pathname-translations-database-pathnames before dumping the image so that some file under your control (~/myhosts, e.g.) is examined along with sys:hosts.cl.
3. Make sure that file has the correct translation for the logical pathnames used to identify the .so files.

For example, suppose you want to load /usr/mydir/mysos/foo.so into an image that will be dumped. The following transcript shows the actions in the pre-dumped image and modifications to the external logical hosts file.

USER(1): (setf (logical-pathname-translations "myso")
           '((";**;*.*"
#p"/usr/mydir/mysos/")))
((#p";**;*.*" #p"/usr/mydir/mysos/"))
USER(2): :ld myso:b.so
; Foreign loading myso:b.so.
USER(3): (push "~/myhosts" (logical-pathname-translations-database-pathnames))
("~/myhosts" "sys:hosts.cl")
USER(4): (dumplisp :name "mycl")
Warning: shared object files have been loaded into this image, and so the 
resulting image will depend on myso:b.so for successful operation.
USER(5):
;; ~/myhosts contains:
"myso" '(";**;*.*" #p"/usr/mydir/mysos/")

Unloading all .so files and reloading them upon startup. In this example, we unload the (in our case single) .so file with ff:unload-foreign-library and show how to modify *restart-init-function* so the .so file is reloaded. We end with a couple of notes and warnings.

USER(1): :ld /usr/mydir/mysos/b.so
; Foreign loading /usr/mydir/mysos/b.so.
USER(2): (ff:unload-foreign-library "/usr/mydir/mysos/b.so")
NIL
USER(3): (defun reload-my-so () (load "/usr/mydir/mysos/b.so"))
RELOAD-MY-SO
USER(4): (setq excl:*restart-init-function* 'reload-my-so)
USER(5): (dumplisp :name "mycl")
;; NOTE: no warning about loaded .so files.
USER(6):
;; The new image comes up as follows (note the message about foreign loading):
Allegro CL 

Copyright (C) Franz Inc., Berkeley, CA, USA. All Rights Reserved.
;; Starting socket daemon and emacs-lisp interface...
; Foreign loading /usr/mydir/mysos/b.so.
USER(1): 

Notes and warnings:

1. Certain Allegro CL utilities may dynamically load .so files. (CLIM does if loading into a running image rather than being built into the image, e.g.). These files are specified with logical pathnames with host sys:, so they should be found without programmer intervention.
2. The *restart-init-function* in our example is very simple and not very useful. In real situations, programmers would likely get information from a command-line argument or an initialization file about the location of the needed files and act on that information.
3. The files must be found and reloaded before any foreign function that requires them is called.

17.0 Logical pathnames and the dumped image

All logical pathname translations are flushed as the first action when a dumped image restarts. This is a feature. There is no way for Lisp to tell whether the translations are valid upon restart (and they almost certainly will not be valid if the image is restarted on a different machine on a different network). Rather than devise a way to communicate to Lisp whether to flush translations on startup (and thus overload the few ways available to communicate with restarting images), Allegro CL instead provides tools for properly re-establishing translations using specific files.

The function logical-pathname-translations-database-pathnames returns a list of pathname namestrings. Initially, it returns the list ("sys:hosts.cl"). You may use push or pushnew to add other strings naming files. For example,

USER(5): (logical-pathname-translations-database-pathnames)
("sys:hosts.cl")
USER(6): (pushnew "~/myhosts"
(logical-pathname-translations-database-pathnames))
("~/myhosts" "sys:hosts.cl")
USER(6): (logical-pathname-translations-database-pathnames)
("~/myhosts" "sys:hosts.cl")

When looking for a translation for a logical pathname, Lisp will look first in ~/myhosts and then in sys:hosts.cl. It is not an error if a named file does not exist. Therefore, your translations should be added to hosts.cl in the Allegro directory or to a file of your own which you push onto logical-pathname-translations-database-pathnames as described above. (Many users of Lisp may not have permission to modify hosts.cl. Indeed, having several users modifying that file is impractical.)

18.0 How the dumped image finds its Allegro directory

The Allegro directory is the location of the executable file (such as mlisp or mlisp.exe). In the separate executable/image model, finding the directory using the executable is simple and should work without problems.

19.0 dumplisp and multiprocessing

When lisp starts from a dumped image, it has exactly one process that is runnable, the Initial Lisp Listener process, and that is a new process, completely unrelated to the Initial Lisp Listener process that was running when the image was dumped.

Any other processes that were in the image when it was dumped must be process-reset before being run again. It would be legal to process-kill any or all of them instead of resetting and restarting them.

20.0 How to use the dumped image

The image is specified to the executable with the -I argument, Thus, if the dumped image is named myimage.dxl, and mlisp.exe or mlisp is the executable, then this starts a Lisp using myimage.dxl:

[Windows]
% mlisp.exe [+ args if any] -I <path>/myimage.dxl [other args]
[Unix]
% mlisp -I <path>/myimage.dxl [other args]

If no -I argument is specified, the executable looks for an image in the same directory with the same name as itself (thus mlisp.exe will look for mlisp.dxl). If you have created an image for general use, you may wish to copy it to the Allegro directory (assuming you have permission to do so) and copy the executable to have the same name.

21.0 Virtual dumplisp: for very large images and pseudo-forks on Windows

dumplisp on Windows only recognizes a new keyword argument virtual. If specified and non-nil, it requests dumplisp to create a two-piece lisp image. The file named by the name keyword argument will be one piece and will be relatively small. Most of the dumped image will be in the second piece, a memory-mapped virtual file. The real file contains a reference to the virtual file.

Example:

   (dumplisp :name "vapp.dxl" :virtual t)

will write a relatively small file named vapp.dxl and create a large virtual file holding most of the image data. vapp.dxl will have a symbolic pointer to the virtual file. As long as the virtual file exists, executing the shell command

    lisp -I vapp

will get its image data from the virtual file.

One important issue is "How long does that virtual file exist"? The answer is that it exists as long as there is a Lisp program using it. The Lisp program that did the virtual dumplisp is a initially the only user. When it exits, it will no longer be a user, obviously, and the virtual data will vanish unless there is another user. A Lisp that is loading data from a virtual image by copying it is a user as long as it is loading the data. Once the data has been copied, that Lisp program is no longer a user. A Lisp program that maps the data rather than copying it will be a user until it exits.

A Lisp program that creates a virtual image file can release the virtual file before exiting, and may wish to do so to release the OS resources dedicated to supporting the virtual file. To do so it must capture the result returned from dumplisp and then call excl:release-dumplisp-virtual-image when it wishes to discard the file.

dumplisp returns no values when it is called with :virtual nil or unspecified. When called with :virtual non-nil, dumplisp returns a value that can later be used to release the virtual file. One usage pattern would be something like this:

(let ((vmem (dumplisp :name "vapp.dxl" :virtual t)))
  (run-shell-command "lisp -I vapp.dxl -- initial")
  (run-shell-command "lisp -I vapp.dcl -- cleanup")
  (excl:release-dumplisp-virtual-image vmem))

This runs one Lisp from the image with initial as a command line argument, then when that one finishes it runs another with command line argument cleanup. Then it releases the virtual image file.

Virtual image files are only available on Windows; requesting a virtual image file on a non-Windows lisp results in a warning and an ordinary file-based dumplisp.

Once a virtual image file has been released, the file part of the dumplisp that references it is no longer a valid image file for starting lisps. An attempt to use it will result in an error.

Calling excl:release-dumplisp-virtual-image a second time on the same otherwise legitimate argument is legal and does nothing.

Calling excl:release-dumplisp-virtual-image with an argument that was not previously returned by dumplisp is an error that may or may not be recognized and signaled.

Copyright (c) 1998-2019, Franz Inc. Oakland, CA., USA. All rights reserved.
This page was not revised from the 8.2 page.
Created 2012.5.30.