|
Allegro CL version 11.0 | ||||||||
This is a list of frequently asked questions on the use of Allegro Common Lisp. Each question applies to all currently supported Allegro Common Lisp versions unless otherwise noted with version specific information. Likewise with Architecture specific information.
Please read this document before sending mail reporting problems to support@franz.com. Periodically, also, some items from this list will be incorporated into our documentation (and dropped from here). Other items stay in the FAQ more or less permanently.
The current version of Allegro CL is shown on this page. This FAQ applies to Allegro CL 11.0 and later. The FAQ for earlier versions is https://github.com/franzinc/cl-faq/ but most answers here apply to earlier versions.
What should be included in a bug report is described in the section Reporting Bugs in introduction.html.
In short, you should include a transcript that demonstrates the failure and error/result seen. Transcripts can be made in a number of ways, such as from an emacs buffer. A logging facility in Allegro CL is also available. Use the function dribble-bug.
To start a session transcript evaluate (dribble-bug "dribble.txt"). This will open the file dribble.txt and write a header containing information about the current lisp environment, which is of help to us when debugging problems. Upon return from this function call, your lisp session is now being logged.
Once finished, evaluate (dribble-bug) (no arguments) to close the dribble bug stream.
The header information is generated by a call to the function print-system-state.
We prefer not to receive screen shots unless there is no other way to transmit info.
If you are using dribble-bug to record output note that dribble-bug should only miss output produced when using the foreign-function interface or running a shell function with run-shell-command and friends, or (because it is produced by C code) information printed by the garbage collector during a garbage collection. Some messages from foreign code may be written directly to Unix stdout or stderr, bypassing Lisp entirely. To circumvent this problem, you can either use an Emacs buffer or the Unix utility "script" to save the entire transaction record.
There is not. Please send mail to support@franz.com for technical inquiries or info@franz.com for all other questions or comments.
The current version is at https://franz.com/support/documentation/current/doc/. It is updated from time to time with corrections or new information. Documentation for earlier versions is at https://franz.com/support/documentation/<version number>/doc/, so for example the version 10.1 documentation is at https://franz.com/support/documentation/10.1/doc/.
Our policy of support of Allegro CL on specific operating system versions is as follows:
For each released version of Allegro CL, we compose a list of current operating system levels on which we have tested and on which we know our product will work. This list is also located in the section Installation sizes and supported Operating System versions in installation.html.
Because Allegro CL runs on so many different operating systems, and because some of these operating systems have almost infinite combinatorial possibilities, including patches and/or user-modification, we do not attempt to certify Allegro CL on specific operating system versions. Instead, we rely on the tendency of the operating system vendors to maintain upward compatibility, and we usually build on the lowest operating system version for which we have listed support. As long as the operating system is truly upward compatible, Allegro CL should work on the newer versions, and our advice is to try it out. But we also recommend strongly that you run such tests on a machine that is not part of your production process, in case things don't go well.
If the operating system you are interested in running Allegro CL on is the same lineage and the underlying architecture is the same as one which we have already listed as supported, the best question to ask us is
We can answer this question "yes" or "no", which will tell you as the customer whether we have any negative experiences with the operating system version. (The information here may also provide the answer.) If the operating system is very new, the answer to this question might more likely be "no", because the assumption is that the new version is compatible, unless and until it proves otherwise. Other questions that you might ask, if you know that your operating system is very new, are
The fundamental issue with Allegro CL working on a new operating system versions is whether the operating system has any fundamental issues that break Allegro CL. Typically, when the operating system is a natural progression from one which we currently support, we will take specific steps to resolve incompatibilities. Incompatibilities most often occur when an operating system vendor changes the signal-handling interface, usually to be more Posix compliant. The change can break Allegro CL. This has happened in the last several years to Linux, LinuxPPC, FreeBSD, and macOS, in that time order.
The various processes for installing patches are described at http://franz.com/support/patches/
Yes. In fact, even if you are using sys:update-allegro or the IDE's Check for New Patches Dialog to download patches (both of which find all now patches), we strongly recommend viewing this file periodically so that you are aware of what each patch being downloaded is affecting.
There are LOG files for each supported version of Allegro CL. The LOG file for the current (i.e. latest) version is always at
LOG files can also be accessed with the version number, so for 11.0 (the current version) use the link above or
For 10.1
and so on.
We also provide RSS Feeds to Patch releases for current Allegro CL Versions, and both technical and general announcements. You can check the patch LOG page above periodically or subscribe to one of the RSS feeds to be informed automatically.
We maintain a publically accessible FTP site from which users can download all available patches. Please visit: ftp://ftp.franz.com/pub/patches/current/. From the parent directory, one can also access patches for past versions of Allegro CL.
This directory contains a README file with instructions for manually downloading patches toward the bottom of the file.
As patches become available, the files in this directory will be updated, so it is a good idea to be aware of when patches are released. The FAQ item here provides information on how you might do so.
For a general explanation on how to find and load the patches for your version of Allegro CL see http://franz.com/support/patches/.
Updating patches is a two-step process: first patches are downloaded, perhaps with the function sys:update-allegro or with another method as described in the Patches section; then all running Lisps are exited and the update.exe program (in the Allegro CL installation directory and also invoked by one of the Allegro CL menu items) is run. If you are having trouble running update.exe, it may be because you must run it as Administrator. To do so, right click on the update.exe menu item in the Allegro CL menu group and choose Run as Administrator.
See this for current information.
Yes. Much earlier versions of Allegro CL did require internet access for installation but current versions do not.
No. If you downloaded the software from our web site, we believe your antivirus software is giving you what is called a false positive. OfficeScan from Trend Micro, and other Trend Micro antivirus products have been known at various times to give false positives on Allegro products (not just for our Express Edition). Each time this happens we contact them and they fix the problem in an update of their software.
You can use it until the next release of the Express Edition has been out for some time. We usually give Express users 6 months or more to migrate to the new version. Note that sometimes the Express license expires before a new Allegro CL Version is released. In this case, an updated license or a new version of Express (with the same Allegro CL version number) is provided.
Yes, we install a license file along with all the other files for the release. This built-in license file will typically last 2 years. Should a new version of the Express not come out before this expiration, we will post a new license file here. Usually, however, we will either come out with a new major version or an updated release with a new license file.
This can happen when the date on your computer is incorrectly set to a date in the future later than the license expiration date. Allegro CL will not start because it thinks the license has expired. You can check the date on your computer by looking at the Date/Time entry on the Control Panel. Please consult your Windows documentation for information on resetting the date on your computer.
Windows: paste this into the Debug window to build one of mlisp.exe or alisp.exe:
;; mlisp:
(progn
(build-lisp-image "sys:mlisp.dxl" :case-mode :case-sensitive-lower
:include-ide nil :restart-app-function nil
:restart-init-function nil)
(when (probe-file "sys:mlisp.exe") (delete-file "sys:mlisp.exe"))
(sys:copy-file "sys:allegro-express.exe" "sys:mlisp.exe"))
;; alisp:
(progn
(build-lisp-image "sys:alisp.dxl" :case-mode :case-insensitive-upper
:include-ide nil :restart-app-function nil
:restart-init-function nil)
(when (probe-file "sys:alisp.exe") (delete-file "sys:alisp.exe"))
(sys:copy-file "sys:allegro-express.exe" "sys:alisp.exe"))
;; allegro:
(progn
(build-lisp-image "sys:allegro.dxl" :case-mode :case-sensitive-lower)
(when (probe-file "sys:allegro.exe") (delete-file "sys:allegro.exe"))
(sys:copy-file "sys:allegro-express.exe" "sys:allegro.exe"))
Evaluating any of the above forms does not add a menu item to the Start Menu. To run the resulting image, you will need to run the executable created in each form.
UNIX: evaluate the following form to build mlisp image from a running alisp:
(progn
(build-lisp-image "sys:mlisp.dxl" :case-mode :case-sensitive-lower
:include-ide nil :restart-app-function nil)
(when (probe-file "sys:mlisp") (delete-file "sys:mlisp"))
(sys:copy-file "sys:alisp" "sys:mlisp"))
You should have received an email containing a URL from which you can download your license file. (Click here if you haven't received that email or if you have mislaid it. Click here if you cannot access the URL.)
The email we sent you in response to your download order contains a customized URL to your license file. Click on that URL or copy it into a browser to retrieve your license file.
Save the text to a file called devel.lic. (the name is "devel" and the file type or extension is "lic".) On Linux, this is usually not a problem, but because Windows usually hides file extensions some users have problems getting Windows to save the file correctly. When you view this in the Windows Explorer, its file type must show as lic, not txt or text or anything else. Therefore, you must be careful to save this file so that its type is lic and its contents are plain text. Each version of Windows has a slightly different method of identifying and listing file types. Please refer to the "Windows Users" section to identify how to do this on your individual machine. In some instances, Windows will automatically save the file as an HTML file (devel.lic.htm), adding an HTML header and footer. These headers will prevent the license file from working.
If you used the "lost license" URL to obtain your license file, be sure that you save only the license file portion of the email into devel.lic. If you save the entire email message, MIME headers and all, it will prevent the license file from working.
Put the devel.lic file into the Allegro directory.
NOTE: Express licenses are part of the download so this item does not apply to Express.
We can send the license to you by email. Go to https://franz.com/lfs/lostlicense and on that page, you are asked for the email address you entered when you requested the license; entering the email address and clicking the Submit button will cause your license file(s) to be emailed to you. Please wait after clicking Submit for the message saying the request was successful. If no license associated with the email address is in the database, the message `No licenses found' is displayed. If that happens, send a message to support@franz.com. Be sure to tell us the email address you used in the body of the message.)
Typically, this problem occurs because your system is behind a firewall or you must use a proxy server. Our license file server rarely experiences any significant downtime.
You should make sure that you should not be using a proxy server to access sites outside of your network and that your browser is configured appropriately. If you do not know if you should be using a proxy server, or don't know what one is, you should check with your Network Administrator for assistance. If your system does not require a proxy server, then perhaps you are behind a firewall that does not give access to certain ports.
If the proxy server is not the problem, please email Franz Inc. at support@franz.co stating that you are unable to access the URL for retrieving your license (include the URL), and we will have the license(s) sent to the email under which you have registered.
Yes. Questions or problems should be sent tosupport@franz.com. Please be sure to include the following information:
CG/JS is a version of Common Graphics and the Integrated Development Environment which works entirely in a browser. No special windowing system (like OpenGL) is required. See https://franz.com/ftp/pri/acl/cgjs/ for more information and instructions on downloading the necessary files (no additional license is required). That link also points to documentation.
No. While the SSL module fasl file was included in the Express, it was not intended that the SSL interface be available in the Express Edition. If you have the need for an SSL interface, we gladly urge you to contact us at info@franz.com for an evaluation of our Professional or Enterprise Editions.
Section 19.4.4 of the ANSI spec says:
After the components supplied explicitly by host, device, directory, name, type, and version are filled in, the merging rules used by merge-pathnames are used to fill in any unsupplied components from the defaults supplied by defaults.
unsupplied is the crucial word here. By specifying a :directory argument you have supplied the directory component, and the directory component of the :defaults argument is not used. Even specifying :directory nil explicitly supplies a directory component of nil, and this will be treated differently from unsupplied.
Here is a transcript showing a stack overflow. Note that the array has one million (10^6) elements.
USER(1): (setq pippo (make-array 1000000 :initial-element 0))
#(0 0 0 0 0 0 0 0 0 0 ...)
USER(2): (sort pippo #'<)
Error: Stack overflow (signal 1000)
[condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]
Restart actions (select using :continue):
0: continue computation
1: Return to Top Level (an "abort" restart)
[1c] USER(3): :pop
=================^^^^
USER(4): (sort pippo #'<)
#(0 0 0 0 0 0 0 0 0 0 ...)
USER(5):
Here the computation continues and Lisp exits with a segmentation violation:
USER(1): (setq pippo (make-array 1000000 :initial-element 0))
#(0 0 0 0 0 0 0 0 0 0 ...)
USER(2): (sort pippo #'<)
Error: Stack overflow (signal 1000)
[condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]
Restart actions (select using :continue):
0: continue computation
1: Return to Top Level (an "abort" restart)
[1c] USER(3): (sort pippo #'<)
Segmentation fault (core dumped)
%
The stack overflow occurs because a large array is being stack-allocated to perform the sort. The size of the array is architecture dependent; Windows platforms only allocate up to 4 Kbyte arrays on the stack, and normally heap allocate any larger arrays needed, while Unix platforms attempt to allocate 4 Mbyte arrays on the stack. On any architecture, the strategy is programmable; as described below.
When the above error occurs, there are several things that can be done.
Instead of popping out of the break loop as in the example above, just continue. The stack overflow automatically reduces the stack cushion (see documentation for sys:stack-cushion and sys:set-stack-cushion), so continuing should allow further execution.
On Unix platforms only, a csh can be run and the limit command used to set the stack limit to something larger than it currently is. We recommend at least 8192 Kbytes (8 megabytes), but if that is not enough, more can be allocated.
Change the sort strategy (documented below). The Allegro CL implementation of the sort function tries to allocate a temporary array on the stack if possible, so that it does not need to do so on the heap. If this strategy is not acceptable or convenient, change the strategy to either allocate from the heap or to use a pre-existing user supplied array.
Just continuing usually works as does, usually, clearing stack with a :reset and retrying. Note, as the second example above shows, trying to redo the sort command in the error prompt (that is, without clearing the error) can result in an abnormal exit from the lisp (Segmentation fault (core dumped) ).
This is an unfortunate hole in our stack-overflow detection strategy; Stack overflow is normally detected for every function call, and enough "slop" is allowed for so that functions that allocate an average amount of stack will not cause a hard stack overflow. But if the function allocates large stack objects (such as large temporary vectors) then the jump in stack usage is too much to detect by either the stack cushion or the hardware overflow detection, and stack-overflow death occurs. We hope to guard against such overflow death in some future version of Allegro CL.
Sort Strategy:
You can tell the system whether to try to stack-allocate things to be sorted. From the documentation in the source code:
;; excl::*simple-vector-sort-strategy*:
;;
;; The sort strategy can be one of three types:
;; :stack - try to allocate stack space for the temp sort; this
;; works easily for 1k elements (4 kbytes), and (on
;; Unix platforms only) for up to 1m elements (4 mbytes)
;; if there is enough stack allocated by the os; more
;; than 1 m elements cause a new svector to be allocated.
;; :alloc - Allocate an svector of size equal to the vector to sort.
;; a new one is allocated each time.
;; <vector> - must be a simple-vector of type t of at least as many
;; elements as are being sorted. During the sort, the global
;; is reset to :alloc so that sort is re-entrant.
(defvar excl::*simple-vector-sort-strategy* :stack)
The stack-cushion (see sys:set-stack-cushion. and sys:stack-cushion) is detected in "symbol trampoline", a short piece of code that is used when one Lisp function calls another. It is meant to flag normal situations where stack is growing too quickly, and to signal a condition before a hard stack-size limit is reached.
There are several possible situations where the stack-overflow is not detected by this mechanism, and careful thought must be given as to how to handle it:
A lisp function may allocate a very large stack size, due to either a large number of variables or due to large stack-allocated arrays or lists. If the amount that the function allocates is larger than the difference between the hard stack limit and the soft stack limit set up by the stack cushion, then there will be no chance for the Lisp to signal the condition before the hard limit is reached. The only way to work around this problem is to be sure that there is sufficient stack-cushion for the worst-case function to allocate its needed stack.
A Lisp function might call itself recursively, which on some architectures generates a fast call to location 0 of the same function. The fast call causes the symbol trampoline to be bypassed, thus causing the stack overflow detection to also be bypassed. The workaround is to declare the function calling itself as notinline within its own body. This will result in slightly slower code generation, but overflows would then be detected. Example:
(defun call-me ( ... )
(declare (notinline call-me))
...
(call-me ...) ... )A non-lisp thread may be called, at which time there is no way to limit the stack on some machines. There is no workaround for this problem, other than to reduce one's dependence on non-lisp code.
A package has an associated hashtable for the names of symbols in the package. When the size of a package is not specified at creation time, a default hashtable is used. Its initial size is small, allowing for 10 entries, and it tends to grow slowly, growing about 20% each time growth is necessary. Those values are reasonable for most uses, but if you know that a package will have many more symbols, particularly if they will be all created at roughly the same time (as when reading a file that interns thousands of symbols), you should specify the :size keyword argument to defpackage appropriately when creating the package. Thus, if you know the package will eventually have about 4000 symbols, define it with a form like this:
(defpackage :foo (:size 4000) (:use :cl :excl))
The function cl:sxhash always returns the same value for structure objects. The reason for this is because it has no extra space to store a unique hash code within it. As a result, using structures as keys is very inefficient. The (unexported) excl::hash-table-stats function demonstrates this when given a hash-table with structs used as keys; the histogram becomes the worst case, because every key wants the same index.
The decision was made to keep the same behavior for structure objects, because the automatic inclusion of a hashing slot in all structure objects would have made all structs an average of one word longer. For small structs this is unacceptable for many of our users.
Instead, a user may define a struct with an extra slot, and the constructor for that struct type could store a unique value into that slot (either a random value or a value gotten by incrementing a counter each time the constructor is run). Also, create a hash generating function which accesses this hash-slot to generate its value. If the structs to be hashed are buried inside a list, then this hash function would need to know how to traverse these keys to obtain a unique value. Finally, then, build your hash-table using the documented :hash-function argument to make-hash-table (still using the equal test argument), to create a hash-table which will be well-distributed.
Alternatively, and if you can guarantee that none of the slots in your structures will be changed after they are used as keys in the hash-table, you can use the equalp test function in your make-hash-table call, rather than equal. If you do, however, make sure that these struct objects don't change, because then they may not be found in the hash-table.
This trace example provides the following code:
(defun fact (n)
(cond ((= n 1) 1)
(t (* n (fact (1- n))))))which works interpreted, but on some architectures produces a truncated result when the function is compiled, e.g.:
cl-user(4): (fact 5)
0[2]: (fact 5)
0[2]: returned 120
120
cl-user(5): Different architectures produce different output, depending on details of their instruction set, number or registers, and so on. 32-bit x86 architectures, for example, tend to produce the full trace output, while 64-bit Lisps on 64-bit x36 architectures tend to produce the above truncated output, bypassing the trace output.
The issue is that a self-call can be made very efficient in architectures that take on the x86-64 style compilation, and as such they bypass the central call site which performs the Lisp function call. This call site is very efficient for calls to other functions, but is normally bypassed for an extra bit of efficiency when calling the same function (since the function register doesn't need to be loaded, because it is already in the right place, and the correct instruction is trivial to locate - it is the pc-relative 0 location in the current code vector). This bypassing of the call site is performed regardless of any tail-call merge settings, but can be thwarted by declaring the function notinline:
(defun fact (n)
(declare (notinline fact))
(cond ((= n 1) 1)
(t (* n (fact (1- n))))))This declaration removes the special handling of the self-call, and thus self-recursive calls are made through the central call site, where the trace mechanism will see the call.
Something like this will not work of some platforms (64-bit Windows, for example):
(build-lisp-image "new.dxl" :lisp-heap-size (* 1024 1024 1025 15))
Instead of allocating a 15 GB heap, it will allocate something significantly smaller. This is caused because the integer resulting from the multiplication (16,121,856,000) requires more than 32-bits to represent and the ascii to integer conversions on some operating systems do not handle more than 32-bits.
Instead, do this:
(build-lisp-image "new.dxl" :lisp-heap-size "15375m")
In large images, this is occasionally a sign that your system has run out of virtual memory. This can occur on platforms that perform lazy swap allocation, which makes possible the overcommitting of virtual memory. Heap relocation or resizing will not help resolve this issue. Instead, you should:
The most common cause of this problem is that you've run out of address space for the lisp heap. The first question to ask yourself, as a developer, is: Do I expect my application to consume as much memory at is it?
If the question is no, you should determine the cause of the unexpected allocations and resultant heap growth. The Space Analyzer can help you do so.
If the answer is yes, then it may be that you need to adjust the locations of the heaps used by Allegro CL. Choosing an adequate location in which to map the Lisp and foreign (Aclmalloc) heaps in a running Lisp image is complex. We refer to these problems collectively as the heap placement problem. While these problems are not in fact new, they are only triggered when the Lisp image is large (typically greater than 500 Mbytes).
Continue reading the questions below for advice on how to proceed.
When Allegro CL starts up, space must be found for the following:
Aclmalloc heap. This is where space allocated by aclmalloc is located, along with the value of certain C variables. Note that Aclmalloc heap is an unfortunate name because it is not directly related to C and is, in fact, directly managed by lisp. The Aclmalloc heap must always be allocated at an address higher than the Lisp heap. This heap is static and will not be relocated if necessary at startup. This is due to a requirement to maintain the accuracy of pointers into the Aclmalloc heap across calls to dumplisp.If you use 32 bit addressing, there are potentially 4 GB of address space. However, most operating systems only allow 31 bit addresses, so only 2 GB are really available. Locations near address 0 (the bottom) are usually reserved by the operating system. Therefore, Allegro CL usually tries to start at some OS-dependent location, typically around 0x20000000, so there is potentially a 1.6 GB block above that. This is plenty for an application that uses less than, say, 0.5 GB, but as the application grows to, say, 1.5 GB, finding enough space becomes problematic. Assuming no intervening shared libraries or other allocated regions in the process address space, you can determine your maximum lisp heap in an image by subtracting the lisp-heap-start (lisp base) from the aclmalloc-heap-start (aclmalloc base).
Lisp is given an idea of how much heap space it will need to operate via the lisp-heap-size argument to build-lisp-image. This value does not describe the extent of the heap in use, rather it indicates a guaranteed size to which the heap should be able to grow. When building large images on 32-bit platforms, process address space is at a premium and will typically be exhausted long before physical RAM or Virtual Memory is consumed. It is important, therefore, to find and claim a suitable block of address space for application needs.
The claiming--via mmap(), specifically--of reserved growth space for the heap has differing behavior under different operating systems. For systems that support the MAP_NORESERVE flag or perform lazy allocation, a lisp-heap-size'd block of address space will be claimed without any resultant mapping to virtual memory (until it is actually used). For other platforms a request for address space will also claim an equivalent amount of virtual memory.
So, what might go wrong when Allegro CL starts up? The following might be problems:
Even though this problem may affect any user of Allegro CL or an Allegro CL application, it is mostly a problem with developers of programs which will be distributed to a user base. Users who have an Allegro CL distribution and use a particular machine can, perhaps with trial and error (and perhaps with assistance from Franz Inc.), figure out how to build an image that will work on that machine. But VAR's, say, who are preparing a distribution requiring large images which will be sent to many customers, each of whom may have a different machine configuration and different programs running, may find it difficult to produce a single image suitable for all potential users on a particular platform.
Programmers can affect heap placements using these arguments to build-lisp-image:
Any application that uses most of the available address space is in danger of running into machine constraints which make the application fail.
On 64-bit machines using 64-bit addresses, you can create images with heap sizes larger than needed for most applications. However, there can be conflicts with other resources at lower addresses so it is often a good idea to put the Lisp head starting point at a high address.
This is a complicated answer. We start with some terminology:
Shared-library: A program unit that can be linked or loaded into a program, which usually has a .dll extension for Windows, a .dylib extension for macOS, and an .so extension for all other UNIX systems. Shared libraries come in three flavors:
The ACL shared library: This shared-library holds the base ACL system, and is sometimes known by the term "acldll". On Windows it is known as aclxxx.dll, and on UNIX it is called libaclxxx.ext where xxx is a version number and .ext is either .dylib or .so.
System libraries: shared-libraries that are pre-linked into either the ACL shared-library or the executable that loads the ACL library, or any shared-library on which a system library is dependent. This is a broad definition of system library, and can include any user-supplied library that has been linked into the executable.
User loaded libraries: These are libraries that are loaded with the Common Lisp LOAD function, but not pre-linked as are system libraries.
Executable: a program that either links or loads and invokes the ACL shared library. We supply several executables in the distribution, one called mlisp.exe on Windows and mlisp on unix. However, any program can be built that does the equivalent (see on Windows, for example, dll.htm). The executable loads the ACL shared library, and invokes it by getting and executing the lisp_init() function. (You can provide your own main() that calls lisp_init(), see main.html).
Heap file: a file with a .dxl extension (also called an image file) that holds the two Lisps heaps. One is the Aclmalloc heap, which holds non-Lisp data including C variable values and data allocated by aclmalloc. The other heap is the Lisp heap, which holds all mutable Lisp data.
Pure Lisp library: A file with a .pll extension, which may be optionally loaded into a Lisp process and which contains read-only Lisp data.
Link: A process of attaching shared-libraries by name to an executable, usually by a linker program or by a compiler that automatically invokes a linker. Shared libraries and their symbols are not usually actually included into a program when linked, but references by name are made and the linked library is required to be available before the executable can start.
Load: A process by which a shared-library is dynamically mapped into the memory space of an already-running executable. The functions to call to load a shared-library are LoadLibrary() on Windows and dlopen() on Unix systems including macOS. The load Common Lisp function of Allegro CL calls one of these functions when it sees that the file being loaded is a shared-library.
Bind: Attaching functions within a shared-library to allow those functions to be called. The ff:def-foreign-call function in Allegro CL creates a binding location that is automatically updated when shared-libraries are loaded that supply the functions. The executable must bind the symbol lisp_init in the ACL shared library in order to call it. Binding is done by getting the address of the function's code, by GetProcAddress() in Windows and dlsym() on Unix systems including macOS.
Invoke: The process of calling a function which has been bound. C invokes a function that has been bound by using a "pointer-to-function" construct. Lisp uses ff:def-foreign-call to bind a C function in such a way that it looks like a Lisp function.
Committed area: An area of memory that is mapped in and which consumes virtual memory, one page of virtual memory for each page of address space. The memory might be physical memory (RAM), or it can be swap memory on disk, or it might even be a specific file, also on disk. The Committed area of the Lisp heap is usually the area bounded by the lisp-heap-start and the "Top" address in the room display.
Reserved area: An address range of memory is reserved when no other program unit has the right to map anything into that range, but the address range is not committed by actual swap space. This is the area requested by the :lisp-heap-size argument to build-lisp-image (see building-images.html). It is carried through all dumplisps. Although the committed area may eventually grown beyond this value, the space for the reserved area is only allocated if there is enough space for it. The reason for having a reserved area, instead of simply allocating memory that the operating system will give, is because as the Lisp heap grows, its addresses must grow monotonically increasing; new spaces must be always at higher addresses.
The Startup Process:
The operating system starts up the executable. Before the executable is able to start running, all system libraries must be loaded into memory and available. Various operating systems do this in various ways, using implementation dependent algorithms to place the libraries into memory. Also, whether all symbols are bound at the time of the library placement or whether the symbols are bound lazily (i.e., when needed only) is OS dependent; some systems may provide both.
The executable begins to run. It may perform any operations it wants to do, including loading shared-libraries.
The ACL shared-library is loaded if necessary and lisp_init is invoked. The operating system ensures that all shared-libraries that were linked into the ACL shared-library are loaded before lisp_init is allowed to start. If the ACL shared-library was linked in, lisp_init can be called immediately. Step 1 above assured that all proper links are done. See main.html for further details.
lisp-init determines a heap (image) file, with extension typically .dxl. The heap file contains state information about the image, including whether a Pure Lisp Library will be used. The .dxl and .pll are loaded into memory in the following manner:
The Aclmalloc heap is mapped in, and C variables are set. If a .pll file is to be used this fact becomes known at this time. The Aclmalloc heap can not be relocated from where it was first built.
The .pll file, if present, is mapped in read-only. If it can't be mapped into the location it had been in a previous incarnation of Lisp, it is moved to another location. (Except when an image is built with build-lisp-image, there is always a previous incarnation of Lisp, perhaps the one that ran when the image was built).
The Lisp heap is mapped in. The best case is if the heap can be placed at the address and size that were specified by the build parameters. If those can't be satisfied, decreasing sizes are tried until either one is successful or the current commit level is passed (at the same address), followed by an operating-system-selected space of the built size, followed by decreasing sizes tried at OS selected locations until one is successful or the current commit level is passed (at which time Lisp startup fails). If the lisp-heap-size happens to be larger than the available swap, then only as much as can be actually allocated is used, as long as it is at least as large as the commit requirement.
Pointers are adjusted if the Lisp heap or the .pll file had to be moved. (The Lisp heap file might have to be moved because the Lisp heap start address is not available, e.g. The .pll file might have to be moved if the address in the previous Lisp invocation is unavailable.) All pointers to the expected locations of the Lisp heap file and the pll file are moved to the new locations. This step is performed all at once, to ensure proper pointer movement.
The Lisp starts. The process by which the Lisp starts is documented by the source file, <allegro directory>/src/aclstart.cl, which is provided in your distribution, and described in What Lisp does when it starts up in startup.html. As one of the items in aclstart.cl, excl::reload-fix-entry-points (this function is not further documented) is called, which ensures that all system and user libraries are loaded if they are not already. This may involve performing a load on any libraries that were not loaded.
Now, there is a potential problem with the last step. If
then it is conceivable that there will be no swap left for the loading of the user libraries.
In this situation, it would be best for the executable to be programmed to pre-load the user librar(y/ies) so that the space is pre-allocated. (You do this by writing a customized main() that loads the needed libraries, see main.htm) But this presents the possibility that the user library might break up the contiguous address space for the Lisp heap, especially on Windows. The problem in intractable in general, but solvable in individual cases.
The ansi standard function room can be used for this. A description of the output generated by this function is described in section 2.4 Getting information on memory management using cl:room of gc.html.
Knowledge of your heap locations is necessary to make use of the below suggestions for finding room for larger heaps or for relocating DLLs so they don't interfere with your heap growth. Likewise, if you are interested in increasing your heap size, you should have an idea of how much space you want based on your application requirements.
There is a C function in the Lisp called memory_status_dump, with this prototype:
void memory_status_dump(char *file);
You can bind a Lisp function to this by evaluating this
(ff:def-foreign-call memory_status_dump () :strings-convert t)
and you can execute it like this:
(memory_status_dump "mem.out")
It will give you a dump of every address in the process, and its status. A string argument will dump the input to the given file. An argument of 0 will send the output to the console on Windows or the tty in which you are running Lisp on UNIX.
For example, on Solaris the output would look something like this:
Mappings: 52, from struct prmap_t in sys/procfs.h:
0x00000000 - 0x0000ffff 65536 free
0x00010000 - 0x00017fff 32768 ---r-x offset = 0
0x00018000 - 0x00025fff 57344 free
0x00026000 - 0x00029fff 16384 ---rwx offset = 24576
0x0002a000 - 0x00033fff 40960 -b-rwx offset = 0
0x00034000 - 0x03ffffff 66895872 free
0x04000000 - 0x04705fff 7364608 ---rwx offset = 24576
0x04706000 - 0x04771fff 442368 ---rwx offset = 0
0x04772000 - 0x04899fff 1212416 ---rwx offset = 7389184
0x0489a000 - 0x04a49fff 1769472 ---rwx offset = 0
0x04a4a000 - 0x53ffffff 1331388416 free
0x54000000 - 0x54005fff 24576 ---rwx offset = 9093120
0x54006000 - 0xfe6fffff 2859442176 free
0xfe700000 - 0xfe7a7fff 688128 ---r-x offset = 0
0xfe7a8000 - 0xfe7b5fff 57344 free
0xfe7b6000 - 0xfe7bffff 40960 ---rwx offset = 679936
0xfe7c0000 - 0xfe7d7fff 98304 ---rwx offset = 0
0xfe7d8000 - 0xfe801fff 172032 free
0xfe802000 - 0xfe803fff 8192 ---rwx offset = 0
0xfe804000 - 0xfe903fff 1048576 free
0xfe904000 - 0xfe905fff 8192 ---rwx offset = 0
0xfe906000 - 0xfea05fff 1048576 free
0xfea06000 - 0xfea07fff 8192 ---rwx offset = 0
0xfea08000 - 0xfeb07fff 1048576 free
0xfeb08000 - 0xfeb09fff 8192 ---rwx offset = 0
0xfeb0a000 - 0xfec09fff 1048576 free
0xfec0a000 - 0xfec0bfff 8192 ---rwx offset = 0
0xfec0c000 - 0xfed0bfff 1048576 free
0xfed0c000 - 0xfed0dfff 8192 ---rwx offset = 0
0xfed0e000 - 0xfee0dfff 1048576 free
0xfee0e000 - 0xfee0ffff 8192 ---rwx offset = 0
0xfee10000 - 0xfef0bfff 1032192 free
0xfef0c000 - 0xfef0dfff 8192 ---rwx offset = 0
0xfef0e000 - 0xfef0ffff 8192 free
0xfef10000 - 0xfef11fff 8192 ---rwx offset = 0
0xfef12000 - 0xff00dfff 1032192 free
0xff00e000 - 0xff00ffff 8192 ---rwx offset = 0
0xff010000 - 0xff0bffff 720896 free
0xff0c0000 - 0xff0ddfff 122880 ---r-x offset = 0
0xff0de000 - 0xff0ebfff 57344 free
0xff0ec000 - 0xff0effff 16384 ---rwx offset = 114688
0xff0f0000 - 0xff0f9fff 40960 ---rwx offset = 0
0xff0fa000 - 0xff0fffff 24576 free
0xff100000 - 0xff1a3fff 671744 ---r-x offset = 0
0xff1a4000 - 0xff1b1fff 57344 free
0xff1b2000 - 0xff1b9fff 32768 ---rwx offset = 663552
0xff1ba000 - 0xff1bbfff 8192 ---rwx offset = 0
0xff1bc000 - 0xff1e3fff 163840 free
0xff1e4000 - 0xff1e5fff 8192 ---rwx offset = 0
0xff1e6000 - 0xff1effff 40960 free
0xff1f0000 - 0xff1f3fff 16384 ---r-x offset = 0
0xff1f4000 - 0xff1fffff 49152 free
0xff200000 - 0xff27ffff 524288 ---r-x offset = 0
0xff280000 - 0xff28dfff 57344 free
0xff28e000 - 0xff297fff 40960 ---rwx offset = 516096
0xff298000 - 0xff29ffff 32768 ---rwx offset = 0
0xff2a0000 - 0xff2affff 65536 free
0xff2b0000 - 0xff2b1fff 8192 ---rwx offset = 0
0xff2b2000 - 0xff2bffff 57344 free
0xff2c0000 - 0xff2c3fff 16384 ---r-x offset = 0
0xff2c4000 - 0xff2d1fff 57344 free
0xff2d2000 - 0xff2d3fff 8192 ---rwx offset = 8192
0xff2d4000 - 0xff2dffff 49152 free
0xff2e0000 - 0xff2e7fff 32768 ---r-x offset = 0
0xff2e8000 - 0xff2f5fff 57344 free
0xff2f6000 - 0xff2f7fff 8192 ---rwx offset = 24576
0xff2f8000 - 0xff2fffff 32768 free
0xff300000 - 0xff307fff 32768 ---r-x offset = 0
0xff308000 - 0xff315fff 57344 free
0xff316000 - 0xff319fff 16384 ---rwx offset = 24576
0xff31a000 - 0xff31ffff 24576 free
0xff320000 - 0xff321fff 8192 --srwx offset = 0
0xff322000 - 0xff32ffff 57344 free
0xff330000 - 0xff345fff 90112 ---r-x offset = 0
0xff346000 - 0xff353fff 57344 free
0xff354000 - 0xff355fff 8192 ---rwx offset = 81920
0xff356000 - 0xff35ffff 40960 free
0xff360000 - 0xff365fff 24576 ---r-x offset = 0
0xff366000 - 0xff373fff 57344 free
0xff374000 - 0xff375fff 8192 ---rwx offset = 16384
0xff376000 - 0xff37ffff 40960 free
0xff380000 - 0xff387fff 32768 ---r-x offset = 0
0xff388000 - 0xff395fff 57344 free
0xff396000 - 0xff397fff 8192 ---rwx offset = 24576
0xff398000 - 0xff39ffff 32768 free
0xff3a0000 - 0xff3a1fff 8192 ---r-x offset = 0
0xff3a2000 - 0xff3affff 57344 free
0xff3b0000 - 0xff3cdfff 122880 ---r-x offset = 0
0xff3ce000 - 0xff3dbfff 57344 free
0xff3dc000 - 0xff3ddfff 8192 ---rwx offset = 114688
0xff3de000 - 0xff3dffff 8192 ---rwx offset = 0
0xff3e0000 - 0xff443fff 409600 free
0xff444000 - 0xff445fff 8192 ---rwx offset = 0
0xff446000 - 0xffbe5fff 7995392 free
0xffbe6000 - 0xffbeffff 40960 s--rwx offset = 4294942720
Note that each section of memory is grouped by how it is currently mapped; the four characters at the end are flags, with a dash representing not available, and one of r, w, x, and c for read, write, execute and copy-on-write (shared until written, when it will get a private copy).
The problem with memory_status_dump is that it doesn't always identify what object is being mapped into any particular address area (such information depends on what's available on a per platform basis). It seems reasonably intuitive what an "object" is by all of the maps on all architectures; each address range constitutes one mapping. However, it is impossible to tell what the objects actually are, and sometimes when two address ranges are exactly adjacent to each other, the higher address range is likely an extension of the lower address range. This is important in deciding what objects to try to identify when deciding how to move things around. For example, if the Aclmalloc heap starts at 0x54000000 and the map has two ranges next to each other, one from 0x54000000 to 0x54007fff and one from 0x54008000 (i.e 0x54007fff + 1) to 0x5401dfff, then it is likely that the Aclmalloc heap was extended once, and thus that if the Aclmalloc heap is moved it will take care of both address ranges.
For x86 windows, we recommend the use of Process Explorer (previously hosted by www.sysinternals.com, but now distributed by Microsoft) and a VMMap tool. These utilities provide a lot of information and have good user interfaces.
The Process Explorer is available for download here. and the VMMap tool here.
Here are some more notes on the Process Explorer: you first download and install this program. Then, start Allegro CL and run Process Explorer. This will bring up a dual-paned window, the top half of which will show a list of running processes. Click on `allegro.exe' (or `mlisp.exe' or `alisp.exe'), and in the lower pane you will be shown what objects have been allocated in the processes address space.
Some Process Explorer configuration tips:
For other platforms, the situation is more complicated, but there are usually operating system tools for interpreting the data.
The preferred Lisp heap starting address is 0x20000000 on x86 Windows (the heap grows to higher memory addresses). If, using the Process Explorer program (discussed above, an x86 Windows application), you find that the Lisp heap cannot grow to the size that you desire because of a DLL that is in the way, you can use the `editbin' program from Microsoft Visual C++ to move the default base address of the DLL. For example, if foo.dll is in a bad location, according to Process Explorer, and you found foo.dll to live in the c:\winnt\system32 directory, then at a DOS prompt type (assuming 0x65000000 is the beginning of a free address range large enough to accomodate the DLL):
cd c:\winnt\system32
editbin foo.dll /rebase:base=0x65000000
and the next time Lisp starts up, Windows will try to locate foo.dll at 0x65000000 instead of the previous inconvenient address that interfered with the Lisp heap. Note that you may need to disable related programs or boot into safe mode in order to perform this operation. It is generally not possible to modify such files when they are active and loaded into a process address space.
Note that the base address for a DLL is only advisory; if the space is already used, Windows will locate it to some other location. By right-clicking on a DLL and selecting "Quick View" (if Quick View support is installed), you can see (among other things) the base address currently assigned to the DLL.
It means that some other program has grabbed part of the address space that Lisp intended to use. This message is much more common on Windows, where various programs can grab predefined address ranges in all running programs. The Iomega Zip/Jaz tools and PGP are two programs known to do this. There are four things that you can do to handle this situation:
The arguments to build-lisp-image that you will need to specify are :lisp-heap-start, :lisp-heap-size, :aclmalloc-heap-size, and :aclmalloc-heap-start.
Both the Lisp and Aclmalloc heap must remain in one individual contiguous piece. (i.e a single contiguous lisp heap, and a separate single, contiguous C heap). You need to find a large enough gap in the address space to cover all growth expectations of your application. Most 'large' apps need lots of room for the lisp heap to grow and very little for the Aclmalloc heap. Your mileage may vary.
On Unix platforms, the OS allocates memory in the sbrk region, which typically resides directly after the mapped location of the lisp executable. If you restrict this region too much, you run the risk of causing your application to fail with an "unable to allocate memory" error that will usually crash the lisp. This malloc region is used by low level OS system calls and typically by any foreign libraries included in your application.
Combined with the knowledge of your current heap locations and sizes, you can use memory-status-dump (see earlier question) to find free regions in the process address space.
The typical way to build new images is via the function build-lisp-image. It accepts the following four keyword arguments that are used to relocate the lisp and Aclmalloc heaps from their default locations and sizes:
The following two keyword arguments let you further shape the lisp heap once it has been allocated.
Once you have chosen values for these parameters (see previous questions in this section for help doing so), simply include them in your call to build-lisp-image. When starting the new image, a call to room should show that the lisp heaps have been relocated.
A common problem with developers of 'large' applications is that default development images such as alisp.dxl and allegro.dxl are built with default heap locations. Further, even if developers customize the heap locations and sizes of these apps, their changes are undone when they update patches and run update.sh or update.exe.
To this end, we have added the following environment variables that build-lisp-image will check for when creating a new image.
Arguments values should use lisp HEX notation (#x).
It is recommended that users not modify their general development environment when making use of these environment variables as that can result in undue confusion and unexpected results. Instead, it is recommended that a new script (or batch) file be created with a name not used by Allegro CL, such as my-update.sh or my-update.bat, which invokes update.sh or update.exe depending on the platform. This script file can then set these environment variables as desired before invoking the Franz provided update script.
The user then need only remember to run their personalized update script and their heap settings should persist even when patches are updated.
No. The Aclmalloc heap in particular is not relocatable, so this approach is not feasible. The Lisp heap is relocatable--and occasionally will shrink if the reserve space is not available--but due to the Aclmalloc heap restriction, the only option would be to lower it in memory. This being only half a possible solution, we have not opened up this limited functionality.
The loop you are running is likely not releasing any bytes of heap for the amount it is allocating. A loop similar to
(compile
(defun gobble ()
(loop for x from 1 to 1000000000000000000000000
with k = 0
do
(incf k)
(when (= k 10000000)
(print ".")
(setq k 0))
collect x)))
(gobble)
is likely generating almost no garbage. Allegro CL's garbage collector makes decisions about how much space to allocate based on the gsgc-parameters that are currently set, and the effect of this loop is to throw the calculations into a non-linear area where the only thing it can do is to cause the heap to grow at a huge rate of increase.
Most real programs never run this way; either they don't cons at all, or at least some of the consing done is ephemeral, even if only a few percent of the total consing. And, of course, any program which continues to cons permanent space should be considered leaky, unless it is explicitly a test to gobble up all of memory, and to see how such gobbling progresses.
To get better gobbling behavior, allocate at least some of the space in such a way that it becomes garbage. This allows the gc to operate in a more natural manner, and will cause a smoother transition toward the inevitable heap-exhaustion:
(compile
(defun gobble2 ()
(loop for x from 1 to 1000000000000000000000000
with k = 0
do
(incf k)
(when (= k 10000000)
(print ".")
(setq k 0))
(when (zerop (mod k 10))
(cons x nil)) ; waste a cons
collect x)))
(gobble2)
Finally: the reason why heap grows so quickly is because newspace must hold all of the objects that have been allocated until they are to be tenured, even though it is known that none of them will die, and so the newspace must continually grow to hold the new allocations as well as the objects waiting to be tenured. An explicit gc with tenuring periodically will serve to flush the newspace out, and will also keep it to a reasonable size, thus making the march toward heap-exhaustion even more smooth:
(compile
(defun gobble3 ()
(loop for x from 1 to 1000000000000000000000000
with k = 0
do
(incf k)
(when (= k 10000000)
(print ".")
(setq k 0)
(gc :tenure)) ; clear newspace
(when (zerop (mod k 10))
(cons x nil)) ; waste a cons
collect x)))
(gobble3)
Leaks (i.e. the non-terminating consing of lisp objects in the heap) are hard to identify, because they are hard to distinguish from cache warm-ups - any data build-up which must occur during the running of the program might appear to be a leak locally, but over time will reach an asymptotic level. A leak will blow past this asymptote, but without knowing what that stable level is, it is hard to know whether the program is leaking or just building up its caches.
A few more terms and some background:
Short-lived data, or ephemeral data: lisp objects that only survive in newspace, whose lives are so short that they are garbage-collected before they can be tenured.
Medium-lived data: lisp objects that survive long enough to become tenured. At some point a medium-lived object's life is ended and a global-gc will collect it.
Long-lived data: lisp-objects whose lifetimes are indefinite. Usually the long-lived object is part of a list or a vector based in a global symbol or some other gc root (any other long-lived object). Its lifetime might be limited, but usually only by some explicit action, like a setq or setf of the variable or slot (or a chain of pointers) to some other value (which then becomes the new long-lived object or chain of objects). Note that except for these end-of-life scenarios, global-gcs don't affect long-lived data.
Closed/open old areas: Old areas are either closed or open. Open old areas function normally, and medium-lived objects can be tenured into them, and a global-gc will clean these open areas out of all of their dead objects. Closed old areas function specially; they are considered full even if they are not. All objects in a closed area are considered to be alive, even if they no longer have any other live objects pointing to them. During a global-gc, no marking or adjustment is done to objects in a closed old area; all objects are assumed to be marked and are stationary. This means that effectively the global-gc doesn't garbage-collect any closed old areas.
The cutoff between short-lived and medium-lived objects can be adjusted by changing the size of the newspace area halves. The larger the newspace, the longer each generation is and the longer time it takes for an object to be tenured. Using resize-areas to adjust the newspace-size can reduce the amount of medium-lived data in oldspace, thus reducing the need for global-gcs.
Q. How do I identify new long-lived objects?
A new technique can be used to create a new open old area set, tenure data while running a test, and then looking at all of the objects in that open old area set.
Here is the technique; explanations will follow:
[1] (setf (sys:gsgc-parameter :open-old-area-fence) 0)
[2] (gc t)
[3] (setf (sys:gsgc-parameter :open-old-area-fence) -1)
[4] (sys:resize-areas :new <newsize> :old <oldsize> |:old-code <old-code-size>|)
[5] (setf (sys:gsgc-parameter :open-old-area-fence) -1)
[6] <exercise>
[7] (print-type-counts :open-old)
[8] (setq x (get-objects <code> :new nil :old :open))
[9] (inspect x)Explanations:
[1] opens all old areas, so that a global-gc [2] will remove all garbage. [3] now sets the open fence in such a way that only the most recent old area is open, and all others are closed. The resize-areas in [4] will force an extra old area to be built if and only if <oldsize> is large enough. Note that for :jitcode and :mprotect lisps :old-code will also have to be specified appropriately. At the same time, whatever <newspace> is chosen will determine how much data will be tenured later. The values of :oldspace and :newspace can be experimented with, but in order for this technique to work as well as possible, the result should be a lisp with two open old areas (one may not be empty, but the newest one should be) and a newspace which will keep as much medium-lived data as long as possible, so that it becomes short-lived and only long-lived data becomes tenured.
The open fence setting in [5] now ensures that there is only one open old area and that that area is completely empty. A (room) will verify this, if desired.
[6] represents the exercising of the application in a way that demonstrates the leaking. This can go on as short or as long as desired.
At this point, or at any point during [6], (room) can be used to show what has now been tenured into the open old area. It may be that newer old areas are created in the process; they will automatically remain open. Only a gross size is shown for these open old areas; when it is desired to see what is in the open old space, move to [7] which will tell you what kinds of objects are there. Select a code (in the left-hand column of the print-type-counts output) and use that as the code input for get-objects. This invocation of get-objects will only grab objects from the open old area(s). The result is a healwalker results vector, which can be inspected at [9]. This vector starts with a count N in the first slot, and then all of the found objects in the next N slots.
These objects can now be traced back using either get-references or the new gcpath facility. Hopefully a reason for the object's liveness can be explained, and if the reason is bogus, a bug can be fixed so the object is no longer alive.
Allegro CL does not support 64-bit integers in 32-bit lisps, since we try to be compatible on all architectures on which we run, and since there are so many different extensions to C to allow for 64-bit integers, which are of course longer than a "long" type.
You can work around this by defining a struct containing two 32-bit values
(def-foreign-type unsignedlonglong
(:struct (high :unsigned-int) (low :unsigned-int)))
(defun unsignedlonglong-value (x)
(+ (ash (fslot-value-typed 'unsignedlonglong nil x 'high) 32)
(fslot-value-typed 'unsignedlonglong nil x 'low)))
When you define your foreign function, have it return the struct, rather than the integer, and have the lisp reconstruct the number using the function above. You can then have the lisp reconstruct the number using the function above.
For passing 64-bit values, you can write a routine to pack 64-bit values into the struct. The struct returned should be passed to the foreign function call.
(defun value-unsignedlonglong (x)
(let ((inst (ff:allocate-fobject 'unsignedlonglong)))
(setf (ff:fslot-value-typed 'unsignedlonglong nil inst 'high)
(ash x -32)
(ff:fslot-value-typed 'unsignedlonglong nil inst 'low)
(logand x #xffffffff))
inst))
The above code could be further generalized to allow for multiple allocation types. Users are cautioned to be careful not to introduce memory leaks into their application.
Downloading all the latest Allegro CL patches will download, if necessary, the latest version of AllegroCache. Evaluating (require :acache) will signal an error but also print information on the available versions of AllegroCache along with instruction on how to load them. See also the AllegroCache page and the AllegroCache Download Page.
Very little (click to see the documentation).
If your code works fine on 32-bit versions of Allegro CL, then it is most likely because of the difference in size between :int and :long.
The :returning keyword argument to def-foreign-call is a key piece of the solution, where the return value has been declared as an :int rather than a :long or an :unsigned-long. Passing an :int into a foreign call is also dangerous, where the expected types might mismatch.
The real problem is that C defines the default declaration to be int (i.e. if a variable or function return is not declared, it is assumed to be int). But in a 64-bit application, the equivalent integer (an integer whose size matches that of a pointer) is long not an int, and so there is a tendency to underdeclare C functions.
And because our def-foreign-call interface tries to match the interface for C, the same issue occurs - but because in our def-foreign-call, where there is no way to check against the C code like a C compiler can (and can thus issue warnings about possible data loss when assigning a pointer to an int variable) the defaults become a disadvantage.
Aside from the items documented in the fork(2) Man page, there are a few other issues that Allegro CL application writers must be aware of when using excl.osi:fork.
All Lisp processes survive in the child process.
The first pitfall is that in a typical C application, only the thread calling fork() will survive in the child. In a non-os-thread lisp, however, lisp processes are not tied to os level threads. They will continue to exist, and possibly share resources, in the child process. Further, in future, when os-threaded lisps become available, while the underlying threads will not exist in the child, the process objects in lisp will, creating an invalid state where lisp continues to believe these threads are valid. We will update this faq with more information on how to resolve this issue when *nix os-thread lisps become available.
Inadvertent shared resources between parent and child. Sockets and Streams.
The second pitfall is that the child process inherits all file descriptors from the parent. In lisp, it is not common to deal with fd's directly, but rather the socket and/or file stream objects which overlay these fds. Any attempts to read or write on sockets or streams, that are not properly synchronized between parent and child, may result in race conditions that cause a forked application to unexpectedly hang, or crash.
Further, there are many add-on modules that start lisp processes that make use of streams, further obfuscating the existence of potentially shared resources. An example is the acldns module, which communicates with local nameservers in order to provide hostname lookup capabilities.
Recommendations
There are two resources which might be helpful to Python users:
As an example, see the below output
CL-USER(12): (time (fib 30))
; cpu time (non-gc) 36,784 msec user, 140 msec system
; cpu time (gc) 3,994 msec user, 10 msec system
; cpu time (total) 40,778 msec user, 150 msec system
; real time 51,274 msec
; space allocation:
; 60,582,248 cons cells, -934,589,280 other bytes, 0 static bytes
1346269
CL-USER(13):
The problem is with the overflow of machine integers. On 32-bit platforms, the allocator, which resides in low level C code, keeps track of counts using 32-bit variables. As these allocations occur over time, these variables can in fact overflow resulting in the negative output seen above. It would be a simple matter to adjust such values in the case where an overflow occurs only once (the typical case), by adding #x100000000 (4 GB) to the negative value. But because we cannot determine how many overflows actually occured, we choose not to adjust in all cases where negative space is reported as that would give wholly false (but apparently valid) results in some cases. We recommend trying the adjustment (adding #x100000000) and seeing if the result looks reasonable.
Symmetric multiprocessing is available as part of Allegro Common Lisp 11.0 on these platforms. (SMP may not be available on some platforms in earlier releases.)
For more information see our SMP documentation.
We do not recommend that because the security model on some versions of Windows requires having Administrator priviledges to modify files in subdirectories of Program Files. Files are modified when patches are installed but also many users use subdirectories of the Allegro CL directories for their own source files, and writing or modifying them will thus also require Administrator priviledges. The default Allegro CL installation procedure installs in the C: drive directly. That is what we recommend.
Current versions of Microsoft operating systems come with a security feature enabled called Data Execution Prevention. While it helps protect against a common class of malicious attacks, it also can prevent our lisp from running! To correct this, firat verify that DEP is enabled and is what is causing the problem.
There are a couple of options here.
If DEP is only enabled for essential Windows programs, then DEP is not causing the problem. Please see the FAQ entry on How to report Bugs and submit a report to support@franz.com. If DEP is enabled for all programs and there are no lisp exceptions, it is likely the cause.
Choosing the first option is the simplest, since Allegro CL provides a number of different executables from which Lisp can be used. Also, the update process creates a temporary executable called update.exe when updating images to contain new patches. If you decide to add DEP exceptions for Allegro CL, you will need to create a temporary file named update.exe in the Allegro directory and add it using the DEP dialog.
dir with run-shell-command?On Windows, excl:run-shell-command executes programs but does not invoke shell commands. The function is therefore misnamed for Windows. It is called run-shell-command to provide cross-platform compatibility between Windows and Unix and back compatibility for ACL4.3.x users. It does behave differently on Windows and Unix. The difference is related to differences between UNIX and Windows. See the documentation page for excl:run-shell-command for details on this point.
The default stack reserve size for the lisp process is 16MB.
The stack-size process attribute was supposed to control the stack allocation for the process. What the Windows documentation did not make clear, unfortunately, was that the running process can control the amount of swap space initially committed to the stack, but cannot select the amount of address space reserved for the stack.
The address space to be reserved for each thread's stack is a property of the executable image being used. The property is specified at link time, but it can be altered without relinking, through the EDITBIN program that is part of the VC+ development package.
To make each thread allocate a 4Meg stack, the command would be (for the executable allergo.exe, repeat the command for other executables):
editbin /stack:4000000 allegro.exe
Unfortunately, MS Windows offers no way to have different reserve sizes for different threads in the same process, so all the threads have to be big enough to support the deepest computation on any thread.
The Windows registry associates file types with programs, so that double clicking on a file of a particular type initiates a program in a clearly defined way. (Thus, typically double clicking on a file with type txt typically initiates the Notepad utility in Windows, opening the file clicked on.) As defined by the installation procedure, lpr files (the files defining projects) are associated with allegro.exe/allegro.dxl, the Modern version of Allegro CL with the IDE. Double-clicking on an lpr file thus starts the Modern image with the IDE, and opens the project associated with the lpr file.
If you want the ANSI Allegro CL with the IDE (allegro-ansi.exe/allegro-ansi.dxl) to start instead, you must modify the registry.
The simplest way to do this is with a Windows tool. While Windows menu choices move about, try opening the View menu on the Windows Explorer and click on the Options... item or try the Tools menu and the Folder Options choice. This will display an Options dialog. Click on the File Types tab, and find Lisp project file in the displayed list of types. Select it and click on the Edit... button. Select open in the Actions pane and click on the Edit... button. Modify the text in the Application used to perform action field, changing allegro.exe to allegro-ansi.exe and allegro.dxl to allegro-ansi.dxl. Click on the OK buttons on the open dialogs to complete the change. On Windows 2000, open the Tools menu and click on the Folder Options item, and choose the LPR extension, click on the Advanced button, click on the open action, click on the Edit... button, find the associated command line, and modify it.
When working with Windows offline files, the following situation has been observed. As a result, we recommend that in general, one not develop using offline files.
A file, foo.cl, is compiled.
The compiler signals an error that goes to a break loop.
Go to an Emacs buffer where the file foo.cl is showing.
Make some correction to the file and save it.
:continue from the break.
The compiler claims that there is no in-package.
The file foo.cl on disk appears to be all hex zero bytes.
If the file foo.cl is still in the Emacs buffer, it can be saved again to recover the content on disk; but if the Emacs buffer was killed, the content of the file is lost.
There seems to be no way for ACL to detect this situation and thus avoid it or warn the user about the possible danger. The only solution is to use caution when working with offline files.
We have found that using an alternate tool such as WinSCP can be one way to avoid this problem while retaining the ability to synchronize remote files conveniently.
Common Graphics does not handle the righthand alt key on the Windows platform because that overrides its functionality on some keyboards (especially European keyboards) for entering characters that don't have dedicated keys. You can revert to the old functionality by setting the CG configuration option reserve-righthand-alt-key to nil, as in:
(setf (cg:reserve-righthand-alt-key (cg:configuration cg:*system*)) nil)
That option doesn't appear on the IDE's Options dialog, but if you modify it once in the IDE then your options file will remember the value for subsequent IDE sessions. In a standalone CG application you would need to set the value in application code.
There are various errors which can occur:
cannot restore segment prot after reloc: Permission deniedThis problem occurs when SELinux (Security-Enhanced Linux) is enabled. This is a security extension originally developed by the NSA that comes enabled by default in some newer linux distributions. One of the security features changes the way shared libraries are loaded by default, and this is preventing Allegro CL from properly starting.
The best solution is to run the following command to enable Allegro CL to load its shared library:
# chcon -t textrel_shlib_t /usr/local/acl11_express/libacli817t.so
You need to substitute /usr/local/acl11_express with the actual path to your Allegro CL installation directory. Also, libacli817t.so is the name of the Allegro CL Express Edition shared library, and for other versions of Allegro CL the name might be different. Run the above command on any file named .so in the Allegro CL directory.
Another solution is to disable SELinux entirely. This is done by either of these methods:
Allegro CL runs on a large number of kernels, for which we do not provide a comprehensive listing. See our Platform Listing for current information about kernel or glibc restrictions.
No entries.
If a function accepts both optional and keyword arguments, and you need to pass any keyword arguments, then you must first pass all optional arguments before any keyword arguments. Because the value of an optional argument could be a keyword symbol, it would otherwise be ambiguous whether a keyword that's passed just after all required arguments is an optional or keyword argument. If you don't want to specify particular values for the optional arguments, then you generally should pass the default values for them, which typically are nil but which may be any value -- and that requires learning the default values of the optional arguments even though you don't care about them.
For these reasons, it's considered poor style to specify both optional and keyword parameters when defining a function, and converting any optionals to keywords in that case is preferred (especially when there are more than a few, or there is a likelihood of adding additional parameters in the future). Nevertheless, the Common Lisp spec does include a few functions such as read-from-string that have both optionals and keywords.
I.e. Shouldn't these return nil?
CL-USER(66): (read-from-string "(something" NIL)
Error: eof encountered on stream #<EXCL:STRING-INPUT-SIMPLE-STREAM
"(something" pos 10 @ #x100eafeceb2> starting at character position
83894.
[condition type: END-OF-FILE]
Restart actions (select using :continue):
0: Return to Top Level (an "abort" restart).
1: Abort entirely from this (lisp) process.
[1] CL-USER(67):
CL-USER(68): (with-input-from-string (s "(huh") (read s NIL))
Error: eof encountered on stream #<EXCL:STRING-INPUT-SIMPLE-STREAM
"(huh" pos 4 @ #x100eb0c7912> starting at character position 83894.
[condition type: END-OF-FILE]
Restart actions (select using :continue):
0: Return to Top Level (an "abort" restart).
1: Abort entirely from this (lisp) process.
[1] CL-USER(69): It seems counterintuitive, but the answer is NO.
First, all of these operations are read operations - in the case of with-input-from-string you are calling read directly, and in the read-from-string case, the description in 23.2.6 in the first paragraph:
Parses the printed representation of an object from the subsequence of string bounded by start and end, as if read had been called on an input stream containing those same characters.
So we are dealing with read here, or equivalent, in all cases.
Now, in a tradition carried over all the way from CLtL and CLtL2, and codified in 23.1.3.1, also second paragraph:
Functions such as read that read the representation of an object rather than a single character always signals an error, regardless of eof-error-p, if the file ends in the middle of an object representation. For example, if a file does not contain enough right parentheses to balance the left parentheses in it, read signals an error. If a file ends in a symbol or a number immediately followed by end-of-file, read reads the symbol or number successfully and when called again will act according to eof-error-p. Similarly, the function read-line successfully reads the last line of a file even if that line is terminated by end-of-file rather than the newline character. Ignorable text, such as lines containing only whitespace or comments, are not considered to begin an object; if read begins to read an expression but sees only such ignorable text, it does not consider the file to end in the middle of an object. Thus an eof-error-p argument controls what happens when the file ends between objects.
Finally, a section that is easy to misinterpret when read by itself is the "Exceptional Situations" section of the read-from-string page:
If the end of the supplied substring occurs before an object can be read, an error is signaled if eof-error-p is true. An error is signaled if the end of the substring occurs in the middle of an incomplete object.
This appears to be two sentences that complement each other, but are really two sentences describing two different things, when taking these other paragraphs into consideration.
Apparently, there are two concepts with eof handling: when seen while reading an object and when seen between the reading of objects. Your examples are instances of the first case, and the second case can be demonstrated in a couple of ways:
CL-USER(2): (read-from-string "" nil :my-eof)
:MY-EOF
0
CL-USER(3): (read-from-string "abc" nil :my-eof :start 3)
:MY-EOF
3
CL-USER(4):According to the ANSI spec, a null eof-error-p argument suppresses an end-of-file condition that occurs just before reading a new object, but NOT one that occurs while in the middle of reading an object. Reading from the string "(foo", for example, will still signal an error at end of file, because it is in the middle of reading a list object at the time.
Here's a rule of thumb to follow: If you know that a string is "well formed", such that a sequence of valid objects can be read from it in the usual iterative way with an end of file immediately afterward, then you can rely on a null eof-error-p argument to avoid signaling an end-of-file error. But in the general case where a string has unknown contents, an end-of-file error (or some other read error such as "too many colons") may still be signaled. Generally when reading from unknown contents, it's a good idea to wrap a handler-case form around the read to trap errors.
Copyright (c) 2023, Franz Inc. Lafayette, CA., USA. All rights reserved.
|
Allegro CL version 11.0 | ||||||||