If you load a lot of data into Lisp, you may get an error that looks like this:
Error: An allocation request for XXXX bytes caused a need for YYYY more bytes of heap. The operating system will not make the space available. [condition type: STORAGE-CONDITION]
XXXX and YYYY and numbers, the second usually being in the millions and the first usually smaller, sometimes much smaller. What this message is telling you is that the Lisp heap needs to expand but for some reason, it is unable to do so. The most common reason that the Lisp heap cannot expand is that it has grown to the point where it has run into something else. And the most common thing that it has hit is the C heap used by Lisp. That is the situation we will discuss in this note though we will mention other potential problems.
First, let us quickly deal with some issues:
Evaluating (room)
prints out information about space used
by Lisp. This is from a 64-bit Lisp:
cl-user(2): (room) area area address(bytes) cons other bytes # type 8 bytes each (free:used) (free:used) Top #x1001c06000 New #x100182e000(4030464) 461:32494 711376:2731456 New #x1001456000(4030464) ----- ----- 3 Old #x10013d6000(524288) 0:0 351616:171808 2 Old #x1001396000(262144) 0:0 16:261520 1 Old #x1001316000(524288) 327:3222 0:466080 0*Old #x1000001250(20008368) 0:127257 0:17923152 OTot(Old Areas) 327:130479 351632:18822560 * = closed old area Root pages: 137 Lisp heap: #x1000000000 pos: #x1001c06000 resrve: #x1004000000 C heap: #x2000000000 pos: #x2000039000 resrve: #x20000fa000 cl-user(3):
For our purposes, the two relevant lines are:
Lisp heap: #x1000000000 pos: #x1001c06000 resrve: #x1004000000 C heap: #x2000000000 pos: #x2000039000 resrve: #x20000fa000These lines tell us where the Lisp heap starts (#x1000000000), how much space is used (the pos value, #x1001c06000), how much has been reserved (the resrve value, #x1004000000) and where the C heap starts (#x2000000000). These are all hex values. If you cannot think in hex, Lisp will translate them to decimal:
cl-user(3): #x1000000000 68719476736 ;; the Lisp heap start, 64 Gigabytes cl-user(4): #x1001c06000 68748861440 ;; the current top of the Lisp heap cl-user(5): (- #x1001c06000 #x1000000000) 29384704 ;; the current Lisp heap is using about 28 Mbytes cl-user(6): #x1004000000 68786585600 ;; the top of the reserved area cl-user(7): #x2000000000 137438953472 ;; the C heap start cl-user(8): (- #x2000000000 #x1000000000) 68719476736 ;; the max potential for the Lisp heap (64 Gbytes) cl-user(9):
I started with a 64-bit Lisp to indicate that only very large amounts of data are likely to cause problems in 64-bit Lisps. Here are some 32-bit Lisp numbers (the numbers you see may differ):
Lisp heap: #x10000000 pos: #x11312000 resrve: #x20000000 C heap: #x54000000 pos: #x54022000 resrve: #x540fa000 cl-user(3): #x10000000 268435456 ;; Lisp heap start cl-user(4): #x11312000 288432128 ;; Current Lisp heap top cl-user(5): #x20000000 536870912 ;; top of reserve cl-user(6): #x54000000 1409286144 ;; C heap start cl-user(7): (- #x54000000 #x10000000) 1140850688 ;; Maximum potential Lisp heap size cl-user(8):
The reserve area is usually not important. If the Lisp heap grows bigger than that value, as long as there is nothing in the way (see below), it just keeps growing, until it hits the C heap start. So the important value is the difference between the C heap start and the Lisp heap start. In our cases, the maximum for our 64-bit image is about 64 Gbytes while the maximum for the 32-bit image is 1 Gbyte.
Now the 32-bit image is bound by the effective address size (32 bits, or 4 Gbytes) and OS restrictions (which further limit things to 2 or sometimes 3 Gbytes). Our image allows about 1 Gbyte and that may be increased. The way to increase it is to specify different values for the :lisp-heap-start and :c-heap-start arguments to build-lisp-image. This will create an image that uses these new values and will therefore allow for a Lisp heap as large as the difference between the values. A call looks like:
(build-lisp-image "biggerimage.dxl" :lisp-heap-start "100M" :c-heap-start "1610M")
Note that we specify the values as strings and use M to stand for Megabyte. The value can be an integer or a string and strings can use the K or M abbreviations. We recommend that you use strings with the M abbreviation as that avoids any problem that might occur with integer values overflowing.
If the image is built successfully, we will have increased the maximum Lisp heap size to 1.5 Gbytes.
The (room) output only reports on where Lisp components are mapped in virtual memory. It is possible that some other non-Lisp component (such as a shared library) is mapped in the area between the top of the Lisp heap and the start of the C heap. In that case, the Lisp heap will stop growing when it hits whatever is mapped. (Nothing can be mapped in the reserve space, because the OS has allocated that space to Lisp. But if you get the cannot grow Storage Condition error mentioned at the start of this note when there seems to be sufficient space between the reserve location and the C heap start, then it is likely something is mapped in that unreserved area.)
If you suspect some object is mapped where it interferes with Lisp growing, you have to find out what it is and decide what to do. It is possible sometimes to move objects, and you can also move the Lisp heap start (and also the C heap start) up so they are above the object. But the first step is finding out what is there. That process is beyond the scope of this note, but there is a FAQ entry describing what to do (the process is different on UNIX and Windows -- Windows users scroll down in the entry)
The following links to the documentation and the FAQ deal with the issues discussed in this note:
Copyright © 2023 Franz Inc., All Rights Reserved | Privacy Statement |