Improved error reporting. Added (ask-serp) to return Google search results.

Documentation is here.

Patch ssl.015 broke :verify :optional to socket:make-ssl-server-stream. This patch fixes that issue. It also makes SSL-specific options work in OpenSSL 3.0 and later.

Fix off-by-one error in shortq.

Documentation is here.

Work with LLM embeddings. See documentation for details.

Documentation is here.

Fix an unfortunate interation between :output-format and :function in ask-chat. ask-chat may return a list when :output-format is not equal to :text. But when :functions is non-nil, the :output-format must be set to :text.

Documentation is here.

Add function calling API. Documentation is here.

There are two changes in this patch:

• The profiler on 32-bit versions is now more stable.
• Fix a longstanding bug in the ARM64 versions, especially evident on the SMP version, where a register was trashed during an interrupt. The symptoms of the bug were unexplained segmentation violations, garbage collection deaths, and objects of an unknown type, seemingly at random.

Optionally recognize and handle HTTP requests made to an SSL port.

This patch adds two arguments to net.aserve:start:

• http-on-ssl-port - if true then if an HTTP requests comes in to a port configured for SSL, then handle it as an HTTP request, not HTTPS. Without this switch enabled, the SSL layer in aserve will not understand the request and it will fail.
• redirect-http-to-ssl - if true and if http-on-ssl-port is true, then an HTTP request to a port configured for SSL will be redirected to an HTTPS request to the same URL.

Adds excl:with-unreachable-code-allowed macro which suppresses warnings about unreachable code - in 10.1 these warnings occur whenever an otherwise clause is seen in either a cond form or a case form and there are more forms to follow. This warning was always generated, but with this macro wrapped around such forms (or around code that is produced by a macro) the warnings are not generated.

Additionally, there is a new, exported special excl:*unreachable-code-warns* which can be used to turn off all unreachable code warnings.

Recent changes resulted in the following line being generated inappropriately during a resize-areas operation:

Problem in slot sifting: 0

Structure-locking (with-locked-structure, critical-section) could not be used if the bodies protected by the lock might themselves try to do structure-locking. This patch removes that restriction.

This patch removes the new keyword argument to generate-application and build-lisp-image, added in a patch on Thu Jun 16 09:29:30 PDT 2022. The keyword, :set-library-path-to-sys, was ineffective for the purpose it was created. False positives during testing indicated it would work to solve the problem of setting LD_LIBRARY_PATH (UNIX systems other than macOS) or DYLD_LIBRARY_PATH (macOS) so that foreign library dependencies could be found during Lisp or application startup. However, dlopen() requires these variables to be set before the Lisp or application binary is executed, not after. This is a documented security feature.

There are only two viable solutions to this problem:

• Set LD_LIBRARY_PATH, et al in the invocation of ACL or the application built on it, such as LD_LIBRARY_PATH=... ./mlisp.
• Load dependent shared libraries before the library which uses them. For example, if library main.so depends on dep1.so and dep2.so, then loading dep1.so and dep2.so then loading main.so would remove the need for using LD_LIBRARY_PATH, et al.

This patch adds a new keyword argument, :set-library-path-to-sys, to generate-application and build-lisp-image. On non-Windows platforms, when this keyword's value is non-nil, the environment variable DYLD_LIBRARY_PATH (macOS) or LD_LIBRARY_PATH (all others) is set, depending on the platform, to be the path of the location of the application, normally referred to as the 'sys:' logical pathname. This is done before the foreign function interface is initialized, which could be important if shared libraries are part of the application and there are dependencies on those shared libraries during the application's startup.

The default value of this new keyword argument is T for generate-application and NIL for build-lisp-image.

Fixes a bug in sys:record-strings if it is called more than once within a lisp session; the second and subsequent files became corrupted and would not be correctly read in by cvdcvt or cvdcvti (as approporiate). With this patch each file is correctly built.

The new, unexported macro (excl::without-syncsig-trap . body) evaluates the forms of body in an environment that suppresses the hang-on-syncsig effect that might have been requested via an environment variable. This macro should be used under the direction of technical support.

Add support for OpenSSL 3.0.0.

Allow applications to manage TLS 1.3 usage. The new keyword argument :ciphersuites specifies ciphersuites used by TLS1.3 protocols, has been added to make-ssl-context, make ssl-server-stream and make-ssl-client-stream. The value may omitted (nil), or a string of colon-separated tokens. If omitted or nil then built-in OpenSSL defaults will be used. The tokens in the string are used to specify a subset of the allowed TLS1.3 ciphersuites, currently the list is

• TLS_AES_256_GCM_SHA384
• TLS_CHACHA20_POLY1305_SHA256
• TLS_AES_128_GCM_SHA256
• TLS_AES_128_CCM_SHA256
• TLS_AES_128_CCM_8_SHA256

This patch, on all little-endian 64-bit Lisps (everything except SPARC), will separate code-vectors from any other writable objects in the heap (including other code-vectors curently being built). The cost of this patch is some extra space usage to create code-vector-only old areas, and to isolate random code vectors from other objects using unused buffer objects, but the benefit is the nearly complete isolation of excutable pages from writable pages in the hardware, thus relieving cache-consistency issues, and also allowing the 64-bit macOS Lisp to be run on Apple Silicon (M1) hardware with much less overhead. We have noticed, in some cases, this patch results in significantly fewer old areas being created when loading large or many fasls files.

sys:resize-areas now accepts an :old-code keyword option, which tells the resize functionality how much minimum code-vector-only space must be present after the resizing.

cl:room now knows about code-vector-only old spaces. These spaces will be listd as "CVO" (code-vector-only) which can only hold code vectors, as opposed to the traditional "Old" for old areas that hold anything else.

When scanning stack slots conservatively, the garbage collector needs to distinguish between real pointers and values that look like pointers but don't actually point to lisp objects. This determination is complicated by stack-allocated objects,temporary lisp objects allocated in the stack frame of a lisp function. The scavenger used to be very careful, checking each value in a conservative slot against all the active thread stacks. This overhead could become significant when there are many threads, or deep threads (or many deep threads), and is not really necessary. The scan now makes a quick check to see if the address lies in the area reserved for heap-allocated objects. If it does, then it must point to a real lisp object or be considered garbage. If it does not, then it can be treated as harmless. Any stack-allocated object will have its slots scanned as part of the regular scan of the thread that allocated it, not when a reference is seen to it.

In particular, stack-allocated uninitialized vectors (a performance optimization) are handled correctly. The first time an uninitialized vector is scanned, its slots are scanned conservatively. Any slots that have not been stored into, and have recognizeable garbage, get filled with references to nil. Any that have been filled with references to stack-allocated objects will be left alone, since stacks cannot occupy the area reserved for lisp objects. After the first scan, the vector's uninitialized status is removed, and subsequent scans of the object will not need to be conservative.

Also, the maximum number of threads lisp will allow has been increased from 2,000 to 8,000. OS resource limits may reduce the number achievable in practice to something less than this.

Improve storage of large objects. The underlying btree files now store and reclaim space for large objects more efficiently.

Compatibility note:

• This version (4.0.0) can open and use older databases. It will silently convert them to 4.0.0 format. 3.x versions cannot open and use 4.0.0 databases.
• The log file format has not changed. A 4.0.0 database can be reconstructed as a 3.x database by using recover-from-logs in a 3.x AllegroCache.

• When any entries are made to a message queue from a thread other than the receiver thread, the later is often waiting for input on the connection; the output request was often not seen until a timeout. With this patch, queing any output interrupts a waiting sender immediately.
• Add an accessor for the HTTP request that initiated the websocket connection.
• Add a predicate that allows an application to test when a message may be available at the websocket connection.

keyword                        OS constant    NOT SUPPORTED
------------------------------------------------------------
:tcp-keepalive-idle-time       TCP_KEEPIDLE   Solaris, macOS
:tcp-keepalive-probe-interval  TCP_KEEPINTVL  Solaris
:tcp-keepalive-max-probes      TCP_KEEPCNT    Solaris

(let ((ans (socket:dns-response-answer
             (socket:dns-query "srv.record.test.franz.com" :type :srv))))
  (socket:dns-rr-answer (car ans)))
=> (33 44 5555 "srvtest-target.franz.com")