Allegro CL Socket Library

This document contains the following sections:

1.0 Introduction and background

Sockets are a mechanism for interprocess communication designed at U.C. Berkeley for use in their version of Unix. Sockets have been added to many other versions of Unix and there is an implementation of sockets for Windows called Winsock. This document describes the Allegro interface to sockets. This interface works on Unix and on Windows.

Symbols naming objects in the socket utility are in the acl-socket package. It has the nickname socket.

The socket module is not included in all versions of Allegro CL. If it is present, it is (by default) included in a development image (one built with the include-devel-env argument to build-lisp-image specified true). To load the socket module if it is not present in an image, evaluate

(require :sock)

Note that runtime images cannot include the development environment (so include-devel-env must be specified nil when a runtime image is being built). If the socket module is needed, it must be loaded when the image is built. See runtime.htm, building-images.htm and delivery.htm for more information.

2.0 Support for Internet Protocol version 6 (IPv6)

Allegro CL supports Internet Protocol version 6 sockets (IPv6 sockets). As part of this support, several new functions have been added and several functions have been modified. The new functions are ipv6, get-ip-interfaces, ipaddrp, ipaddr-equalp, ipv6-address-p, and (added in a June, s006 update) dotted-address-p. The modified functions are dotted-to-ipaddr, dns-query, lookup-hostname, make-socket, and send-to. There is also a new variable *ipv6*.

The feature :ipv6 is added to the *features* list to indicate IPv6 support.

IPv6 support may be unusable on MacOS X 64-bit

Because of apparent bugs in Mac OS X 64-bit, certain IPv6 functionality may be restricted or unusable. In particular:

• The get-ip-interfaces function will signal an error.
• Name-based scope ids will not work.

3.0 IP addresses in Allegro CL

Throughout the socket documentation, we make use of the term IP address. But what exactly is an IP address? Unless further clarified in the context in which it is used, an IP address is either an unsigned 32-bit integer or an ipv6 address structure.

The function socket:ipaddrp returns true when passed an IP adress. That function can be used to identify an object as an IP address. (Unsigned 32 bit integers obviously have other uses that representing IP addresses. The function simply determines whether the type and form of its argument is suitable as an IP address.)

4.0 Characteristics

There are three independent characteristics of sockets:

Note on bivalent format:

Starting in release 5.0.1, the bivalent format is accepted for stream sockets. Bivalent means that the stream will accept text and binary stream functions. That is, you can write-byte or write-char, read-byte or read-char. A bivalent stream is useful in the http protocol (used between web browsers and web servers) since in that protocol the header data is sent in text format and the body can be in binary data (image files, for example).

Internally a bivalent socket stream is configured like a binary socket stream with 8 bit bytes. Character position is not maintained.

Bivalent socket streams have very efficient read-sequence and write-sequence implementations (as long as the sequence is either a vector of element-type character, (unsigned-byte 8) or (signed-byte 8)).

Bivalent socket streams also support the chunking protocol found in http/1.1. This protocol allows the sender to signal end of file without closing down the stream.

5.0 Stream Sockets

5.1 Connections

Stream sockets have a fourth characteristic called connect, with a value :active or :passive. In order to use stream sockets you have to set up a link between two of them. That link is called a connection. You set up a connection in this way:

1. Machine A: create a passive socket at port port-b:

   (setq s-a (make-socket :connect :passive :local-port port-b))
   

2. Machine B: create an active socket telling it to connect to Machine A, port port-b:

   (setq s-b (make-socket :remote-host "machine-a" 
                          :remote-port port-b))
   

3. Machine A: wait for a connect request from anyone and when it occurs return a stream for I/O:

   (setq str-a (accept-connection s-a))
   

4. When the accept-connection returns, machine A can use stream str-a to send messages to machine B and machine B can use stream s-b to send messages to machine A.

Note that steps 2 and 3 can occur in either order.

Note the asymmetry: a passive socket is not a Lisp stream (you can't do read and write to it). An active socket is a Lisp stream.

When accept-connection is called on a passive socket, it does not return until a connection is made to the passive socket. The value accept-connection returns is a stream.

As long as the passive socket is not closed, new connections can still be made to the port of that socket.

An active socket can be used for only one connection. Once that connection has been made, the socket should be closed and a new active socket created.

5.2 Host Naming

Host naming conventions: this package supports three conventions for naming a host:

6.0 Variables

The variables defined by the interface are:

*socket-version*

Please provide the value of this variable when asking for technical support with sockets as it tells us whether you have the latest version.

*print-hostname-in-stream*

This variable controls whether the socket printing code converts the ip address of a socket into a hostname. This is usually what you want, however this can be a slow process (taking up to a minute to accomplish). The default value for this variable is t. See the full description for a discussion of the causes of the possible slowdown when the value is t.

*ipv6*

Specifies the default value of the ipv6 keyword argument to lookup-hostname and make-socket.

7.0 Functions

The first table shows general functions defined by the interface and the second shows accessors.

Socket Accessors

These functions retrieve slot values from socket instances. The values of these slots are set when the socket is created.

8.0 Errors

When errors are raised by the socket interface, Lisp conditions are signaled. This section describes those conditions.

A condition is a CLOS class and thus fits into the hierarchy of CLOS classes. The condition socket-error is a subclass of the condition stream-error.

socket-error is the superclass for all socket related errors. See More on cl:stream-error in errors.htm.

socket-error denotes operating system detected socket errors. It has the following slots:

Handling socket error is difficult because the error returned in exceptional situations can depend on the operating system and the address of the other side of the connection. For example, attempting to make a connection to a machine that is down may result in a "Connection Timed Out" or a "Host Unreachable" error, or maybe something else on certain systems.

The error codes assigned to socket errors vary from operating system to operating system. We translate a large set of the common error codes from a machine dependent number to a symbol which we call the identifier to make it easier for you to write portable code. Condition handling code should check the identifier field (using stream-error-identifier) If the identifier value is :unknown then this is not a common socket error and the operating system dependent code value of the condition must be used.

Possible identifier values and their meanings:

9.0 Sockets and SMP

An SMP (symmetric multiprocessing) Lisp runs one of more OS threads which can use more than one processor (see multiprocessing.htm). When a socket is opened in one SMP process, do not close it from another process.

If you have a process which is reading from a socket and that socket fails so the reading process hangs, you can wake the process up using a process interrupt. So assuming hung-process is the reading process and hung-socket is the socket, the following form executed in another process breaks the read (with an error) and closes the socket:

   (mp:process-interrupt hung-process 'close hung-socket)

You can also wrap the read in a (catch 'dead-partner ...) and the from another process do:

   (mp:process-run-function hung-process
                            #'(lambda ()
                                 (throw 'dead-partner t)))

10.0 Examples

USER(1): (let ((s (make-socket :remote-host "vapor" :remote-port "chargen")))
           (dotimes (i 5) (print (read-char s))) (close s))
#\space
#\!
#\"
#\#
#\$

USER(1): (print (read (accept-connection
                       (make-socket :connect :passive :local-port 9933))))
.. this hangs ...

USER(1): (let ((s (make-socket :remote-host "vapor" :remote-port 9933)))
    (format s "Secret-message~%") (close s)) 

Secret-message
Secret-message
USER(2): 

user(2): (setq s (make-socket :type :datagram :local-port 9999))
#<text datagram socket waiting for connection at */9999 @ #x20664e82>
user(3):  

user(10): (setq x (make-socket :type :datagram))
#<text datagram socket waiting for connection at */45602 @ #x20717fb2>
user(11): (send-to x "foo-the-bar" 11 :remote-host "vapor" :remote-port 9999)
11
user(12): 

user(3): (receive-from s 100 :extract t)

"foo-the-bar"

11 ;; length of result

3229900653 ;; frisky's IP address

45602 ;; the port number chosen for the socket by frisky

user(4): 

11.0 Secure Socket Layer (SSL)

Allegro CL supports Secure Socket layers as described in this section. See also aserve/aserve.html, which describes Webserver support in Allegro CL. Using any https feature in aserve triggers loading of the :ssl module (i.e. evaluates (require :ssl)).

Allegro CL loads OpenSSL libraries dynamically when the :ssl module is loaded. Also loaded is the appropriate one of the Allegro CL-specific shared libraries aclssl, aclissl, aclssl11, and aclissl11. See *aclssl-name* for descriptions of these shared library files.

The system must be told either the version of SSL to be loaded (the system knows whether it uses 8-bit or 16-bit characters) or exactly which of the libraries should be loaded. How this information is determined is described in the next section. If the incorrect Allegro CL library is loaded (say aclissl rather than aclissl11 when SSL 1.1 is loaded) there will not be an immediate error but difficult to debug errors will likely occur later.

Newly installed versions of OpenSSL libraries will be used when installed on your computer without (usually) requiring an Allegro CL update. It is however the user's responsibility to ensure that OpenSSL libraries are available, properly updated, and findable.

To support OpenSSL 1.1, there is a new patch (released in August, 2019) that handles that version as well as continuing to handle SSL 1.0 (in both cases, all minor SSL releases -- the third number in release X.Y.Z -- are also handled.) This documentation has been updated to reflect behavior assuming the new patch has been installed. See sys:update-allegro for information on getting patches.

Users must themselves download and install OpenSSL libraries for their computers, and make sure these are updated when necessary. Further, users must ensure that Allegro CL knows where to find the OpenSSL libraries. On some platforms, there are standard locations where Allegro CL will look. On all platforms, environment variables or Lisp variables can be used to specify the library location.

Here is how to set things up for using OpenSSL:

• If your machine automatically installs OpenSSL libraries, it is likely everything will work without further action. Try (require :ssl) in Lisp. If it works, you are done. If things load without error, you can confirm which SSL version is loaded by calling openssl-version.
• If your machine does not automatically install and update OpenSSL libraries, you must get them and update them yourself. See Installing OpenSSL libraries in installation.htm for information on installing OpenSSL libraries. If you install them in the standard location on your platform, nothing further should be required. Again try (require :ssl) and call openssl-version. If the library cannot be found, you must set the appropriate environment variable (always prior to starting Allegro CL).
• If you want to use a version of the OpenSSL libraries not in the standard location and perhaps not with their standard names, that is supported. Simply set the appropriate environment variable and/or Lisp variable appropriately. See the next section and also *ssl-library-names*.

In all cases, if (require :ssl) loads the :ssl module and the aclssl/aclissl library and loads the OpenSSL library without error, you are ready to use SSL.

The function openssl-version returns information about the version of the OpenSSL library which is loaded and the version used to build the aclssl/acl1ssl/aclssl11/aclissl11 library.

Most Allegro CL platforms support SSL. All that do have :ssl-support included on the *features* list.

The SSL functionality is in the ssl module. To ensure it is loaded, evaluate (require :ssl). Calling either of the two SSL functions, make-ssl-client-stream and make-ssl-server-stream, automatically loads that module.

If you are including the SSL facility in an application intended for delivery, be sure to include the :ssl module by adding the keyword :ssl to the list which is the value of the input-files argument to generate-application. If the copy-shared-libraries argument to that function is false, then the aclssl/aclissl shared library must be copied explicitly from the Allegro CL installation directory to the generated application directory. When the generated application is run, it must be possible to locate the the OpenSSL shared libraries in the same way that is described above.

11.1 The algorithm for loading the OpenSSL library

In this section, we give the exact steps on the various platforms for finding and loading the OpenSSL library. Usually user want the latest version available on their machines (so 1.1 if it is available, otherwise 1.0) but there may be cases where 1.0 is wanted even though 1.1 is also installed.

To repeat what we said in the previous section: try evaluating (require :ssl). If it works and openssl-version indicates the desired version of OpenSSL has been loaded, you are done.

If (require :ssl) signals an error or results in the wrong version of OpenSSL being loaded, then you must take further action.

Allegro CL must know the following things in order to load the :ssl module successfully:

• The names of the two OpenSSL library files. These are usually libcrypto and libssl on UNIX machines, libeay32 and ssleay32 on Windows, with some extension but users or the system may have given the files other names. You can tell the system what these names are by setting the environment variable ACL_SSL_LIBRARY_NAMES or by setting the value of *ssl-library-names* to something other than nil (the variable value overrides the environment value). If the variable is nil and the environment variable is unset, default names are used based on the OpenSSL version being loaded. See below for how the variable or environment variable should be set. If you set either the environment variable or the Lisp variable, you must also specify the OpenSSL version, either by setting *aclssl-version* or by setting the ACL_OPENSSL_VERSION as described the the third bullet.
• The location of those OpenSSL libraries. They are looked for using the paths listed in LD_LIBRARY_PATH (on Solaris and UNIX machines), DYLD_LIBRARY_PATH (on Macs), and PATH on Windows machines.
• The version of OpenSSL to load (1.0 or 1.1). If the library names are specified as described in the first bullet above (that is, either the environment variable ACL_SSL_LIBRARY_NAMES has a value or the Lisp variable *ssl-library-names* is given a non-nil value, you must also specify the OpenSSL version. You do this by setting the Lisp variable *aclssl-version* to (1 0) or (1 1) as the OpenSSL version is 1.0 or 1.1, or setting the environment variable ACL_OPENSSL_VERSION to "10" (meaning OpenSSL 1.0) or "11" (meaning OpenSSL 1.1). If the Lisp variable is non-nil it overrides the environment variable. If ACL_SSL_LIBRARY_NAMES does not have a value and *ssl-library-names* is nil, then the system checks the environment variable ACL_SSL_LIBRARY_NAMES and the Lisp variable *aclssl-version* to determine the version (suitable values are given above). If both are set (and *aclssl-version* is non-nil), the environment variable is ignored. If the environment variable is not set and *aclssl-version* is nil, then the shell command openssl version is run (the PATH environment variable must contain the correct directory to allow that command to run).
• The name of the Allegro CL OpenSSL shared library file, a linkage to the OpenSSL libraries on your system. The standard names are aclssl, aclissl, aclssl11, and aclissl11. The 'acl' (no 'i') files are for 8-bit Lisps. The 'acli' files are for 16-bit Lisps. The files ending in '11' are for OpenSSL 1.1. The others are for OpenSSL 1.0. The value of the variable *aclssl-name* is the library name. If *ssl-library-names* is set to a non-nil value, the value of *aclssl-name* must also be set to the correct standard name. If *ssl-library-names* is nil but the environment variable ACL_SSL_LIBRARY_NAMES has a value, the appropriate one of the associated environment variables ACL_ACLSSL_NAME (for 8-bit Lisps) and ACL_ACLISSL_NAME (for 16-bit Lisps) must be set to the correct standard Allegro CL OpenSSL library name. That value will be used to set *aclssl-name*.

With those four pieces of information, Allegro CL can load the :ssl module and associated library files. Standard defaults are used and usually if (require :ssl) succeeds, all the needed files are successfully found and loaded. If (require :ssl) fails or loads a version of OpenSSL that is not the desired version, then corrected information must be provided so the right files and libraries can be found. We describe how to do that in the remainder of this section.

As described, you can set these variables:

• *ssl-library-names*: if set (and non-nil), all other variables are ignored except *aclssl-name*, which should be non-nil. (The variable if nil will get its value from one of the environment variables ACL_ACLSSL_NAME or ACL_ACLISSL_NAME if the appropriate one, the first for 8-bit character Lisps, the second for 16-bit, is set.)
• ACL_SSL_LIBRARY_NAMES: if this environment variable is set and *ssl-library-names* is nil, all other variables are ignored except *aclssl-name*, which should be non-nil. (The variable if nil will get its value from one of the environment variables ACL_ACLSSL_NAME or ACL_ACLISSL_NAME if the appropriate one, the first for 8-bit character Lisps, the second for 16-bit, is set.)
• *aclssl-version*: if set and non-nil, the value must be either (1 0) for OpenSSL 1.0 or (1 1) for OpenSSL 1.1. Then *aclssl-name* is set to the appropriate standard Allegro CL library name (if already set, that value is ignored) and *ssl-library-names* is set to built-in values.
• None of the variable above are given values: the shell command openssl version is run and the system proceeds as if *aclssl-version* were set to the value returned.

Finding OpenSSL library on Windows:

The procedure above is used. Note only that the value of *ssl-library-names*, if set, should be a list of two items, and the value of ACL_SSL_LIBRARY_NAMES, if set, should be a string concatenating two filenames (with extensions) separated by a single space.

Finding OpenSSL library on UNIX and the Mac:

The procedure above is used. Note only that the value of *ssl-library-names*, if set, should be a two-item list. The items can be a string naming a file (with extensions) or a list of strings naming files (with extensions). ACL_SSL_LIBRARY_NAMES, if set, a string concatenating two filenames (with extensions) separated by a single space.

11.2 SSL History

In 1994 Netscape Corporation designed the Secure Socket Layer (SSL) protocol to provide a means of safely and securely transporting private data (such as credit card numbers) between a Web Browser and a Web Server. Rather than tie SSL to the http protocol, Netscape wrote it as a protocol for making any TCP/IP connection secure.

At the end of 1994 version 2 of SSL was introduced and this was the first version shipped with a commercial web browser (Netscape Navigator (r)). In 1995 version 3 of SSL was introduced. At that point an international standards organization (IETF) took over work on SSL and introduced Transport Layer Security (or TLS) protocol (which is based on SSL but has a different handshake protocol). The IETF introduced TLS version 1.0 in 1999.

Allegro CL, starting in release 6.0, provides an interface that supports SSL version 2, SSL version 3 and TLS version 1 (subversions 1.2 and 1.3). When we use the name SSL, we mean SSL or TLS.

11.3 Secure connections

A secure TCP connection exists between two processes when both agree on the following:

• A cryptographic algorithm (called a cipher) to convert data (plaintext) to encrypted data (ciphertext) and back to plaintext.
• A secret key to use in the cryptographic algorithm. A secret key is essential since the cryptographic algorithms themselves are well known. There is a distinct key for each direction of data transmission (client to server and server to client).
• An authentication method used in each packet to ensure that the contents of the packets have not been altered in transit in some way.

These three items are determined via negotiation when the connection is made and the first data is to be sent.

11.4 Client/Server

In an SSL connection, one side is the client and the other side is the server. In the http environment, the web browser is the client and the web server is the server.

When a secure connection is started, the client starts the negotation by telling the server all the possible ways that it can communicate securely. The server then chooses one of the possible ways and informs the client.

Then the server sends its certificate and possibly other certificates if they are needed to prove that its certificate can be trusted. The important item in the certificate is the public key for the server. The client will use this public key to encrypt a random value which will be used by both the client and server to create the keys needed for the cipher chosen for data transmission.

In theory a certificate isn't necessary if both the client and server side support a key exchange algorithm that can generate a public key on the fly. The SSL libraries we use do not have this capability, thus you must always supply a server certificate.

Once both sides know the keys the other side will use to transmit, the secure data transmission can occur.

11.5 Authentication

The SSL protocol also permits each side of the connection to declare who they are. This is done by the exchange of certificates. The server must send a certificate describing itself to the client. The server can request that the client send a certificate to the server (although in the use of SSL on the web this is never done).

11.6 Certificates

A certificate is a digital document that stores information about an entity in such a way that it can be verified to be true. The primary use of certificates is to store the public key that can be used to send encrypted messages to the entity.

In the SSL protocol certificates have two uses:

1. Encrpytion - by providing a public key they enable encrypted messages to be sent.
2. Authentication - the certificate proves that the entity on the other end of the socket is who it claims to be.

Strictly speaking a certificate isn't required for SSL communication if both sides support a certain key exchange protocol. The OpenSSL libraries we use do not support this protocol thus whenever you create a server SSL stream you must supply a certificate (if you don't have your own we supply one in <Allegro directory>/examples/ssl/server.pem that you can use).

While certificates support authentication, the SSL protocol doesn't require that you take advantage of this facility.

A certificate contains the following:

1. A Subject Identifier: a set of fields describing where the subject is geographically and its role within an organization.
2. A Subject Public Key: the key that can be used to encrypt messages that only the Subject can decrypt since only the Subject has the associated private key.
3. A Valid Time Interval: the interval of time during which this certificate is valid.
4. An Issuer Identifier: just like the Subject Identifier but describing the entity that certifies that the Subject is who it says it is and that the public key is the correct one for the subject.
5. An Issuer signature: a value which can be used by anyone to verify that the Issuer and only the Issuer signed this document testifying to it being correct.
6. Various other fields: like serial numbers, version numbers, and other minor things.

A certificate is a combination of text and binary data and in order to make it easy to transport certificates they are usually encoded in a form called PEM which turns them into a sequence of printable characters.

When a web browser connects to a site via SSL (which is caused by the use of the 'https:' at the beginning of the url), it checks three things about the certificate:

1. Does it know the Issuer and did the Issuer sign the certificate? A web browser knows about a set of Issuers (called Certificate Authorities) when it's installed on the machine (the Issuer certificates are part of the files that make up the web browser).
2. Is the certificate valid right now or has it expired?
3. Is the certificate for the machine we've contacted? If the url was https://www.foo.com/whatever then the certificate must be for www.foo.com. The convention used is to store the name of the server machine in the CommonName slot of the Subject Identifier field of the certificate.

If all three tests pass then the web browser silently accepts the certificate and does a secure web page access. If any of the tests fail then the web browser notifies the user and waits for a response. Each browser displays the failure differently. For example, the Microsoft Internet Explorer (r) shows which of the three tests passed and which failed while the Netscape Navigator (r) just says that it received an invalid certificate. In both cases the person using the web browser is given the option of continuing with the web access. Transmission will still be secure if it is elected to continue. The only issue in doubt is the authenticity of the web server.

11.7 CRL support

The SSL implementation includes certificate revocation list (CRL) support. CRL checking is controlled by the crl-check and crl-file keyword arguments to make-ssl-client-stream and make-ssl-server-stream.

If you enable CRL checking, you must supply a proper PEM-encoded CRL, even if it contains zero revocations. If you do not supply a CRL, peer verification will never succeed.

11.8 TLS 1.3 support and ciphersuites

The TLS 1.3 protocols use ciphersuites. The currently defined (when this document was created) ciphersuite tokens are:

• TLS_AES_256_GCM_SHA384
• TLS_CHACHA20_POLY1305_SHA256
• TLS_AES_128_GCM_SHA256
• TLS_AES_128_CCM_SHA256
• TLS_AES_128_CCM_8_SHA256

Any subset of these may be specified to the :cypersuites keyword argument to make-ssl-client-stream, make-ssl-server-stream, make-ssl-server-context, and make-ssl-client-context.

The argument may be unspecified or given the value nil, in which case the built-in OpenSSL defaults will be used. Otherwise the value should be a string containing the desired tokens delimited by colons, like "TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:TLS_CHACHA20_POLY1305_SHA256".

The :ciphers keyword argument to those functions still exists and specifies the ciphers to be used with TLS1.2 and lower protocols.

For more information on TLS 1.3, see https://datatracker.ietf.org/doc/html/rfc8446 and https://wiki.openssl.org/index.php/TLS1.3#Ciphersuites.

11.9 The Allegro CL SSL API

The following operators, variable and class comprise the SSL API. make-ssl-client-stream and make-ssl-server-stream create the streams that are used for communication. All symbols naming these objects are in the acl-socket (nicknamed socket) package.

• get-ssl-peer-certificate (generic function)
• get-ssl-verify-result (generic function)
• make-ssl-server-context (function)
• make-ssl-client-context (function)
• make-ssl-client-stream (function)
• make-ssl-server-stream (function)
• ssl-do-handshake (generic function)
• x509-certificate (class)
• x509-certificate-issuer (generic function)
• x509-certificate-not-after (generic function)
• x509-certificate-not-before (generic function)
• x509-certificate-serial-number (generic function)
• x509-certificate-issuer (generic function)
• x509-certificate-subject (generic function)
• x509-certificate-subject-alt-name (generic function)
• x509-certificate-version (generic function)

The following function, variable, and classes (naming conditions) are related to loading and using OpenSSL libraries. The conditions are signaled when there are problems loading the OpenSSL library. All these symbols are in the excl package.

• *ssl-library-names*
• openssl-version
• openssl-load-error
• aclssl-load-error

The file <Allegro directory>/examples/ssl/server.pem is a sample certificate and private key file. You can use this file when starting the server side of an SSL connection. The AllegroServe facility uses SSL. It is described in aserve/aserve.html.

Copyright (c) 1998-2022, Franz Inc. Lafayette, CA., USA. All rights reserved.
This page was not revised from the 10.0 page.
Created 2019.8.20.