Lisp as a Shared Library

Allegro CL supports the use of lisp via a shared library by C/C++ programs in the same manner as they use any other function libraries. This document describes how Allegro Common Lisp developers on UNIX machines can create applications with Lisp as a shared library. See dll.html for information on creating similar files on Windows.

This functionality makes use of posix threads and is supported on all Unix-style Allegro CL platforms.

1.0 Why an updated interface

The previous Lisp as a shared library example was written at a time when good thread library support was not available on many platforms. As a result, the example was available on very few platforms. Further, the example was built in a way that was not compatible with how we build our lisp binaries, link in the allegro shared library, and load in the lisp environment at startup. This often led to problems in customer applications that were difficult to debug. Lastly, many changes have gone into our lisp environment since the example was written. A refactoring of the Lisp as a shared library interface allows us to provide a more simplified and concise API.

Many of the steps involved in loading the lisp environment need not be exposed to the user, nor clutter their code. This updated interface hopes to better separate lisp startup code from user application code. A single call is needed to start lisp initialization. This call handles choosing the appropriate allegro shared library, spawning a thread, and returning control to the user once initialization is complete. Various hooks, described below, allow you to perform app specific initialization at various points, as well as synchronize access to the Lisp control thread, from which calls into the lisp environment can be made.

2.0 Lisp as a shared library application files

The following files are needed to build a lisp shared library application.

• examples/unix-shared-library/lnkacl.{so,sl,dylib,a}
• examples/unix-shared-library/lnkacl.fasl
• misc/lisp.h
• misc/shlibs.h
• The Allegro shared library, though this typically will be copied into your application directory automatically when you it is built.

Also in examples/unix-shared-library/ is code for a sample C application that calculates factorials by calling into lisp. An architecture specific makefile is also provided with the recommended build commands for compiling and linking user applications. Typing make in this directory will build the factorial example (ftest), placing it in the subdirectory fact/.

Run fact/ftest to test the demo. You will likely need to set your dynamic library search path environment variable to include this directory. See OS Specific Library Search Path for what environment variable is expected to be set.

3.0 Lisp as a shared library application components

The components of a Lisp as a shared library application are as follows:

Foreign Code

• User application code, including a user-defined main() routine.
• lisp_ready_hook(). This function must be defined by the user with this name. It is the last routine called in the lisp thread during initialization of the lisp environment. This function hands off control of the lisp thread to the user-app and should be used to synchronize access to the lisp environment.
• The lnkacl shared library (liblnkacl). This contains the exported C API of the lisp as a shared library interface and must be linked in with your application.

Lisp code

• User application code.
• lnkacl.fasl. This file should be included when building your application image file. When loaded, this file sets *restart-init-function* in order to properly set up the lisp side of the interface. Your application should instead use the *restart-app-function* for initialization and application startup code.

4.0 What happens at Lisp as a shared library initialization

The basic flow of control of a Lisp as a shared library application appears as follows:

MAIN THREAD

- user defined C main()
  - calls initialize_lisp(...)
    - spawns a new thread to load and initialize lisp
    - (wait for lisp_init_complete() signal)     
  - ...continue app execution... (see FOREIGN THREAD below)       

LISP THREAD

- lisp initialization begins, normal lisp startup sequence begins.
  - *restart-init-function* is called 
    (user:initialize-lisp, from lnkacl.fasl)

    - internal set up (such as define callback routine 
      lisp_lookup_address() )
    - call *restart-app-function*, a user defined function 
      to perform necessary lisp setup for user application.
    - call lisp_ready_hook()
      - signal init complete by calling lisp_init_complete()
      - user-defined code to handle requests to be handled by lisp
      - perform any application setup as necessary.
      - (wait for work)
      - call lisp routine. lisp_lookup_address() takes a string 
        as argument and returns a function pointer to the 
        associated lisp callback function.
      - pass return value(s) back to calling thread.
      - resume (wait for work) or return, which will terminate 
        the lisp thread.
   - exit_routine(int value)
     cleanup as necessary for shutting down lisp environment.

FOREIGN THREAD (MAIN or other)

    - contact lisp thread to request work be performed.
    - wait for (user-defined) synchronization event signifying
      completion of lisp work.
    - use value
    ... continue app execution ...

5.0 A note about using Foreign Functions on non-os-thread platforms

On non-os-thread platforms, only a single thread (the thread spawned by initialize_lisp()) can call into the lisp environment. As a result, multi-threaded applications must synchronize with this thread in order to dispatch work performed by lisp. This is done either from within lisp in the application's *restart-init-function* or via foreign code in the lisp_ready_hook()).

See Foreign functions and multiprocessing in foreign-functions.html.

Synchronization with the lisp thread is left to the application developer. A factorial example is provided which uses one possible method.

6.0 C API (routines and data structures)

The following functions must be defined (except where labeled optional).

int initialize_lisp(char **envp, char *image_arg, int libtype, 
                    int suppress_io,
                    void (*exit_routine) (int),
                    struct shlib_library_item **shlib_items)

This existing function must be called to set up the lisp environment. The arguments are:

• envp: by default, use the environment argument passed into main().
• image_arg: the name of the lisp image that you want loaded.
• libtype: specifies the Allegro CL shared library that will be loaded with this application. Accepted values are 0, for the non-international Allegro CL library, 1 for the International Allegro CL library, and 2 for the Allegro CL Free Express Edition shared library.
• suppress_io: suppress lisp output to stdout or allow it. Express users must pass an argument of 0 or the unix-shared-library interface will not function.
• exit_routine: a function pointer to a function you want called when the lisp thread exits. A NULL value means the default exit routine is called that invokes pthread_exit().
• shlib_items: this is an array of structs indicating the shared libraries you linked into your application at build time. This information is passed into lisp at startup so that exported symbols in these libraries will be accessible via the foreign functions interface. Typically, the variable linked_shared_libraries, defined in misc/shlibs.h, should be modified accordingly, #include'd, and passed in as this argument. struct shlib_library_item is defined in misc/lisp.h, and should also be #include'd by your application.

The Allegro shared library, and the image_arg if specified with no path component, are searched for at startup using the OS specific library search path environment variable. See OS Specific Library Search Path to find out what the search path is for your target platform.

This function returns 0 on success, or a negative value when an error occurs.

void *lisp_lookup_address(char *func_name)

This existing function is a simplified routine for finding entry points (function pointers) into lisp. This is provided as an alternative to the index based lookup API provided by register-foreign-callable and lisp_call_address(). As of Allegro CL 8.1, the address returned by register-foreign-callable is allocated in Allegro's C heap, and is static even across dumplisps. As a result, it is possible to map directly from function name to function pointer without the index redirection.

The arguments are:

• func_name: a null-terminated string naming a function in the lisp environment.

This function returns a function pointer to the indicated function or 0 if not found.

int lisp_init_complete()

This existing function is called from the callback routine lisp_ready_hook(). This signals initialize_lisp() that the lisp environment has been initialized and control can be returned to it's calling function.

This function returns a value as pthread_cond_signal().

void lisp_ready_hook()

This function must be defined. The user must establish a C function that will be called into by the lisp thread, once initialization is complete. The body of this function must call lisp_init_complete(). The rest of the code in this routine should contain whatever application specific synchronization is required to communicate with the lisp thread.

The function has no return value. When this function returns, the *restart-init-function* in the lisp environment calls (exit) (see exit). This will then cause your exit routine, if specified, or the default exit routine, to be invoked.

Optional

exit_routine(int exit_value)

A default exit function is defined, but users may define their own to perform whatever cleanup of the lisp environment is desired. The exit-rountine argument to initialize_lisp specifies what exit routine to call. The default exit routine calls pthread_exit().

If you define an exit routine and expect your application to continue execution after the lisp thread is complete, it must call pthread_exit(). If the exit routine returns, exit() will be called, stopping the entire running process.

7.0 Lisp API

The lisp API is almost completely user-defined. After the lisp environment is loaded, the following values should be set in order to set up the lisp as a shared library application environments.

• *restart-init-function*: must be set to user:initialize-lisp. This will occur by default if you load lnkacl.fasl. Most notably, this establishes the callback routine lisp_lookup_address().
• *restart-app-function*: when non-`nil should name a lisp function used to perform any necessary lisp-side initialization of the user application. It is called by user:initialize-lisp prior to it calling lisp_ready_hook().

8.0 Compilation and Delivery

examples/unix-shared-library/makefile contains the recommended architecture specific compile/link commands for building your application. It also contains the recommended build script for generating your lisp application.

In addition, please note the following:

• Your C application must include misc/lisp.h and misc/shlibs.h. The latter should be modified to list all shared libraries linked into your application at build time.
• Your C application must be linked with the lnkacl shared library (liblnkacl.).

When building your lisp application,

• the third argument to generate-application must include lnkacl.fasl, along with all modules required by your application.
• the :application-files argument to generate-application must include the lnkacl library, as well as the file(s) necessary to run your C application.

See generate-application and delivery.html for general help with building lisp deliverables.

9.0 OS-Specific Library Search Path

On most *nix platforms, the dynamic loader library search path can be modified by setting the value of the environment variable LD_LIBRARY_PATH, with the following exceptions.

macOS

Apple uses DYLD_LIBRARY_PATH for finding dependent shared libraries. In some cases, this variable is removed from the environment across calls to fork() and exec() (the system calls used to create a new process). If the first idiom below (for the BASH shell) does not work, then you must use the second:

$ export DYLD_LIBRARY_PATH=...
$ allegro

This is more cumbersome, but it will work when the above does not:

$ env DYLD_LIBRARY_PATH=... allegro

The above instructions apply to allegro as well as mlisp, alisp or any application built with Allegro CL.

Copyright (c) Franz Inc. Lafayette, CA., USA. All rights reserved.