Lisp as a UNIX shared library: revised interface and new documentation

Support/Doc | About | Purchase | Advanced Search

Success
Resources
Learn More About Knowledge Graph Databases

Lisp as a UNIX shared library: revised interface and new documentation

Allegro CL supports the use of Lisp via a shared library by C/C++ programs in the same manner as they use any other function libraries. (There is also support for Lisp as a DLL on Windows, see dll.htm. This note discusses UNIX only.)

The interface and the documentation for Unix Shared Libraries in Allegro CL 8.1 were replaced in an update released in mid February, 2008. The new documentation is unix-shared-library.htm.

In this note, we briefly discuss the changes and the example provided, which creates a C program that calls Lisp to calculate a factorial.

What has changed

The new material, which is available for release 8.1 only, is downloaded when you update Allegro CL with sys:update-allegro. You must have done this if you want to replicate what is done in this essay.

There is actually little new in the Lisp. Instead, changes to UNIX, particularly in the area of thread library support, allow the same method to be used on all UNIX-like platforms and for a simpler interface. New files are provided in the examples/unix-shared-library/ directory.

The following files are needed to build a lisp shared library application. The ones in the examples/unix-shared-library/ directory are new:

examples/unix-shared-library/lnkacl.{so,sl,dylib,a}
examples/unix-shared-library/lnkacl.fasl
misc/lisp.h (unchanged from before the update)
misc/shlibs.h (unchanged from before the update)

Finally, you need the Allegro shared library, though this typically will be copied into your application directory automatically when you it is built.

The (updated) examples/unix-shared-library/ directory contains code for a sample C application that calculates factorials by calling into lisp. An architecture specific makefile is also provided with the recommended build commands for compiling and linking user applications. Typing make in this directory will build the factorial example (ftest), placing it in the subdirectory fact/.

Express Users: Before typing make to build the example, it is first necessary to modify the call to initialize_lisp() in ftest.c, at line 113. The call should appear, instead, as follows:

    if (res = initialize_lisp(envp,
			      (char *)"fact.dxl", /* image name */
			      2,  /* express shared library */
			      0,  /* do _not_ suppress i/o */
			      0,  /* exit routine. 0 for default */
			      linked_shared_libraries)
	!= 0)

Once this change is made, a call to make should run to completion. Due to the extra output, a failed test will result, but one should see by visual inspection that the correct result is returned.

The factorial example

Since Lisp easily handles large valued integer calculations, a Lisp based factorial() function is a good, simple example. To avoid C integer overflow, the answer is placed in a result string, rather than returned as a number. (The factorial of n is the product of the integers from 1 to n, so factorial of 3 is 1 times 2 times 3 = 6.)

The example uses a number of files, all in the examples/unix-shared-library/ subdirectory of the Allegro directory on UNIX machines. (The files are not included in the Windows distribution of Allegro CL.) The files are:

fact.h: this file contains a C #define macro that specifies the Allegro CL shared library. The #define macro must be changed when used with a new Allegro CL release in the future, since that release would include a differently named Allegro CL shared library.
fact.c: this file contains the shared library C wrapper source. There are a number of OS specific preprocessor macros. These are related to architecture differences and OS thread support differences.
fact.cl: this file contains the code that computes the factorial.
ftest.c: this file is an example C program that uses the factorial calculating shared library.

fact.c details

The load_lisp() private function runs in a new thread created by the original main thread. It loads the Lisp image and starts it running. It calls the Allegro CL library function lisp_init(), which is described in main.htm.

Note that the shlibs.h file is found in the Allegro CL distribution misc/ directory. Use the compiler -I option to add that directory to the include file search list when compiling fact.c.

The Allegro CL library function tcm() is called to prevent the Lisp from writing to stdout and stderr during initialization and subsequent processing. During your development phase, you may wish to comment this call out, in order to see format or print debugging output.

The wait_on_semaphore() and release_semaphore() private functions are used to synchronize processing between the Lisp thread and the original main thread. This example uses pipes as semaphores.

The initialize_factorial() public function creates the Lisp thread. Its argument is a pointer to the main program environment that is available as the third main() argument. Besides initializing the synchronization semaphores, it uses the Allegro CL library function find_file_using_pathstring() to identify the directory containing the fact.dxl Lisp image. It's a good design decision to use the same environmental variable that the OS uses to find shared libraries for the third argument. For example, on Solaris, use LD_LIBRARY_PATH. The wait_on_semaphore() function is called to wait for the Lisp thread to respond that all initialization activities are completed.

The release_end_semaphore() function is provided for the Lisp image to call when initialization steps are complete, releasing the semaphore initialize_factorial() is waiting on, allowing it to return to its caller.

The set_factorial_callback() function is provided for the Lisp image to call during initialization steps to set the callback address for the Lisp 'factorial callback.

The factorial() public function wraps the C/Lisp operations that calculate the factorial result. It sets some global variables that the Lisp thread will access, releases the semaphore that the wait_for_work() function is waiting on in the Lisp thread, and then waits on another semaphore for the Lisp thread to respond that the factorial calculation is complete. It then returns, using a global value set by the Lisp thread.

The wait_for_work() function is called by the Lisp image after initialization steps are complete and it has released the semaphore used to synchronize the initialization phase. It loops forever, responding when the original main thread releases a semaphore that indicates that factorial() has been called. It invokes the factorial callback and then releases a semaphore that tells the main thread that Lisp processing is complete. It then goes back to waiting for more work.

The copy_factorial_result() function is called by Lisp during factorial processing to pass the result string into C data.

The terminate_factorial() public function wakes up wait_for_work() in such a way that it breaks out of its endless loop, causing the Lisp thread to unwind. On Linux, where separate threads have separate process id's, it is imperative that the Lisp thread unwind; otherwise a hang will occur.

fact.cl details

The factorial function calculates the factorial result.

The wrapper C shared library is loaded and foreign function definitions are made as necessary.

The 'factorial-callback callback calls 'factorial, converts the result to a string, and passes the result string to C data using the copy_factorial_result() function.

The 'initialize-factorial function passes the callback address back to C, releases the initialization semaphore, and goes into a "wait for work" mode, all using C functions defined in fact.c. When the wait_for_work() call returns, indicating that terminate_factorial() has been called, 'exit is called, to unwind the Lisp thread.

The *restart-init-function* variable is set to 'initialize-factorial, so that the initialization steps commence when the Lisp image is started.

ftest.c details

The ftest.c file is fairly straightforward. It calls the C wrapper shared library public functions initialize_factorial() and factorial(). On the way out, terminate_factorial() is called. Looking at this code, there is no way to tell that the factorial shared library is Lisp based.

The example demonstrated

We start in a shell in the Allegro CL directory. We change into the examples/unix-shared-library/ directory:

% cd examples/unix-shared-library
%

We type make to create the fact/ directory and the ftest executable (we have added some line breaks not in the original transcript):

% make
rm -f build.tmp
rm -fr fact
echo '(compile-file "fact")' >> build.tmp
echo '(generate-application "fact" "fact/" 
  (list :list2 :foreign :defftype "lnkacl.fasl" "fact.fasl") 
  :application-type :exe :application-files (list "ftest" "liblnkacl.so") 
  :restart-init-function nil :restart-app-function  nil 
  :include-compiler nil)' >> build.tmp
echo '(exit 0)' >> build.tmp
LD_LIBRARY_PATH=.; export LD_LIBRARY_PATH; ../../mlisp -L build.tmp
;;; Compiling file fact.cl
;;; Writing fasl file fact.fasl
;;; Fasl write complete
; Autoloading for generate-application:
;   Fast loading /net/gemini/home/dm/acl81/acl81/code/genapp.fasl
;     Fast loading from bundle code/fileutil.fasl.
;     Fast loading /net/gemini/home/dm/acl81/acl81/code/build.001
;;; Installing build patch, version 1.
Bundle is up to date.
Initial generation spread = 1
Allocated 10485760 bytes for old space
Allocated 5242880 bytes for new space
;;;;;;;;;;;;;;;;;;;;; fasl /net/gemini/home/dm/acl81/acl81/code/hash.002
;;;;;;;;;;;;;;;;;;;;;;;;;;;; fasl /net/gemini/home/dm/acl81/acl81/code/misc.001
; fasl /net/gemini/home/dm/acl81/acl81/code/aclstart.001
;;;;;;;;;;;;;;;;;;;;;;


International Allegro CL [master]
8.1 [Linux (x86)] (Mar 13, 2008 10:56)
Copyright (C) 1985-2007, Franz Inc., Oakland, CA, USA.  All Rights Reserved.

This development copy of Allegro CL is licensed to:
   Franz Inc. Staff


 #|
 (setq excl::*batch-mode* t)
 |#

t 
[many lines deleted for space]
 (exit 0)
 |#
; Exiting
rm -f build.tmp
LD_LIBRARY_PATH=.; export LD_LIBRARY_PATH; if test "`(cd fact; ./ftest 5)`" -eq 120; then \
		echo Test for lnkacl-unix completed successfully.; \
	else \
		echo "lnkacl test failed."; \
		exit 1; \
	fi
Test for lnkacl-unix completed successfully.
%

If the make experienced a problem, the last line would be different.

The dynamic library search path environment variable needs to be modified to include the new fact directory because the program needs to find the executable while running. The name of this environment variable is system-dependent (an appendix to unix-shared-library.htm lists the variable name on all supported systems). On Linux, where we are running the example, the name is LD_LIBRARY_PATH, and the following command updates it (you must use the path appropriate for your system, of course):

% setenv LD_LIBRARY_PATH {$LD_LIBRARY_PATH}:/home/dm/acl81/examples/unix-shared-library/fact

Now we are ready to calculate factorials. We start with an easy one:

% fact/ftest 3
6

But because Lisp can handle integers of any (reasonable) size, quite large factorials can be calculated. Because the value is actually returned as a string, no overflows occur (but we have added linebreaks where necessary to avoid making the browser page too wide):

% fact/ftest 20
2432902008176640000
% fact/ftest 200
788657867364790503552363213932185062295135977687173263
294742533244359449963403342920304284011984623904177212
138919638830257642790242637105061926624952829931113462
857270763317237396988943922445621451664240254033291864
131227428294853277524242407573903240321257405579568660
226031904170324062351700858796178922222789623703897374
720000000000000000000000000000000000000000000000000
%

Files in the fact/ directory

In the fact/ directory after the application is built, we have on Linux the following files:

fact: the Lisp executable, a copy of mlisp in the Allegro directory.
fact.dxl: the Lisp image file, created by the make.
fact.lic: the license file, created by the make.
ftest: the program (C) executable, created by the make.
libacli817.so: the Lisp shared library file (copied from the Allegro directory).
liblnkacl.so: an additional shared library file needed for Lisp as a shared library, copied from the examples/unix-shared-library/ directory.

The filenames or extensions may be different on other platforms but the basic set of files will be the same.