Foreign Types

1.0 Foreign types introduction

The purpose of the foreign types facility is to permit the creation, reading and modification of objects that are described in non-lisp terms. By non-lisp we generally mean C or C++.

The cstruct object, that is an object of type (array excl::foreign), has been enhanced so that the first slot contains a Lisp object. make-array and the garbage collector have been enhanced to allow lisp-valued arrays to be stored in static space. Combining these two enhancements, you get the ability to allocate cstructs in static space and for those cstructs to contain a pointer to their lisp class (in the first slot).

Symbols implementing ftype functionality are in the :foreign-functions (nicknamed :ff) package.

2.0 The foreign types facility

The foreign types facility tries to blend the best of the Allegro CL Unix and aclwin foreign types facility.

• It is functionally based. The primitives are all functions yet there are compile-time macros to move some of the work to compile-time when that is possible.
• It is navigationally based rather than defstruct-like.
• The definition of a type does not determine on which heap it is allocated.
• There is a foreign structure allocation type that is typed, leading to easier debugging and runtime type and bounds checking.

3.0 Examples

To get an idea of how this facility works, here are some examples. First we show how we can define, allocate, set and access values in a foreign structure.

;; define the structure
user(3): (ff:def-foreign-type my-point (:struct (x :int) (y :int)))
#<foreign-functions::foreign-structure my-point>

;; allocate an object, using the default 
;; allocation type of :foreign
user(4): (setq obj (ff:allocate-fobject 'my-point))
#<foreign object of class my-point>

;; set a slot in the object
user(5): (setf (ff:fslot-value obj 'x) 3)
3

;; verify that the slot is set with the correct value
user(6): (ff:fslot-value obj 'x)
3

The def-foreign-type macro defines the my-point structure and returns the clos class that was defined. Note that the metaclass of a foreign structure is ff:foreign-structure.

Next an object is allocated with allocate-fobject. We didn't specify an allocation type, thus the type :foreign was used. A :foreign object is stored in the lisp heap in an (array excl::foreign) object (which is commonly called cstruct object). A nice feature of a :foreign object is that it is typed. You can use that type to specialize on objects of this foreign type in CLOS generic functions.

Another advantage of a foreign object being typed is that in the (setf fslot-value) calls we didn't have to specify the type of obj. The type was automatically determined at runtime.

Runtime determination of the type is handy and enhances the safety of the program since checks will be made at runtime to ensure that the desired access is appropriate for the object given. There is a cost to this check though, and if the foreign structure access is to be done many times, you'll want to use an accessor that allows you to specify the type at compile time:

user(9): (setf (ff:fslot-value-typed 'my-point :foreign obj 'x) 3)
3
user(10): (ff:fslot-value-typed 'my-point :foreign obj 'x)
3

The fslot-value-typed function takes two extra arguments: a type and an allocation method. With certain settings of the optimization values (safety, size, space, speed), the compiler will generate code to do the access in a few machine instructions.

Allocations and accesses can be done of types that have no name. These are called anonymous types.

user(15): (setq obj (ff:allocate-fobject '(:struct (x :int) (y :int))))
#<foreign object of class #:anon-type-2>
user(16): (setf (ff:fslot-value-typed 
                  '(:struct (x :int) (y :int)) :foreign obj 'x) 
                234)
234
user(17): (ff:fslot-value-typed 
               '(:struct (x :int) (y :int)) :foreign obj 'x)
234

When you use anonymous types, you must use fslot-value-typed. This may be relaxed in the future to permit fslot-value to be used.

4.0 The Syntax for Foreign Types

The type syntax of C is mostly postfix with occasional prefix bits. Also, C tries to get by describing structures using the fewest numbers of characters, and that doesn't always make things readable. Previous foreign types facilities have tried to mimic the C syntax, leading sometimes to even more confusion since they couldn't mimic it exactly. This facility uses prefix syntax exclusively and is a bit more verbose where that is warranted. The syntax for a foreign type (ftype below) is described next.

ftype := scalar-type
     composite-type
     function-type
     user-type

primitive-type :=   :fixnum
                    :int
                :long
                    :long-long (see note after chart)
                    :aligned
                :short
                :char
            :void
                :unsigned-int
                :unsigned-long
                    :unsigned-long-long (see note after chart)
                :unsigned-short
                :unsigned-char
                    :float
                    :double
                    :nat
                    :unsigned-nat
           
scalar-type :=  primitive-type
        (* ftype)
        (& ftype)
                (:aligned ftype)

composite-type := (:struct sfield ...)
          (:class  field ...)  
          (:union  field ...)
          (:array  ftype  [dim ...])
          

function-type  := (:function (ftype ...) ftype [attributes])

user-type  := <symbol>      [where <symbol> has an associated foreign type]
    
dim := <positive integer>

field := ftype
     (field-name ftype)

[fields in structures can contain bit specifiers]

sfield := field
          bit-specifier
          multibit-specifier

bit-specifier :=
          (:bit number-of-bits)
          (field-name (:bit number-of-bits))

number-of-bits := <integer>

multibit-specifier :=
      (:bits number-of-bytes mbit-specifier ...)
          (field-name (:bits number-of-bytes mbit-specifier ...))

mbit-specifier  :=
          number-of-bits
          (field-name number-of-bits)

number-of-bytes := <integer>

Some notes on the syntax above:

• The :aligned primitive type is not exactly a type. It uses the fact that a Lisp fixnum corresponds in its bit pattern to an address aligned along an 8 byte boundary (in 64-bit Lisps) or a 4-bit boundary (in 32-bit Lisps) and so allows passing such addresses about without having to create bignums to refer to them. (Creating bignums has a performance cost but more importantly causes consing and so may force a garbage collection). The :aligned type is useful as long as the programmer is aware that the values are fixnum objects in Lisp and integers if foreign code, so adding 1 to the value is Lisp is the same as adding 8 (in 64-bit Lisps) or 4 (in 32-bit Lisps) when foreign code sees the value. See Aligned Pointers and the :aligned type below.
• In scalar-type, the syntax (& ftype) denotes a C++ reference type. Currently we treat these just like pointer types, but we may make a distinction at some time in the future. In def-foreign-call the type specifier (& ftype) means pass a reference to the argument to the foreign function.
• In scalar-type, the syntax (:aligned ftype) denotes a pointer to an ftype where the low three bits (in 64-bit Lisps) or two bits (in 32-bit Lisps) of the pointer will always be zero. Accessing such a slot will return an aligned pointer to the C object which will always fit into a fixnum. (:aligned ftype) cannot be used as a return type. See See Aligned Pointers and the :aligned type below.
• A field name in a :struct composite type may be nil which means that the slot does not have a name, but is present in the structure. This is useful for padding structures to a certain size.
• The function type is generally found inside a pointer type, that is a structure would contain a pointer to a function rather than the function itself. In a class however a field with a function type denotes a member function of that class, and such a function is not stored in the class object itself. We do not yet make this distinction in :class objects. Therefore, do not declare slots to be simply of type :function.
• The :fixnum type is like the :long type except that it implies that the value in that slot is between most-negative-fixnum and most-positive-fixnum inclusive. By specifying this limit on the value of the slot Allegro can generate faster code to access and set the slot.
• A contiguous sequence of bit-fields is always padded out to an integral number of bytes. This applies to array elements as well. A field of type (:array (:bit 4) 8) is 8 bytes wide since each element gets padded out to a whole byte.
• :nat and :unsigned-nat: :nat is short for "natural", and is intended to be the size of integer which fits into the natural word size of the machine - 32 bits on a 32-bit lisp and 64 bits on a 64-bit lisp. Note that :long and :unsigned-long do not fit this bill - they are 64-bits in some 32-bit lisps and 32 bits in the Windows XP/64 64-bit Lisp. See The :nat and :unsigned-nat types in implementation.html.
• The :long-long and :unsigned-long-long types are 64-bit values, the same as :long in 64-bit Lisps. They can be used anywhere a foreign type can be used except in callbacks, where they are not supported. Also, call-direct calls cannot be made with arguments of these types (so an attempt to define a foreign function as :call-direct with an argument specified as :long-long or :unsigned-long-long will signal a warning and the forsign call will work, but not be call-direct).

4.1 The union of two structs

Here is an example of the union composite type.

(in-package :user)
(use-package :ff)

(def-foreign-type changeable
  (:struct
   (key :int)
   (varying
    (:union
     (numbers 
      (:struct
       (a :int)
       (b :int)
       (c :int)))
     (strings
      (:struct
       (d (:array :char 6))
       (c (:array :char 6))))
     ))
   ))

(defun try-changeable ()
  (let ((ch (allocate-fobject 'changeable :foreign)))
    (setf (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'a) 123)
    (setf (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'b) 456)
    (setf (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'c) 789)
    (format t "~& slots of numbers struct ~S ~S ~S ~%"
        (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'a)
        (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'b)
            (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'c))

    (dotimes (i 6)
      (setf (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'd i)
          (char-code (elt "abcdef" i)))
      (setf (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'c i)
          (char-code (elt "hijklm" i)))
      )
    (format t "~& slots of strings struct ~S ~S ~%"
        (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'd 0)
        (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'c 0)
            )
    ch))

-------------------------------
cl-user(20): (try-changeable)
 slots of numbers struct 123 456 789 
 slots of strings struct 97 104 
#<foreign object of class changeable>
cl-user(21):  :i *
A new short simple foreign vector (6) @ #x205afd02
   0-> simple t vector (7) = #(483681 t 901 ...)
   1-> The field #x00000000
   2-> The field #x64636261
   3-> The field #x69686665
   4-> The field #x6d6c6b6a
   5-> The field #x00000000
cl-user(22): 

5.0 Primitive Types

The sizes of the primitives types vary by machine architecture, as this table shows.

6.0 Allocation types

Objects can be allocated in a variety of places. The default allocation location is :foreign.

• :foreign - The object is allocated in the lisp heap using a lisp data type unique to Allegro called the (array excl::foreign). This kind of array has one lisp slot at the beginning, with all subsequent slots holding raw integer data. The foreign types facility uses the first, lisp-valued, slot to hold a pointer to a description of the type of data held in this object. Because these objects are stored in the Lisp heap, it doesn't make sense to ask for the address of a slot of one of these objects, as it may have moved by the time you get your answer.

• :foreign-static-gc - These objects look just like :foreign allocated objects. The difference is that they are allocated in static space (i.e. space allocated with the C function aclmalloc). Unlike other objects allocated in static space, these objects are automatically gc'ed (the C aclfree function is called on them) when the last pointer to them in the Lisp heap disappears. The advantage of this allocation type over :foreign is that the object doesn't move. This makes it very useful when the object must be passed to a C foreign-function that releases the heap (e.g. most functions in the Windows package). See Releasing the heap when calling foreign functions in foreign-functions.html for more information on releasing the heap.

• :lisp - A lisp array of type (unsigned-byte 8) of sufficient size to hold the object is allocated in the lisp heap. The object does not record the foreign type of which it is an instance. This allocation type is useful if, for example, you've got a binary file you want to read and take apart, and the format of the file is described by a C struct definition. In that case you can allocate an object of type :lisp and then call read-sequence to read the data from the binary file into the foreign object. Note, however, that read-sequence will also work for :foreign allocated objects. There is one tricky case: if the foreign structure requires 8 byte alignment (e.g. it begins with a double float and you are running on an architecture like the Sparc that requires 8 byte alignment for double floats) then the first four bytes of the Lisp array allocated will not be used by fslot-value-typed. If you read-sequence into such an array, then your data will be displaced 4 bytes from where fslot-value-typed thinks it is. In addition to read-sequence, used as an example, this type also works with read-vector, write-sequence, and write-vector.

  The function foreign-type-lisp-pre-padding returns the amount of this offset for the given structure on the current architecture.

• :lisp-short - A lisp short-array of type (unsigned-byte 8) of sufficient size to hold the object is allocated in the lisp heap. This type is the same as the :lisp type described just above, except that the data is held in a short-array. (short-arrays are a new data type corresponding to the older array type in Allegro CL. See Arrays and short arrays section in implementation.html.) Except for using a short-array, this type is the same as :lisp.

• :c - The object is allocated in static space (via *aclmalloc() *which is like C's malloc except that the data is preserved through a call to dumplisp) and a pointer to the space is returned. Thus the object is represented by an integer value. The object is not automatically freed. You must do the freeing with free-fobject. Be warned that some pointers can only be represented in Lisp with bignums (because pointers are (unsigned-byte 32) in 32-bit Lisps and (unsigned-byte 64) in 64-bit Lisps, but fixnums are represented by 30 or 61 bits (in 32-bit Lisps and 64-bit Lisps respectively). If this is an issue, you might use :aligned instead.

• :aligned - This is just like :c except that it returns an aligned pointer. See Aligned Pointers and the :aligned type below for more information about the :aligned type.

7.0 Aligned Pointers and the :aligned type

The :aligned foreign type is described here. Related is the :aligned specification as argument and return values values for def-foreign-call and defun-foreign-callable.

Lisp can reference data stored in the Lisp heap or outside the heap in what we call C-space. Objects in C-space are normally referenced by their addresses. On 32-bit machines, pointers are unsigned 32-bit integers and on 64-bit machines, pointers are unsigned 64-bit integers.

But the full range of unsigned 32 or 64 bit integers cannot be represented as Lisp fixnums because the lower 2 (in 32-bit Lisps) or 3 (in 64-bit Lisps) bits are the tag identifying the object as a fixnum and not part of the value. High addresses therefore must be respresented as bignums. Creating and manipulating bignums takes longer than creating and manipulating fixnums but more importantly it causes consing, and consing can result in garbage collections which cause Lisp objects to move. Foreign code which calls back to Lisp or Lisp code which calls out to foreign code may need to avoid consing in order to avoid garbage collection but if such code needs to manipulate pointers from the full address range, consing may occur.

Within Lisp, an aligned pointer looks like a fixnum, as we have said. If it looks like a positive fixnum, its value as a fixnum is the actual pointer value divided by 8 (in 64-bit Lisp) or 4 (in 32-bit Lisps). If it looks like a negative fixnum, things are too complicated for a simple formula.

The :aligned type allows for aligned pointers (those whose lower two bit are 0 on 32-bit machines and whose lower 3 bits are 0 on 64-bit machines) to be passed about, being treated as pointers in foreign code and as fixnums is Lisp code. This can be useful in foreign calls and call backs and in defining foreign types.

Considering foreign calls and call backs first, when an argument to the function being defined by def-foreign-call is specified to be of type :aligned, it must be a fixnum when passed to the foreign function. The whole bit pattern including zeros in the lowest 2 (32-bit) or 3 (64-bit) bits is passed to the foreign code. Similarly, when the argument to a defun-foreign-callable function is specified as :aligned, it will be received by Lisp with its bits unmodified and in Lisp will be treated as a fixnum. Consider these examples:

(def-foreign-call my-ff ((x :aligned) (y :int)) ...)

Suppose we call my-ff with arguments 1 and 1. These are Lisp fixnums so their internal respresentations in Lisp are (in 64-bit Lisps) #b1000 -- the value (1) and the fixnum tag (#b000). The foreign code will see as arguments #b1000 and #b1, that is the x argument is not converted and the y argument is.

It is similar with defun-foreign-callable. Consider the following:

(defun-foreign-callable add-c-aligned-to-int ((x :aligned) (y :int))
  (setq *x7* (+ x y)))

If foreign calls back into Lisp with (again in a 64-bit Lisp) with values #b1000 and #b1, the value of x7 will be 2. (The x argument, passed in unchanged, will be interpreted in Lisp as the fixnum 1 and the y argument will be converted to the fixnum 1.)

The :aligned type can be used in foreign type specifications as well.

There are two ways to create an aligned pointer: allocating an object with an :aligned allocation type or referencing a slot of a foreign object that is declared of type (:aligned some-type). Next, we'll show these cases in detail.

Given any foreign type foo we can allocate an aligned pointer to it with

(allocate-fobject 'foo :aligned)

The return value will always be a fixnum.

Suppose we have types point and rect:

(def-foreign-type point
    (:struct (x :int)
             (y :int)))

(def-foreign-type rect
    (:struct  (topleft (* point))
              (bottomright (:aligned point))))

Suppose the variable rr contains a pointer to a rect object. We'll further assume that the pointer to the rect object was passed back to us from a C program and that the pointer is a normal :c (not aligned) pointer. We can access the x slot of the topleft and bottomright fields using the same kind of expression:

(fslot-value-typed 'rect :c rr :topleft '* :x)
(fslot-value-typed 'rect :c rr :bottomright '* :x)

This shows that you can treat the (* ftype) and (:aligned ftype) specifier the same when you're referencing objects through them.

If you just access the pointer values, you'll see big differences.

(fslot-value-typed 'rect :c rr :topleft)

is a normal :c pointer whereas

(fslot-value-typed 'rect :c rr :bottomright)

is an :aligned pointer.

Using aligned pointers requires careful programming. Here are the rules for using aligned pointers:

1. If an aligned pointer is used in fslot-value-typed or (setf fslot-value-typed) then the allocation type of :aligned must be specified.

2. Setting a slot of an object declared to be (:aligned some-type) must be done with an aligned pointer. The function address-to-aligned is useful in creating an aligned pointer from a normal :c pointer. aligned-to-address takes an aligned pointer and returns the object.

3. When accessing a slot of an object declared as (:aligned some-type), the pointer contained therein must have its low two bits zero. Failure to abide by this will likely result in illegal lisp object pointers being stored in the heap, which will usually cause lisp to exit during the next garbage collection when the illegal pointers are discovered.

4. fslot-address and fslot-address-typed will always return a normal **:c **pointer.

8.0 Bit Fields

Since C compilers use a variety of alignment and packing rules for bitfields, the Allegro CL foreign type facility must attempt to accommodate all of them. Therefore the basic facility allows bitfields to be packed into bytes on arbitrary byte boundaries.

The def-foreign-type definition of a particular structure must be adapted to the format required by a specific compiler.

For example, consider the following declaration:

struct {
    long a[1];
    char aa;
    unsigned int b : 3;
    unsigned int c : 5;
    unsigned int d : 3;
    unsigned int e : 7;
    unsigned int f : 17;
    char w;
    long z;
};

MSVC 2.1 allocates 6 longs with a, aa, b, f, w, and z on long boundaries. Gcc 2.7 on Solaris allocates only 5 longs by packing the fields b, c, d, and e into the the 3 bytes following aa.

The following Allegro CL definition would match the layout generated by the Microsoft Visual C compiler or the GNU C compiler on Solaris.

(def-foreign-type foo
  (:struct
     (a (:array :long 1))
     (aa :char)
     #+mswindows
     (:array :char 3) ;; filler needed to match MSVC alignment
     (b (:bit 3))
     (c (:bit 5))
     (d (:bit 3))
     (e (:bit 7))
     (:bit 14)       ;; filler to align next field to int
     (f (:bit 17))
     (:bit 15)       ;; filler to align next field to int
     (w :char)
     (z :long)))

9.0 The Programming Interface

The foreign type interface includes the following operators:

10.0 Passing Foreign Objects to Foreign Functions

We will take a bottom up approach to describing just what foreign type descriptions mean, and how that relates to what a C program would see receiving a foreign object. We'll use the :int type as an example.

1. Suppose you want to pass an integer to a foreign function. You do that by just passing the integer value in the foreign function call. You don't need to use the foreign type structures at all.

2. Suppose you execute this: (allocate-fobject :int) What does this do? Does it return an integer? No, it doesn't. As per case 1, we can use lisp integers to represent integers to be passed to foreign code. In order to understand (allocate-fobject :int) you should remember that allocate-fobject always allocates foreign structures (or arrays of foreign structures) and thus you can rewrite this as (allocate-fobject (:struct (nil :int))) [The nil means that this slot has no name.] Thus you can see that allocate-fobject is going to create a foreign structure that has one field, an :int valued field. If you pass the result of this allocate-fobject to C, what happens is that the lisp foreign-function interface passes a pointer to the start of the :int field. Thus the C program should declare a parameter of type int * to receive this value.

3. Suppose you execute this: (allocate-fobject '(* :int)) From the discussion above we can conclude that this creates a structure with one unnamed field of type (* :int). When this object is passed to C, it should declare the argument as int **

4. Suppose you execute this: (allocate-fobject '(:array :int 5)) You end up with a structure with 5 :int objects. If passed to C, you can either declare the arguments to be int * or int[], depending on the syntax you want to use to reference the objects.

Now lets look at what fslot-value operations are possible on each of the kinds of objects mentioned above.

1. a raw integer: lisp integers are constants. Their values can't be changed.

2. (setq x (allocate-fobject :int)) This is a structure with single unnamed slot of type :int. We might refer to this an an :int box. We can get the value with (fslot-value x) and set it with (setf (fslot-value x) 4) or, more verbosely: get the value with (fslot-value-typed :int nil x) and set it with (setf (fslot-value-typed :int nil x) 4)

3. (setq x (allocate-fobject '(* :int))) with this we can either get/set the value in the box, or what it points to. To set the value in the box: (setf (fslot-value x) 1321231) or (setf (fslot-value-typed '(* :int) nil x) 1321231). To set the value at the location pointed to by the value in the box (setf (fslot-value x '*) 1231) or (setf (fslot-value-typed '(* :int) nil x '*) 1231)

4. (setq x (allocate-fobject '(:array :int 5))) we can get/set each individual object, for example: (setf (fslot-value x 3) 4444) or (setf (fslot-value-typed '(:array :int 5) nil x 3) 4444)

Notes:

1. A foreign type X is equivalent to the foreign type (:struct (nil X))

2. We never specified an allocation type in the examples above, and this is because what is written above applies to all allocation types (:foreign, :foreign-static-gc, :lisp, :c)

10.1 String representation

Allegro CL has images that represent characters with 16-bits and images that represent characters with 8-bits (only one representation is available in each image). See Allegro CL Executables in startup.html for a list of executables.

If you create a foreign type for a string and store a Lisp string in it, what character representation will be used? Consider this definition:

(def-foreign-type foo
  (:struct (str (:array :char 20))))

(setq obj (allocate-fobject 'foo :foreign-static-gc))

Now store a string in the array:

(setf (fslot-value obj 'str) "abcde")

The system will store the external format of the string (computed by string-to-native) into the foreign object. In a lisp with 8-bit characters the external format of a string is identical to the string itself. In a Lisp with 16-bit characters, it will not be.

A lisp string can be created from the values in the foreign object by

(native-to-string (fslot-value obj 'str))

See native-to-string.

10.2 A Quiz

Now a little quiz to see how well you understand what was done above. A lisp function is passed an integer value str which is the address of a sequence of characters in memory. How do you access the third character in the string?

The answer to the quiz: The correct answer is b. Answer a can't be right since the type (* :char) is the same as (:struct (nil (* :char))) and thus says that str points to a structure in memory with one slot, that slot being a pointer to a character string. But we know that str itself points to the string. Answer b is correct. The size of the array we specified isn't important (as long as it is greater than the index we are using to access the array).

Calling C code:

As we've seen above, we take a C type (be it a struct or primitive type) and create a Lisp equivalent type. Let X be the C type and Y the Lisp type. When we pass Y to C, the C code gets not an X object but an X* object since we always pass a pointer to a foreign structure. Likewise when C returns an X structure, it doesn't usually return the X structure, it returns an X* value. Lisp then sees that value as an instance of the Y foreign type. Thus going to C we add a * to the type since we pass by reference. Coming back from C we remove a * from the type since Lisp always refers to types by pointers, thus using the * is superfluous.

Copyright (c) Franz Inc. Lafayette, CA., USA. All rights reserved.