| Allegro CL version 8.2 Unrevised from 8.1 to 8.2. 8.1 version |
This document contains the following sections:
1.0 Foreign types introductionThe purpose of the foreign types facility is to permit the creation, reading and modification of objects that are described in non-lisp terms. By non-lisp we generally mean C or C++.
The cstruct object, that is an object of type (array excl::foreign), has been enhanced so that the first slot contains a Lisp object. make-array and the garbage collector have been enhanced to allow lisp-valued arrays to be stored in static space. Combining these two enhancements, you get the ability to allocate cstructs in static space and for those cstructs to contain a pointer to their lisp class (in the first slot).
Symbols implementing ftype functionality are in
the :foreign-functions
(nicknamed :ff
) package.
The foreign types facility tries to blend the best of the Allegro CL Unix and aclwin foreign types facility.
To get an idea of how this facility works, here are some examples. First we show how we can define, allocate, set and access values in a foreign structure.
;; define the structure user(3): (ff:def-foreign-type my-point (:struct (x :int) (y :int))) #<foreign-functions::foreign-structure my-point> ;; allocate an object, using the default ;; allocation type of :foreign user(4): (setq obj (ff:allocate-fobject 'my-point)) #<foreign object of class my-point> ;; set a slot in the object user(5): (setf (ff:fslot-value obj 'x) 3) 3 ;; verify that the slot is set with the correct value user(6): (ff:fslot-value obj 'x) 3
The def-foreign-type macro defines the my-point structure and returns the clos class that was defined. Note that the metaclass of a foreign structure is ff:foreign-structure.
Next an object is allocated with allocate-fobject. We didn't
specify an allocation type, thus the type
:foreign
was used. A
:foreign
object is stored
in the lisp heap in an (array
excl::foreign)
object (which is commonly called
cstruct object). A nice feature of a
:foreign
object is that it is typed. You can use that type to specialize on
objects of this foreign type in CLOS generic
functions.
Another advantage of a foreign object being typed is that in the (setf fslot-value) calls we didn't have to specify the type of obj. The type was automatically determined at runtime.
Runtime determination of the type is handy and enhances the safety of the program since checks will be made at runtime to ensure that the desired access is appropriate for the object given. There is a cost to this check though, and if the foreign structure access is to be done many times, you'll want to use an accessor that allows you to specify the type at compile time:
user(9): (setf (ff:fslot-value-typed 'my-point :foreign obj 'x) 3) 3 user(10): (ff:fslot-value-typed 'my-point :foreign obj 'x) 3
The fslot-value-typed function takes two extra arguments: a type and an allocation method. With certain settings of the optimization values (safety, size, space, speed), the compiler will generate code to do the access in a few machine instructions.
Allocations and accesses can be done of types that have no name. These are called anonymous types.
user(15): (setq obj (ff:allocate-fobject '(:struct (x :int) (y :int)))) #<foreign object of class #:anon-type-2> user(16): (setf (ff:fslot-value-typed '(:struct (x :int) (y :int)) :foreign obj 'x) 234) 234 user(17): (ff:fslot-value-typed '(:struct (x :int) (y :int)) :foreign obj 'x) 234
When you use anonymous types, you must use fslot-value-typed. This may be relaxed in the future to permit fslot-value to be used.
The type syntax of C is mostly postfix with occasional prefix bits. Also, C tries to get by describing structures using the fewest numbers of characters, and that doesn't always make things readable. Previous foreign types facilities have tried to mimic the C syntax, leading sometimes to even more confusion since they couldn't mimic it exactly. This facility uses prefix syntax exclusively and is a bit more verbose where that is warranted. The syntax for a foreign type (ftype below) is described next.
ftype := scalar-type composite-type function-type user-type primitive-type := :fixnum :int :long :short :char :void :unsigned-int :unsigned-long :unsigned-short :unsigned-char :float :double :nat :unsigned-nat scalar-type := primitive-type (* ftype) (& ftype) (:aligned ftype) composite-type := (:struct sfield ...) (:class field ...) (:union field ...) (:array ftype [dim ...]) function-type := (:function (ftype ...) ftype [attributes]) user-type := <symbol> [where <symbol> has an associated foreign type] dim := <positive integer> field := ftype (field-name ftype) [fields in structures can contain bit specifiers] sfield := field bit-specifier multibit-specifier bit-specifier := (:bit number-of-bits) (field-name (:bit number-of-bits)) number-of-bits := <integer> multibit-specifier := (:bits number-of-bytes mbit-specifier ...) (field-name (:bits number-of-bytes mbit-specifier ...)) mbit-specifier := number-of-bits (field-name number-of-bits) number-of-bytes := <integer>
:class
objects. Therefore, do not
declare slots to be simply of type :function
.
most-negative-fixnum
and most-positive-fixnum
inclusive. By specifying
this limit on the value of the slot Allegro can generate faster code
to access and set the slot.
(:array (:bit 4) 8)
is 8 bytes wide
since each element gets padded out to a whole byte.
Here is an example of the union composite type.
(in-package :user) (use-package :ff) (def-foreign-type changeable (:struct (key :int) (varying (:union (numbers (:struct (a :int) (b :int) (c :int))) (strings (:struct (d (:array :char 6)) (c (:array :char 6)))) )) )) (defun try-changeable () (let ((ch (allocate-fobject 'changeable :foreign))) (setf (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'a) 123) (setf (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'b) 456) (setf (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'c) 789) (format t "~& slots of numbers struct ~S ~S ~S ~%" (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'a) (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'b) (fslot-value-typed 'changeable :foreign ch 'varying 'numbers 'c)) (dotimes (i 6) (setf (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'd i) (char-code (elt "abcdef" i))) (setf (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'c i) (char-code (elt "hijklm" i))) ) (format t "~& slots of strings struct ~S ~S ~%" (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'd 0) (fslot-value-typed 'changeable :foreign ch 'varying 'strings 'c 0) ) ch)) ------------------------------- cl-user(20): (try-changeable) slots of numbers struct 123 456 789 slots of strings struct 97 104 #<foreign object of class changeable> cl-user(21): :i * A new short simple foreign vector (6) @ #x205afd02 0-> simple t vector (7) = #(483681 t 901 ...) 1-> The field #x00000000 2-> The field #x64636261 3-> The field #x69686665 4-> The field #x6d6c6b6a 5-> The field #x00000000 cl-user(22):
The sizes of the primitives types vary by machine architecture, as this table shows.
Type | Size | Alignment | Notes |
:void |
0 | 0 | Used in (* :void)
type specifications and used to prototype a foreign call
of no arguments, just like C and C++ do (i.e. an argument list
specification of () really says that we don't know how many
arguments there are, whereas an arg list specification of
(:void) says that there are precisely 0 arguments expected.) |
:char |
1 | 1 | A signed one byte access |
:unsigned-char |
1 | 1 | [none] |
:short |
2 | 2 | A signed two byte access |
:unsigned-short |
2 | 2 | [none] |
:int |
4 | 4 | A signed four byte access |
:unsigned-int |
4 | 4 | [none] |
:long |
4 on all machines except the Dec Alpha, IRIX, and all 64-bit implementations (Alpha, HP, Sparc), where it is 8 | 4 on all machines except the Dec Alpha, IRIX, and all 64-bit implementations (Alpha, HP, Sparc), where it is 8 | A signed access of an architecture specific size. |
:unsigned-long |
4 on all machines except the Dec Alpha, IRIX, and all 64-bit implementations (Alpha, HP, Sparc), where it is 8 | 4 on all machines except the Dec Alpha, IRIX, and all 64-bit implementations (Alpha, HP, Sparc), where it is 8 | [none] |
:float |
4 | 4 | [none] |
:double |
8 | 8 on all machines except Linux and FreeBSD on an x86 and the RS/6000 where it is 4 | [none] |
Objects can be allocated in a variety of places. The default
allocation location is :foreign
.
:foreign
- The object is allocated in the
lisp heap using a lisp data type unique to Allegro called the
(array excl::foreign)
. This kind of array has
one lisp slot at the beginning, with all subsequent slots holding
raw integer data. The foreign types facility uses the first,
lisp-valued, slot to hold a pointer to a description of the type of
data held in this object. Because these objects are stored in the
Lisp heap, it doesn't make sense to ask for the address of a slot of
one of these objects, as it may have moved by the time you get your
answer.
:foreign-static-gc
- These
objects look just like :foreign
allocated
objects. The difference is that they are allocated in static space
(i.e. space allocated with the C function aclmalloc). Unlike
other objects allocated in static space, these objects are
automatically gc'ed (the C aclfree function is called on
them) when the last pointer to them in the Lisp heap disappears.
The advantage of this allocation type over
:foreign
is that the object doesn't move. This
makes it very useful when the object must be passed to a C
foreign-function that releases the heap (e.g. most functions in the
Windows package).
See Releasing the
heap when calling foreign functions in
foreign-functions.htm for more information on
releasing the heap.
:lisp
- A lisp array of type
(unsigned-byte 8)
of sufficient size to hold the
object is allocated in the lisp heap. The object does not record the
foreign type of which it is an instance. This allocation type is
useful if you've got a binary file you want to read and take apart,
and the format of the file is described by a C struct definition. In
that case you can allocate an object of type :lisp
and then call read-sequence to read the data from the binary
file into the foreign object. Note, however, that read-sequence will also work for
:foreign
allocated objects. There is one tricky
case: if the foreign structure requires 8 byte alignment (e.g. it
begins with a double float and you are running on an architecture like
the Sparc that requires 8 byte alignment for double floats) then the
first four bytes of the Lisp array allocated will not be used by
fslot-value-typed. If you
read-sequence into such an array, then your data will be
displaced 4 bytes from where fslot-value-typed thinks it is.
The function foreign-type-lisp-pre-padding returns the amount of this offset for the given structure on the current architecture.
:lisp-short
- A lisp short-array of type
(unsigned-byte 8)
of sufficient size to hold the
object is allocated in the lisp heap. This type is the same as the
:lisp
type described just above, except that the
data is held in a short-array. (short-arrays are a new data type
corresponding to the older array type in Allegro CL. See Arrays and short
arrays section in implementation.htm.)
Except for using a short-array, this type is the same as
:lisp
.
:c
- The object is
allocated in static space (via aclmalloc() which is like
C's malloc except that the data is preserved through a call
to dumplisp) and a pointer to the space is returned. Thus the object
is represented by an integer value. The object is
not automatically freed. You must do the freeing
with free-fobject
:aligned
- This is just like
:c
except that it returns an aligned
pointer.
Lisp can reference data stored in the Lisp heap or outside the heap in what we call C-space. Objects in C-space are normally referenced by their addresses, and on many platforms addresses in C-space are so large they must be represented as bignums. In most programs the overhead of allocating a bignum to represent a C-space object isn't a big factor in the overall speed of the program. In those cases where the allocation is a problem we offer the aligned pointer to a C-space object.
If the address of the C-space object is a multiple of four (in 32-bit Lisps) or eight (in 64-bit Lisps), then we can divide the address by four/eight and its value will be small enough to be represented by a Lisp fixnum object. We call such a fixnum an aligned pointer. There is no space overhead in allocating a fixnum.
It's impossible for Lisp to distinguish an aligned pointer from a normal C-space pointer. Thus when an aligned pointer is used, the programmer must specify that the value is an aligned pointer.
There are two ways to create an aligned pointer: allocating an object
with an :aligned
allocation type or referencing a
slot of a foreign object that is declared of type
(:aligned some-type)
. Next, we'll show
these cases in detail.
Given any foreign type foo we can allocate an aligned pointer to it with
(allocate-fobject 'foo :aligned)
The return value will always be a fixnum.
Suppose we have types point and rect:
(def-foreign-type point (:struct (x :int) (y :int))) (def-foreign-type rect (:struct (topleft (* point)) (bottomright (:aligned point))))
Suppose the variable rr contains a pointer to a rect object. We'll further assume that the pointer to the rect object was passed back to us from a C program and that the pointer is a normal :c (not aligned) pointer. We can access the x slot of the topleft and bottomright fields using the same kind of expression:
(fslot-value-typed 'rect :c rr :topleft '* :x) (fslot-value-typed 'rect :c rr :bottomright '* :x)
This shows that you can treat the (* ftype) and (:aligned ftype) specifier the same when you're referencing objects through them.
If you just access the pointer values, you'll see big differences.
(fslot-value-typed 'rect :c rr :topleft)
is a normal :c pointer whereas
(fslot-value-typed 'rect :c rr :bottomright)
is an :aligned pointer.
Using aligned pointers requires careful programming. Here are the rules for using aligned pointers:
(:aligned
some-type)
must be done with an aligned pointer. The
function address-to-aligned is useful in
creating an aligned pointer from a normal :c
pointer. aligned-to-address takes an
aligned pointer and returns the object.
(:aligned
some-type)
, the pointer contained therein must have its low
two bits zero. Failure to abide by this will likely result in illegal
lisp object pointers being stored in the heap, which will usually
cause lisp to exit during the next garbage collection when the illegal
pointers are discovered.
Since C compilers use a variety of alignment and packing rules for bitfields, the Allegro CL foreign type facility must attempt to accommodate all of them. Therefore the basic facility allows bitfields to be packed into bytes on arbitrary byte boundaries.
The def-foreign-type definition of a particular structure must be adapted to the format required by a specific compiler.
For example, consider the following declaration:
struct { long a[1]; char aa; unsigned int b : 3; unsigned int c : 5; unsigned int d : 3; unsigned int e : 7; unsigned int f : 17; char w; long z; };
MSVC 2.1 allocates 6 longs with a, aa, b, f, w, and z on long boundaries. Gcc 2.7 on Solaris allocates only 5 longs by packing the fields b, c, d, and e into the the 3 bytes following aa.
The following Allegro CL definition would match the layout generated by the Microsoft Visual C compiler or the GNU C compiler on Solaris.
(def-foreign-type foo (:struct (a (:array :long 1)) (aa :char) #+mswindows (:array :char 3) ;; filler needed to match MSVC alignment (b (:bit 3)) (c (:bit 5)) (d (:bit 3)) (e (:bit 7)) (:bit 14) ;; filler to align next field to int (f (:bit 17)) (:bit 15) ;; filler to align next field to int (w :char) (z :long)))
The foreign type interface includes the following operators:
Name | Arguments | Notes |
address-to-aligned | address | Convert the integer pointer to an object in memory to an aligned pointer. See the section Section 7.0 Aligned Pointers for more information. |
aligned-to-address | aligned | Convert the aligned pointer (which is a fixnum) to the address of the object into memory to which it points. See the section Section 7.0 Aligned Pointers for more information. |
allocate-fobject | type &optional allocation size | Allocate an object of the given type in heap described by the allocation argument. If the size argument is given, then it is the minimum size (in bytes) of the data portion of the object that will be allocated. The valid allocation arguments are shown above. |
canonical-ftype | type | If type is or names a foreign type, return the symbol or list that describes that type, otherwise return nil. If type is a symbol defined using def-foreign-type, then the definition form is returned. If type is one of the primitive foreign type symbols or is a list in the form valid for def-foreign-type, then type itself is returned. If type is a symbol that has been given a foreign type definition through def-foreign-type, then the foreign definition is returned. Using canonical-ftype allows a quick determination of whether a symbol names a simple type or a structured type. |
describe-fobject | fobject &optional ftype | This function prints a description of the contents of a foreign object. |
def-foreign-type | name definition | defines name to be a user-defined foreign type with the given definition. Name must either be a symbol or a list beginning with a symbol and followed by attributes (see below). Definition is not evaluated and must be a foreign type description. |
ensure-foreign-type | &key name definition | This is the functional equivalent of the macro def-foreign-type. |
fobjectp | obj | This function returns t if
object is appropriate as an argument to foreign
type accessors. |
free-fobject | obj | Free an object that was allocated by allocate-fobject with the :allocation
of :c . An object should only be freed once. |
free-fobject-aligned | obj | Free an object that was allocated by allocate-fobject with the :allocation
of :aligned . An object should only be freed once. |
fslot-value-typed | type allocation object &rest slot-names | Access a slot from an object. The type must be
The allocation must be one of |
fslot-value | object &rest slot-names | This is like fslot-value-typed
except it can only be used to access slots from objects with
:foreign or :foreign-static-gc
allocations, since these are the only objects that are runtime typed.
This function is a
lot more convenient to use than fslot-value-typed
since the type and allocation needn't be specified,
however it can't at present be open
coded. Thus for speed critical parts of the program,
fslot-value-typed
should be used. |
fslot-address-typed | type allocation object &rest slot-names | This is just like fslot-value-typed
except that it returns the address of the object rather than the value. Asking for the
address of a :lisp allocated object isn't useful since that object can
move during a garbage collection and a program can't predict when a garbage collection can
occur. |
fslot-address | object &rest slot-names | This is just like fslot-address-typed
except that it works only for :foreign and :foreign-static-gc
objects and can't be open coded by the compiler. |
foreign-type-p | name | name must be a symbol. If name is the name of a foreign type defined using this facility, then t is returned. |
sizeof-fobject | ftype | ftype must be a symbol naming a foreign type. The size of an object of the foreign type ftype is returned. |
with-stack-fobject | (var type) &rest body) | Allocate an object of type type on the stack and bind it to var while evaluating body |
with-stack-fobjects | bind-clauses &rest body) | This variant of with-stack-fobject allows multiple bindings. |
We will take a bottom up approach to describing just what foreign type
descriptions mean, and how that relates to what a C program would see
receiving a foreign object. We'll use the :int
type
as an example.
nil
means that this slot has no name.]
Thus you can see that allocate-fobject is going to create a foreign
structure that has one field, an :int
valued field. If you pass the
result of this allocate-fobject to C, what happens is that the lisp
foreign-function interface passes a pointer to the start of the
:int
field. Thus the C program should declare a parameter of type 'int *' to
receive this value.(* :int)
. When this object is passed to C,
it should declare the
argument as 'int **
' :int
objects.
If passed to C, you can either
declare the arguments to be 'int *
' or
int[]
, depending
on the syntax you want to use to reference the objects. Now lets look at what fslot-value operations are possible on each of the kinds of objects mentioned above.
(setq x (allocate-fobject :int))
This is a structure with single unnamed slot of type
:int
. We might refer
to this an an :int
box. We can get the value
with (fslot-value x)
and set it with (setf (fslot-value x) 4)
or,
more verbosely: get the
value with (fslot-value-typed :int nil x)
and set it with (setf
(fslot-value-typed :int nil x) 4)
(setq x (allocate-fobject '(* :int)))
with this we can either get/set the value in the box, or what it
points to. To set the
value in the box: (setf (fslot-value x) 1321231)
or (setf
(fslot-value-typed '(* :int) nil x) 1321231)
.
To set the value at the
location pointed to by the value in the box
(setf (fslot-value x '*) 1231)
or (setf (fslot-value-typed '(* :int) nil x '*) 1231)
(setq x (allocate-fobject '(:array :int 5)))
we can get/set each individual object, for example:
(setf (fslot-value x 3) 4444)
or (setf (fslot-value-typed '(:array :int 5) nil x 3) 4444)
Notes:
X
is equivalent to the foreign type (:struct
(nil X))
:foreign
,
:foreign-static-gc
,
:lisp
, :c
) Allegro CL has images that represent characters with 16-bits and images that represent characters with 8-bits (only one representation is available in each image). See Allegro CL Executables in startup.htm for a list of executables.
If you create a foreign type for a string and store a Lisp string in it, what character representation will be used? Consider this definition:
(def-foreign-type foo (:struct (str (:array :char 20)))) (setq obj (allocate-fobject 'foo :foreign-static-gc))
Now store a string in the array:
(setf (fslot-value obj 'str) "abcde")
The system will store the external format of the string (computed by string-to-native) into the foreign object. In a lisp with 8-bit characters the external format of a string is identical to the string itself. In a Lisp with 16-bit characters, it will not be.
A lisp string can be created from the values in the foreign object by
(native-to-string (fslot-value obj 'str))
See native-to-string.
Now a little quiz to see how well you understand what was done
above. A lisp function is passed an integer value str
which is the address of a sequence of characters in memory. How do you
access the third character in the string?
a. | (fslot-value-typed '(* :char) :c str 2) |
b. | (fslot-value-typed '(:array :char 10) :c str 2) |
c. | all of the above |
d. | none of the above |
The answer to the quiz: The correct answer is
b. Answer a can't be right since the
type (* :char)
is the same as (:struct (nil
(* :char)))
and thus says that str
points to
a structure in memory with one slot, that slot being a pointer to a
character string. But we know that str
itself points
to the string. Answer b is correct. The size of the
array we specified isn't important (as long as it is greater than the
index we are using to access the array).
As we've seen above, we take a C type (be it a struct or primitive
type) and create a Lisp equivalent type. Let X
be the
C type and Y
the Lisp type. When we pass
Y
to C, the C code gets not an X
object but an X* object since we always pass a
pointer to a foreign structure. Likewise when C returns an
X
structure, it doesn't usually return the
X
structure, it returns an X*
value. Lisp then sees that value as an instance of the
Y
foreign type. Thus going to C we add a
'*
' to the type since we pass by reference. Coming
back from C we remove a '*
' from the type since Lisp
always refers to types by pointers, thus using the
'*
' is superfluous.
Copyright (c) 1998-2016, Franz Inc. Oakland, CA., USA. All rights reserved.
This page was not revised from the 8.1 page.
Created 2010.1.21.
| Allegro CL version 8.2 Unrevised from 8.1 to 8.2. 8.1 version |