Referencing and using foreign memory addresses in OpenMCL

Overview.

Basics.

For a variety of technical reasons, it isn't generally possible to directly reference arbitrary absolute addresses (such as those returned by the C library function malloc(), for instance) in OpenMCL. In OpenMCL (and in MCL), such addresses need to be encapsulated in objects of type CCL:MACPTR; one can think of a MACPTR as being a specialized type of structure whose sole purpose is to provide a way of referring to an underlying "raw" address.

It's sometimes convenient to blur the distinction between a MACPTR and the address it represents; it's sometimes necessary to maintain that distiction. It's important to remember that a MACPTR is (generally) a first-class Lisp object in the same sense that a CONS cell is: it'll get GCed when it's no longer possible to reference it. The "lifetime" of a MACPTR doesn't generally have anything to do with the lifetime of the block of memory its address points to.

It might be tempting to ask "How does one obtain the address encapsulated by a MACPTR ?". The answer to that question is that one doesn't do that (and there's no way to do that): addresses aren't first-class objects, and there's no way to refer to one.

Two MACPTRs that encapsulate the same address are EQL to each other.

There are a small number of ways to directly create a MACPTR (and there's a fair amount of syntactic sugar built on top of of those primitives.) These primitives will be discussed in greater detail below, but they include:

All of these primitive MACPTR-creating operations are usually open-coded by the compiler; it has a fairly good notion of what low-level operations "produce" MACPTRs and which operations "consume" the addresses that the encapsulate, and will usually optimize out the introduction of intermediate MACPTRs in a simple expression.

One consequence of the use of MACPTR objects to encapsulate foreign addresses is that (naively) every reference to a foreign address causes a MACPTR to be allocated. Consider a code fragment like the following:

 (defun get-next-event ()
   "get the next event from a hypothetical window system"
   (loop
     (let* ((event (#_get_next_window_system_event))) ; via an FF-CALL
        (unless (null-event-p event)
          (handle-event event)))))
    

As this is written, each call to the (hypothetical) foreign function #_get_next_window_system_event will return a new MACPTR object. Ignoring for the sake of argument the question of whether this code fragment exhibits a good way to poll for external events (it doesn't), it's not hard to imagine that this loop could execute several millon times per second (producing several million MACPTRs per second.) Clearly, the "naive" approach is impractical in many cases.

Stack allocation of - and destructive operations on - MACPTRs.

If certain conditions held in the environment in which GET-NEXT-EVENT ran - namely, if it was guaranteed that neither NULL-EVENT-P nor HANDLE-EVENT cached or otherwise retained their arguments (the "event" pointer) - there'd be a few alternatives to the naive approach. One of those approaches would be to use the primitive function %SETF-MACPTR (described in greater detail below) to destructively modify a MACPTR (to change the value of the address it encapsulates.) The GET-NEXT-EVENT example could be re-written as:

(defun get-next-event ()
  (let* ((event (%int-to-ptr 0))) ; create a MACPTR with
                                  ; address 0
    (loop
      (%setf-macptr event (#_get_next_window_system_event)) ; re-use it
        (unless (null-event-p event)
          (handle-event event)))))
    

That version's a bit more realistic: it allocates a single MACPTR outside if the loop, then changes its address to point to the current address of the hypothetical event structure on each loop iteration. If there are a million loop iterations per call to GET-NEXT-EVENT, we're allocating a million times fewer MACPTRs per call; that sounds like a Good Thing.

An Even Better Thing would be to advise the compiler that the initial value (the null MACPTR) bound to the variable event has dynamic extent (that value won't be referenced once control leaves the extent of the binding of that variable.) Common Lisp allows us to make such an assertion via a DYNAMIC-EXTENT declaration; OpenMCL's compiler can recognize the "primitive MACPTR-creating operation" involved and can replace it with an equivalent operation that stack-allocates the MACPTR object. If we're not worried about the cost of allocating that MACPTR on every iteration (the cost is small and there's no hidden GC cost), we could move the binding back inside the loop:

(defun get-next-event ()
  (loop
    (let* ((event (%null-ptr))) ; (%NULL-PTR) is shorthand for (%INT-TO-PTR 0)
      (declare (dynamic-extent event))
      (%setf-macptr event (#_get_next_window_system_event))
      (unless (null-event-p event)
        (handle-event event)))))
    

The idiom of binding one or more variables to stack-allocated MACPTRs, then destructively modifying those MACPTRs before executing a body of code is common enough that OpenMCL provides a macro (WITH-MACPTRS) that handles all of the gory details. The following version of GET-NEXT-EVENT is semantically equivalent to the previous version, but hopefully a bit more concise:

(defun get-next-event ()
  (loop
    (with-macptrs ((event (#_get_next_window_system_event)))
      (unless (null-event-p event)
        (handle-event event)))))
    

Stack-allocated memory (and stack-allocated pointers to it.)

Fairly often, the blocks of foreign memory (obtained by malloc or something similar) have well-defined lifetimes (they can safely be freed at some point when it's known that they're no longer needed and it's known that they're no longer referenced.) A common idiom might be:

    ...
    (with-macptrs (p (#_allocate_foreign_memory size))
      (unwind-protect
        (use-foreign-memory p)
        (#_deallocate_foreign_memory p)))
    ...
    

That's not unreasonable code, but it's fairly expensive for a number of reasons: foreign functions calls are themselves fairly expensive (as is UNWIND-PROTECT), and most library routines for allocating and deallocating foreign memory (things like malloc and free can be fairly expensive in their own right.

In the idiomatic code above, both the MACPTR P and the block of memory that's being allocated and freed have dynamic extent and are therefore good candidates for stack allocation. OpenMCL provides the %STACK-BLOCK macro, which executes a body of code with one or more variables bound to stack-allocated MACPTRs which encapsulate the addresses of stack-allocated blocks of foreign memory. Using %STACK-BLOCK, the idiomatic code is:

    ...
    (%stack-block ((p size))
      (use-foreign-memory p))
    ...
    

which is a bit more efficient and a bit more concise than the version presented earlier.

%STACK-BLOCK is used as the basis for slightly higher-level things like RLET. (See the foreign type system documentation for more information about RLET.)

Caveats.

Reading from, writing to, allocating, and freeing foreign memory are all potentially dangerous operations; this is no less true when these operations are performed in OpenMCL than when they're done in C or some other lower-level language. In addition, destructive operations on Lisp objects be dangerous, as can stack allocation if it's abused (if DYNAMIC-EXTENT declarations are violated.) Correct use of the constructs and primitives described here is reliable and safe; slightly incorrect use of these constructs and primitives can crash OpenMCL.

Functional Reference

Unless otherwise noted, all of the symbols mentioned below are exported from the CCL package.

Scalar memory reference

Syntax
%get-signed-byte ptr &optional (offset 0)
%get-unsigned-byte ptr &optional (offset 0)
%get-signed-word ptr &optional (offset 0)
%get-unsigned-word ptr &optional (offset 0)
%get-signed-long ptr &optional (offset 0)
%get-unsigned-long ptr &optional (offset 0)
%%get-signed-longlong ptr &optional (offset 0)
%%get-unsigned-longlong ptr &optional (offset 0)
%get-ptr ptr &optional (offset 0)
%get-single-float ptr &optional (offset 0)
%get-double-float ptr &optional (offset 0)
Description References and returns the signed or unsigned 8-bit byte, signed or unsigned 16-bit word, signed or unsigned 32-bit long word, signed or unsigned 64-bit long long word, 32-bit address, 32-bit single-float, or 64-bit double-float at the effective byte address formed by adding offset to the address encapsulated by ptr.
Arguments
ptr
A MACPTR
offset
A fixnum

All of the memory reference primitives described above can be used with SETF.

%GET-BIT [Function]

Syntax
%get-bit ptr bit-offset
Description References and returns the bit-offsetth bit at the address encapsulated by ptr. (Bit 0 at a given address is the most significant bit of the byte at that address.) Can be used with SETF.
Arguments
ptr
A MACPTR
bit-offset
A fixnum

%GET-BITFIELD [Function]

Syntax
%get-bitfield ptr bit-offset width
Description References and returns an unsigned integer composed from the width bits found bit-offsetbits from the address encapsulated by ptr. (The least significant bit of the result is the value of (%get-bit ptr (1- (+ bit-offset width))) Can be used with SETF.
Arguments
ptr
A MACPTR
bit-offset
A fixnum
width
A positive fixnum

%INT-TO-PTR [Function]

Syntax
%int-to-ptr int
Description Creates and returns a MACPTR whose address matches int.
Arguments
int
An (unsigned-byte 32)

%inc-ptr [Function]

Syntax
%inc-ptr ptr &optional (delta 1)
Description Creates and returns a MACPTR whose address is the address of ptr plus delta. The idiom (%inc-ptr ptr 0) is sometimes used to copy a MACPTR, e.g., to create a new MACPTR encapsulating the same address as ptr.
Arguments
ptr
A MACPTR
delta
A fixnum

%ptr-to-int [Function]

Syntax
%ptr-to-int ptr
Description Returns the address encapsulated by ptr, as an (unsigned-byte 32).
Arguments
ptr
A MACPTR

%null-ptr [Macro]

Syntax
%null-ptr
Description Equivalent to (%ptr-to-int 0).

%null-ptr-p [Function]

Syntax
%null-ptr-p ptr
Description Returns T If ptr is a MACPTR encapsulating the address 0, NILif ptr encapsulates some other address.
Arguments
ptr
A MACPTR

%setf-macptr [Function]

Syntax
%setf-macptr dest-ptr src-ptr
Description Causes dest-ptr to encapsulate the same address that src-ptr does, then returns dest-ptr.
Arguments
dest-ptr
A MACPTR
src-ptr
A MACPTR

%incf-ptr [Macro]

Syntax
%incf-ptr ptr &optional (delta 1)
Description Destructively modifies ptr, by adding delta to the address it encapsulates. Returns ptr.
Arguments
ptr
A MACPTR
delta
A fixnum

with-macptrs [Macro]

Syntax
with-macptrs (var expr)* &body body
Description Executes body in an environment in which each var is bound to a stack-allocated macptr which encapsulates the foreign address yielded by the corresponding expr. Returns whatever value(s) body returns.
Arguments
var
A symbol (variable name)
expr
A MACPTR-valued expression

%stack-block [Macro]

Syntax
%stack-block (var expr)* &body body
Description Executes body in an environment in which each var is bound to a stack-allocated macptr which encapsulates the address of a stack-allocated region of size expr bytes. Returns whatever value(s) body returns.
Arguments
var
A symbol (variable name)
An expression which should evaluate to a non-negative fixnum
Note There's currently a rather small (~4K byte) limit on the size of any individual block that can be stack-allocated. Larger blocks are heap-allocated and are marked for eventual reclamation by the GC.

make-cstring [Function]

Syntax
make-cstring string
Description Allocates a block of memory (via malloc) of length (1+ (length string)). Copies the string to this block and appends a trailing NUL byte; returns a MACPTR to the block.
Arguments
string
A lisp string

with-cstrs [Macro]

Syntax
%stack-block (var string)* &body body
Description Executes body in an environment in which each var is bound to a stack-allocated macptr which encapsulates the address of a stack-allocated region of into which each string (and a trailing NUL byte) has been copied. Returns whatever value(s) body returns.
Arguments
var
A symbol (variable name)
An expression which should evaluate to a lisp string

%get-cstring [Function]

Syntax
%get-cstring ptr
Description Interprets ptr as a pointer to a (NUL -terminated) C string; returns an equivalent lisp string.
Arguments
ptr
A MACPTR

Last modified: Sat Jun 22 22:09:17 MDT 2002