Previous Section Next Section Table of Contents Glossary Index

Chapter 13. The Foreign-Function Interface

13.3. Referencing and Using Foreign Memory Addresses

13.3.1. Overview

13.3.1.1. Basics

For a variety of technical reasons, it isn't generally possible to directly reference arbitrary absolute addresses (such as those returned by the C library function malloc(), for instance) in CCL. In CCL (and in MCL), such addresses need to be encapsulated in objects of type CCL:MACPTR; one can think of a MACPTR as being a specialized type of structure whose sole purpose is to provide a way of referring to an underlying "raw" address.

It's sometimes convenient to blur the distinction between a MACPTR and the address it represents; it's sometimes necessary to maintain that distinction. It's important to remember that a MACPTR is (generally) a first-class Lisp object in the same sense that a CONS cell is: it'll get GCed when it's no longer possible to reference it. The "lifetime" of a MACPTR doesn't generally have anything to do with the lifetime of the block of memory its address points to.

It might be tempting to ask "How does one obtain the address encapsulated by a MACPTR ?". The answer to that question is that one doesn't do that (and there's no way to do that): addresses aren't first-class objects, and there's no way to refer to one.

Two MACPTRs that encapsulate the same address are EQL to each other.

There are a small number of ways to directly create a MACPTR (and there's a fair amount of syntactic sugar built on top of of those primitives.) These primitives will be discussed in greater detail below, but they include:

  • Creating a MACPTR with a specified address, usually via the function CCL:%INT-TO-PTR.

  • Referencing the return value of a foreign function call (see )that's specified to return an address.

  • Referencing a memory location that's specified to contain an address.

All of these primitive MACPTR-creating operations are usually open-coded by the compiler; it has a fairly good notion of what low-level operations "produce" MACPTRs and which operations "consume" the addresses that the encapsulate, and will usually optimize out the introduction of intermediate MACPTRs in a simple expression.

One consequence of the use of MACPTR objects to encapsulate foreign addresses is that (naively) every reference to a foreign address causes a MACPTR to be allocated.

Consider a code fragment like the following:

(defun get-next-event ()
  "get the next event from a hypothetical window system"
  (loop
     (let* ((event (#_get_next_window_system_event))) ; via an FF-CALL
       (unless (null-event-p event)
         (handle-event event)))))
        

As this is written, each call to the (hypothetical) foreign function #_get_next_window_system_event will return a new MACPTR object. Ignoring for the sake of argument the question of whether this code fragment exhibits a good way to poll for external events (it doesn't), it's not hard to imagine that this loop could execute several million times per second (producing several million MACPTRs per second.) Clearly, the "naive" approach is impractical in many cases.

13.3.1.2. Stack allocation of—and destructive operations on—MACPTRs.

If certain conditions held in the environment in which GET-NEXT-EVENT ran—namely, if it was guaranteed that neither NULL-EVENT-P nor HANDLE-EVENT cached or otherwise retained their arguments (the "event" pointer)—there'd be a few alternatives to the naive approach. One of those approaches would be to use the primitive function %SETF-MACPTR (described in greater detail below) to destructively modify a MACPTR (to change the value of the address it encapsulates.) The GET-NEXT-EVENT example could be re-written as:

(defun get-next-event ()
  (let* ((event (%int-to-ptr 0)))     ; create a MACPTR with address 0
    (loop
       (%setf-macptr event (#_get_next_window_system_event)) ; re-use it
       (unless (null-event-p event)
         (handle-event event)))))
        

That version's a bit more realistic: it allocates a single MACPTR outside if the loop, then changes its address to point to the current address of the hypothetical event structure on each loop iteration. If there are a million loop iterations per call to GET-NEXT-EVENT, we're allocating a million times fewer MACPTRs per call; that sounds like a Good Thing.

An Even Better Thing would be to advise the compiler that the initial value (the null MACPTR) bound to the variable event has dynamic extent (that value won't be referenced once control leaves the extent of the binding of that variable.) Common Lisp allows us to make such an assertion via a DYNAMIC-EXTENT declaration; CCL's compiler can recognize the "primitive MACPTR-creating operation" involved and can replace it with an equivalent operation that stack-allocates the MACPTR object. If we're not worried about the cost of allocating that MACPTR on every iteration (the cost is small and there's no hidden GC cost), we could move the binding back inside the loop:

(defun get-next-event ()
  (loop
     (let* ((event (%null-ptr))) ; (%NULL-PTR) is shorthand for (%INT-TO-PTR 0)
       (declare (dynamic-extent event))
       (%setf-macptr event (#_get_next_window_system_event))
       (unless (null-event-p event)
         (handle-event event)))))
        

The idiom of binding one or more variables to stack-allocated MACPTRs, then destructively modifying those MACPTRs before executing a body of code is common enough that CCL provides a macro (WITH-MACPTRS) that handles all of the gory details. The following version of GET-NEXT-EVENT is semantically equivalent to the previous version, but hopefully a bit more concise:

(defun get-next-event ()
  (loop
     (with-macptrs ((event (#_get_next_window_system_event)))
       (unless (null-event-p event)
         (handle-event event)))))
        

13.3.1.3. Stack-allocated memory (and stack-allocated pointers to it.)

Fairly often, the blocks of foreign memory (obtained by malloc or something similar) have well-defined lifetimes (they can safely be freed at some point when it's known that they're no longer needed and it's known that they're no longer referenced.) A common idiom might be:

(with-macptrs (p (#_allocate_foreign_memory size))
  (unwind-protect
       (use-foreign-memory p)
    (#_deallocate_foreign_memory p)))
        

That's not unreasonable code, but it's fairly expensive for a number of reasons: foreign functions calls are themselves fairly expensive (as is UNWIND-PROTECT), and most library routines for allocating and deallocating foreign memory (things like malloc and free) can be fairly expensive in their own right.

In the idiomatic code above, both the MACPTR P and the block of memory that's being allocated and freed have dynamic extent and are therefore good candidates for stack allocation. CCL provides the %STACK-BLOCK macro, which executes a body of code with one or more variables bound to stack-allocated MACPTRs which encapsulate the addresses of stack-allocated blocks of foreign memory. Using %STACK-BLOCK, the idiomatic code is:

(%stack-block ((p size))
              (use-foreign-memory p))
        

which is a bit more efficient and a bit more concise than the version presented earlier.

%STACK-BLOCK is used as the basis for slightly higher-level things like RLET. (See FIXTHIS for more information about RLET.)

13.3.1.4. Caveats

Reading from, writing to, allocating, and freeing foreign memory are all potentially dangerous operations; this is no less true when these operations are performed in CCL than when they're done in C or some other lower-level language. In addition, destructive operations on Lisp objects be dangerous, as can stack allocation if it's abused (if DYNAMIC-EXTENT declarations are violated.) Correct use of the constructs and primitives described here is reliable and safe; slightly incorrect use of these constructs and primitives can crash CCL.

13.3.2. Foreign-Memory-Addresses Dictionary

Unless otherwise noted, all of the symbols mentioned below are exported from the CCL package.

13.3.2.1. Scalar memory reference

Syntax

%get-signed-byte ptr &optional (offset 0)

%get-unsigned-byte ptr &optional (offset 0)

%get-signed-word ptr &optional (offset 0)

%get-unsigned-word ptr &optional (offset 0)

%get-signed-long ptr &optional (offset 0)

%get-unsigned-long ptr &optional (offset 0)

%%get-signed-longlong ptr &optional (offset 0)

%%get-unsigned-longlong ptr &optional (offset 0)

%get-ptr ptr &optional (offset 0)

%get-single-float ptr &optional (offset 0)

%get-double-float ptr &optional (offset 0)

Description

References and returns the signed or unsigned 8-bit byte, signed or unsigned 16-bit word, signed or unsigned 32-bit long word, signed or unsigned 64-bit long long word, 32-bit address, 32-bit single-float, or 64-bit double-float at the effective byte address formed by adding offset to the address encapsulated by ptr.

Arguments
ptr

A MACPTR

offset

A fixnum

All of the memory reference primitives described above can be

used with SETF.

13.3.2.2. %get-bit [Function]

Syntax

%get-bit ptr bit-offset

Description

References and returns the bit-offsetth bit at the address encapsulated by ptr. (Bit 0 at a given address is the most significant bit of the byte at that address.) Can be used with SETF.

Arguments

 

ptr

A MACPTR

bit-offset

A fixnum

13.3.2.3. %get-bitfield [Function]

Syntax

%get-bitfield ptr bit-offset width

Description

References and returns an unsigned integer composed from the width bits found bit-offset bits from the address encapsulated by ptr. (The least significant bit of the result is the value of (%get-bit ptr (1- (+ bit-offset width))). Can be used with SETF.

Arguments

 

ptr

A MACPTR

bit-offset

A fixnum

width

A positive fixnum

13.3.2.4. %int-to-ptr [Function]

Syntax

%int-to-ptr int

Description

Creates and returns a MACPTR whose address matches int.

Arguments

 

int

An (unsigned-byte 32)

13.3.2.5. %inc-ptr [Function]

Syntax

%inc-ptr ptr &optional (delta 1)

Description

Creates and returns a MACPTR whose address is the address of ptr plus delta. The idiom (%inc-ptr ptr 0) is sometimes used to copy a MACPTR, e.g., to create a new MACPTR encapsulating the same address as ptr.

Arguments

 

ptr

A MACPTR

delta

A fixnum

13.3.2.6. %ptr-to-int [Function]

Syntax

%ptr-to-int ptr

Description

Returns the address encapsulated by ptr, as an (unsigned-byte 32).

Arguments

 

ptr

A MACPTR

13.3.2.7. %null-ptr [Macro]

Syntax

%null-ptr

Description

Equivalent to (%int-to-ptr 0).

13.3.2.8. %null-ptr-p [Function]

Syntax

%null-ptr-p ptr

Description

Returns T If ptr is a MACPTR encapsulating the address 0, NIL if ptr encapsulates some other address.

Arguments

 

ptr

A MACPTR

13.3.2.9. %setf-macptr [Function]

Syntax

%setf-macptr dest-ptr src-ptr

Description

Causes dest-ptr to encapsulate the same address that src-ptr does, then returns dest-ptr.

Arguments

 

dest-ptr

A MACPTR

src-ptr

A MACPTR

13.3.2.10. %incf-ptr [Macro]

Syntax

%incf-ptr ptr &optional (delta 1)

Description

Destructively modifies ptr, by adding delta to the address it encapsulates. Returns ptr.

Arguments

 

ptr

A MACPTR

delta

A fixnum

13.3.2.11. with-macptrs [Macro]

Syntax

with-macptrs (var expr)* &body body

Description

Executes body in an environment in which each var is bound to a stack-allocated macptr which encapsulates the foreign address yielded by the corresponding expr. Returns whatever value(s) body returns.

Arguments

 

var

A symbol (variable name)

expr

A MACPTR-valued expression

13.3.2.12. %stack-block [Macro]

Syntax

%stack-block (var expr)* &body body

Description

Executes body in an environment in which each var is bound to a stack-allocated macptr which encapsulates the address of a stack-allocated region of size expr bytes. Returns whatever value(s) body returns.

Arguments

 

var

A symbol (variable name)

expr

An expression which should evaluate to a non-negative fixnum

13.3.2.13. make-cstring [Function]

Syntax

make-cstring string

Description

Allocates a block of memory (via malloc) of length (1+ (length string)). Copies the string to this block and appends a trailing NUL byte; returns a MACPTR to the block.

Arguments

 

string

A lisp string

13.3.2.14. with-cstrs [Macro]

Syntax

with-cstrs (var string)* &body body

Description

Executes body in an environment in which each var is bound to a stack-allocated macptr which encapsulates the %address of a stack-allocated region of into which each string (and a trailing NUL byte) has been copied. Returns whatever value(s) body returns.

Arguments

 

var

A symbol (variable name)

string

An expression which should evaluate to a lisp string

13.3.2.15. with-encoded-cstrs [Macro]

Syntax

with-encoded-cstrs ENCODING-NAME (varI stringI)* &body body

Description

Executes body in an environment in which each varI is bound to a macptr which encapsulates the %address of a stack-allocated region of into which each stringI (and a trailing NUL character) has been copied. Returns whatever value(s) body returns.

ENCODING-NAME is a keyword constant that names a character encoding. Each foreign string is encoded in the named encoding. Each foreign string has dynamic extent.

WITH-ENCODED-CSTRS does not automatically prepend byte-order marks to its output; the size of the terminating #\NUL character depends on the number of octets per code unit in the encoding.

The expression

(ccl:with-cstrs ((x "x")) (#_puts x))

is functionally equivalent to

(ccl:with-encoded-cstrs :iso-8859-1 ((x "x")) (#_puts x))
Arguments

 

varI

A symbol (variable name)

stringI

An expression which should evaluate to a lisp string

13.3.2.16. %get-cstring [Function]

Syntax

%get-cstring ptr

Description

Interprets ptr as a pointer to a (NUL -terminated) C string; returns an equivalent lisp string.

Arguments

ptr

A MACPTR

13.3.2.17. %str-from-ptr [Function]

Syntax

%str-from-ptr ptr length

Description

Returns a lisp string of length length, whose contents are initialized from the bytes at ptr.

Arguments
ptr

A MACPTR

length

a non-negative fixnum


Previous Section Next Section Table of Contents Glossary Index