In release 1.2 and later, Clozure CL supports memory-mapped files. On operating systems that support memory-mapped files (including Mac OS X, Linux, and FreeBSD), the operating system can arrange for a range of virtual memory addresses to refer to the contents of an open file. As long as the file remains open, programs can read values from the file by reading addresses in the mapped range.

Using memory-mapped files may in some cases be more efficient than reading the contents of a file into a data structure in memory.

Clozure CL provides the functions CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR to support memory-mapping. These functions return vectors whose contents are the contents of memory-mapped files. Reading an element of such a vector returns data from the corresponding position in the file.

Without memory-mapped files, a common idiom for reading the contents of files might be something like this:

(let* ((stream (open pathname :direction :input :element-type '(unsigned-byte 8)))
       (vector (make-array (file-size-to-vector-size stream)
                           :element-type '(unsigned-byte 8))))
  (read-sequence vector stream))
    

Using a memory-mapped files has a result that is the same in that, like the above example, it returns a vector whose contents are the same as the contents of the file. It differs in that the above example creates a new vector in memory and copies the file's contents into it; using a memory-mapped file instead arranges for the vector's elements to point to the file's contents on disk directly, without copying them into memory first.

The vectors returned by CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR are read-only; any attempt to change an element of a vector returned by these functions results in a memory-access error. Clozure CL does not currently support writing data to memory-mapped files.

Vectors created by CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR are required to respect Clozure CL's limit on the total size of an array. That means that you cannot use these functions to create a vector longer than ARRAY-TOTAL-SIZE-LIMIT, even if the filesystem supports file sizes that are larger. The value of ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) on 32-but platforms; and (EXPT 2 56) on 64-bit platforms.

CCL:MAP-FILE-TO-IVECTOR pathname element-type [Function]

The map-file-to-ivector function tries to open the file at pathname for reading. If successful, the function maps the file's contents to a range of virtual addresses. If successful, it returns a read-only vector whose element-type is given by element-type, and whose contents are the contents of the memory-mapped file.

The returned vector is a displaced-array whose element-type is (UPGRADED-ARRAY-ELEMENT-TYPE element-type). The target of the displaced array is a vector of type (SIMPLE-ARRAY element-type (*)) whose elements are the contents of the memory-mapped file.

Because of alignment issues, the mapped file's contents start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit platforms) into the vector. The displaced array returned by CCL:MAP-FILE-TO-IVECTOR hides this overhead, but it's usually more efficient to operate on the underlying simple 1-dimensional array. Given a displaced array (like the value returned by CCL:MAP-FILE-TO-IVECTOR), the function ARRAY-DISPLACEMENT returns the underlying array and the displacement index in elements.

Currently, Clozure CL supports only read operations on memory-mapped files. If you try to change the contents of an array returned by map-file-to-ivector, Clozure CL signals a memory error.

CCL:UNMAP-IVECTOR displaced-array [Function]

If the argument is a displaced-array returned by map-file-to-ivector, and if it has not yet been unmapped by this function, then unmap-ivector undoes the memory mapping, closes the mapped file, and changes the displaced-array so that its target is an empty vector (of length zero).

CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]

This function is a synonym for (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)) It is provided as a convenience for the common case of memory-mapping a file as a vector of bytes.

CCL:UNMAP-OCTET-VECTOR displaced-array [Function]

This function is a synonym for (CCL:UNMAP-IVECTOR)