Clozure CL Documentation

Clozure CL is a fast, mature, open source Common Lisp implementation that runs on Linux, Mac OS X and BSD on either Intel x86-64 or PPC. Clozure CL was forked from Macintosh Common Lisp (MCL) in 1998 and the development has been entirely separate since. Ports to IA32 and Windows are under development.

When it was forked from MCL in 1998, the new Lisp was named OpenMCL. Recently, Clozure renamed its Lisp to Clozure CL, partly because its ancestor MCL has lately been released as open source. Clozure thought it might be confusing for users if there were two independent open-source projects with such similar names. The new name also reflects Clozure CL's current status as the flagship product of Clozure Associates.

Furthermore, the new name refers to Clozure CL's ancestry: in its early years, MCL was known as Coral Common Lisp, or "CCL". For years the package that contains most of Clozure CL's implementation-specific symbols has been named "CCL", an acronym that once stood for the name of the Lisp product. It seems fitting that "CCL" once again stands for the name of the product.

Some commands and source files may still refer to "OpenMCL" instead of Clozure CL.

Clozure CL compiles to native code and supports multithreading using native OS threads. It includes a foreign-function interface, and supports both Lisp code that calls external code, and external code that calls Lisp code. Clozure CL can create standalone executables on all supported platforms.

On Mac OS X, Clozure CL supports building GUI applications that use OS X's native Cocoa frameworks, and the OS X distributions include an IDE written with Cocoa, and distributed with complete sources.

On all supported platforms, Clozure CL can run as a command-line process, or as an inferior Emacs process using either SLIME or ILISP.

Features of Clozure CL include

Although it's an open-source project, available free of charge under a liberal license, Clozure CL is also a fully-supported product of Clozure Associates. Clozure continues to extend, improve, and develop Clozure CL in response to customer and user needs, and offers full support and development services for Clozure CL.

Version 1.3 is the latest stable release of Clozure CL as of April 2009.

Version 1.3 is available for seven platform configurations:

A 64-bit version of Clozure CL requires a 64-bit processor running a 64-bit OS variant.

Additional platform-specific information is given in the following subsections.

Older versions are still available for downloading as tarballs. Version 1.0 was a stable version released in late 2005. Version 1.1 was under active development until late 2007. A final 1.1 release was never made. It was distributed as a series of development "snapshots" and CVS updates. 1.1 snapshots introduced support for x86-64 platforms, internal use of Unicode, and many other features, but were moving targets. Version 1.2 was a stable version released in April 2008.

There two main ways to obtain Clozure CL. For Mac OS X, there are disk images that can be used to install Clozure CL in the usual Macintosh way. For other OSes, Subversion is the best way to obtain Clozure CL. Mac OS X users can also use Subversion if they prefer. Tarballs are available for those who prefer them, but if you have Subversion installed, it is simpler and more flexible to use Subversion than tarballs.

There are three popular ways to use Clozure CL: as a stand-alone double-clickable application (Mac OS X only), as a command-line application, or with EMACS and SLIME. The following sections describe these options.

          
svn co http://svn.clozure.com/publicsvn/openmcl/trunk/darwinx86/ccl
        

          
svn co http://svn.clozure.com/publicsvn/openmcl/release/1.3/darwinx86/ccl
        

          
joe:ccl> ./dx86cl64
Welcome to Clozure Common Lisp Version 1.2  (DarwinX8664)!
? (rebuild-ccl :full t)

<lots of compilation output>

  ? (quit)
  joe:ccl>
        

          
shell> svn
        

          
Type 'svn help' for usage
        

          
-bash: svn: command not found
        

Sometimes it's convenient to use Clozure CL from a Unix shell command line. This is especially true when using Clozure CL as a way to run Common Lisp utilities.

> ccl [args ...]
Welcome to Clozure CL Version 1.2 (DarwinPPC32)!
?
      

? (probe-file "ccl:level-1;level-1.lisp")  ;returns the physical pathname of the file
#P"/Users/alms/my_lisp_stuff/ccl/level-1/level-1.lisp"
      

shell>ccl args
	    

By default Clozure CL tries to load the file "home:ccl-init.lisp" or the compiled "home:ccl-init.fasl" upon starting up. Clozure CL does this by executing (load "home:ccl-init"). If it's unable to load the file (for example because the file doesn't exist), Clozure CL doesn't signal an error or warning, it just completes its startup normally.

On Unix systems, if "ccl-init.lisp" is not present, Clozure CL will look for ".ccl-init.lisp" (post 1.2 versions only).

The "home:" prefix to the filename is a Common Lisp logical host, which Clozure CL initializes to refer to your home directory. Clozure CL therefore looks for either of the files ~/ccl-init.lisp or ~/ccl-init.fasl.

Because the init file is loaded the same way as normal Lisp code is, you can put anything you want in it. For example, you can change the working directory, and load packages that you use frequently.

To suppress the loading of this init-file, invoke Clozure CL with the --no-init option.

When using Clozure CL from the command line, the following options may be used to modify its behavior. The exact set of Clozure CL command-line arguments may vary per platform and slowly changes over time. The current set of command line options may be retrieved by using the --help option.

The --load and --eval options can each be provided multiple times. They're executed in the order specified on the command line, after the init file (if there is one) is loaded and before the toplevel read-eval-print loop is entered.

A very common way to use Clozure CL is to run it within the GNU Emacs editor, using a Lisp interface called SLIME ("Superior Lisp Interaction Mode for Emacs"). SLIME is an Emacs package designed to provide good support within Emacs for any of several Common Lisp implementations; one of the supported implementations is Clozure CL. This page describes how you can download SLIME and set it up to work with your Clozure CL installation.

Why use SLIME? With SLIME, you can do the following things from within an Emacs editing session:

For complete information about SLIME, see the SLIME home page. The SLIME home page provides up-to-date downloads, plus documentation, tutorials, and instructional screencasts.

A number (ok, a small number), of example programs are distributed in the "ccl:examples;" directory of the source distribution. See the README-OPENMCL-EXAMPLES text file in that directory for information about prerequisites and usage.

Some of the example programs are derived from C examples in textbooks, etc.; in those cases, the original author and work are cited in the source code.

Unless the original author or contributor claims other rights, you're free to incorporate any of this example code or derivative thereof in any of your own works without restriction. In doing so, you agree that the code was provided "as is", and that no other party is legally or otherwise responsible for any consequences of your decision to use it.

If you've developed Clozure CL examples that you'd like to see added to the distribution, please send mail to the Clozure CL mailing lists. Any such contributions would be welcome and appreciated (as would bug fixes and improvements to the existing examples.)

The following terms are used in subsequent sections; it may be helpful to refer to these definitions.

fasl files are the object files produced by compile-file. fasl files store the machine code associated with function definitions and the external representation of other lisp objects in a compact, machine-readable form. fasl is short for “FASt Loading”. Clozure CL uses different pathname types (extensions) to name fasl files on different platforms; see Table 3.1, “Platform-specific filename conventions”

The Lisp kernel is a C program with a fair amount of platform-specific assembly language code. Its basic job is to map a lisp heap image into memory, transfer control to some compiled lisp code that the image contains, handle any exceptions that occur during the execution of that lisp code, and provide various other forms of runtime support for that code. Clozure CL uses different filenames to name the lisp kernel files on different platforms; see Table 3.1, “Platform-specific filename conventions”.

A heap image is a file that can be quickly mapped into a process's address space. Conceptually, it's not too different from an executable file or shared library in the OS's native format (ELF or Mach-O/dyld format); for historical reasons, Clozure CL's own heap images are in their own (fairly simple) format. The term full heap image refers to a heap image file that contains all of the code and data that comprise Clozure CL. Clozure CL uses different filenames to name the standard full heap image files on different platforms; see Table 3.1, “Platform-specific filename conventions”.

A bootstrapping image is a minimal heap image used in the process of building Clozure CL itself. The bootstrapping image contains just enough code to load the rest of Clozure CL from fasl files. It may help to think of the bootstrapping image as the egg and the full heap image as the chicken. Clozure CL uses different filenames to name the standard bootstrapping image files on different platforms; see Table 3.1, “Platform-specific filename conventions” .

Each supported platform (and possibly a few as-yet-unsupported ones) has a uniquely named subdirectory of ccl/lisp-kernel/; each such contains a Makefile and may contain some auxiliary files (linker scripts, etc.) that are used to build the lisp kernel on a particular platform.The platform-specific name of the kernel build directory is described in Table 3.1, “Platform-specific filename conventions”.

At a given time, there are generally two versions of Clozure CL that you might want to use (and therefore might want to build from source):

All versions are available for download from svn.clozure.com via the Subversion source control system.

For example, to get a released version (1.3 in this example), use a command like:

	svn co http://svn.clozure.com/publicsvn/openmcl/release/1.3/xxx/ccl
      

To get the trunk version, use:

	svn co http://svn.clozure.com/publicsvn/openmcl/trunk/xxx/ccl
      

Change the "xxx" to one of the following names: darwinx86, linuxx86, freebsdx86, solarisx86, window, linuxppc, or darwinppc.

In the case of released versions, there may also be tar archives available. See the Clozure CL Trac for details.

Subversion client programs are pre-installed on Mac OS X 10.5 and later and are typically either pre-installed or readily available on Linux and FreeBSD platforms. The Subversion web page contains links to subversion client programs for many platforms; users of Mac OS X 10.4 can also install Subversion clients via Fink or MacPorts.

Given that you now have everything you need, do the following in a running Clozure CL to bring your Lisp system completely up to date.

? (ccl:rebuild-ccl :full t)
    

That call to the function rebuild-ccl performs the following steps:

If all goes well, it'll all happen without user intervention and with some simple progress messages. If anything goes wrong during execution of either of the external processes, the process output is displayed as part of a lisp error message.

rebuild-ccl is essentially just a short cut for running all the individual steps involved in rebuilding the system. You can also execute these steps individually, as described below.

The Lisp kernel is the executable that you run to use Lisp. It doesn't actually contain the entire Lisp implementation; rather, it loads a heap image which contains the specifics—the "library", as it might be called if this was a C program. The kernel also provides runtime support to the heap image, such as garbage collection, memory allocation, exception handling, and the OS interface.

The Lisp kernel file has different names on different platforms. See Table 3.1, “Platform-specific filename conventions”. On all platforms the lisp kernel sources reside in ccl/lisp-kernel.

This section gives directions on how to rebuild the Lisp kernel from its source code. Most Clozure CL users will rarely have to do this. You probably will only need to do it if you are attempting to port Clozure CL to a new architecture or extend or enhance its kernel in some way. As mentioned above, this step happens automatically when you do

? (rebuild-ccl :full t)
      

shell> cd ccl/lisp-kernel/PLATFORM
shell> make
	    

The initial heap image is loaded by the Lisp kernel, and provides most of the language implementation The heap image captures the entire state of a running Lisp (except for external resources, such as open files and TCP sockets). After it is loaded, the contents of the new Lisp process's memory are exactly the same as those of the old Lisp process when the image was created.

The heap image is how we get around the fact that we can't run Lisp code until we have a working Lisp implementation, and we can't make our Lisp implementation work until we can run Lisp code. Since the heap image already contains a fully-working implementation, all we need to do is load it into memory and start using it.

If you're building a new version of Clozure CL, you need to build a new heap image.

(You might also wish to build a heap image if you have a large program that is very complicated or time-consuming to load, so that you will be able to load it once, save an image, and thenceforth never have to load it again. At any time, a heap image capturing the entire memory state of a running Lisp can be created by calling the function ccl:save-application.)

shell> ccl
Welcome to Clozure CL .... !
?
	  

? (ccl:xload-level-0)
	  

? (ccl:xload-level-0 :force)
      

? (ccl:compile-ccl)
	  

$ cd ccl                        # wherever your ccl directory is
$ ./KERNEL BOOT_IMAGE
	  

? (ccl:save-application "image_name") ; Overwiting the existing heap image
	  

The Common Lisp standard allows considerable latitude in the details of an implementation, and each particular Common Lisp system has some idiosyncrasies. This chapter describes ordinary user-level features of Clozure CL, including features that may be part of the Common Lisp standard, but which may have quirks or details in the Clozure CL implementation that are not described by the standard. It also describes extensions to the standard; that is, features of Clozure CL that are not part of the Common Lisp standard at all.

Clozure CL's tracing facility is invoked by an extended version of the Common Lisp trace macro. Extensions allow tracing of methods, as well as finer control over tracing actions.

TRACE {keyword global-value}* {spec | (spec {keyword local-value}*)}* [Macro]

The trace macro encapsulates the functions named by specs, causing trace actions to take place on entry and exit from each function. The default actions print a message on function entry and exit. Keyword/value options can be used to specify changes in the default behavior.

Invoking (trace) without arguments returns a list of functions being traced.

A spec is either a symbol that is the name of a function, or an expression of the form (setf symbol), or a specific method of a generic function in the form (:method gf-name {qualifier}* ({specializer}*)), where a specializer can be the name of a class or an EQL specializer.

A spec can also be a string naming a package, or equivalently a list (:package package-name), in order to request that all functions in the package to be traced.

By default, whenever a traced function is entered or exited, a short message is printed on *trace-output* showing the arguments on entry and values on exit. Options specified as key/value pairs can be used to modify this behavior. Options preceding the function specs apply to all the functions being traced. Options specified along with a spec apply to that spec only and override any global options. The following options are supported:

CCL:*TRACE-LEVEL* [Variable]

Variable bound to the current nesting level during execution of before and after trace actions. The default printing actions use it to determine the amount of indentation.

CCL:*TRACE-MAX-INDENT* [Variable]

The default before and after print actions will not indent by more than the value of ccl:*trace-max-indent* regardless of the current trace level.

CCL:TRACE-FUNCTION spec &key {keyword value}* [Function]

This is a functional version of the TRACE macro. spec and keywords are as for TRACE, except that all arguments are evaluated.

CCL:*TRACE-PRINT-LEVEL* [Variable]

The default print actions bind CL:*PRINT-LEVEL* to this value while printing. Note that this rebinding is only in effect during the default entry and exit messages. It does not apply to printing of :print-before/:print-after forms or any explicit printing done by user code.

CCL:*TRACE-PRINT-LENGTH* [Variable]

The default print actions bind CL:*PRINT-LENGTH* to this value while printing. Note that this rebinding is only in effect during the default entry and exit messages. It does not apply to printing of :print-before/:print-after forms or any explicit printing done by user code.

CCL:*TRACE-BAR-FREQUENCY* [Variable]

By default, this is nil. If non-nil it should be a integer, and the default entry and exit messages will print a | instead of space every this many levels of indentation.

All characters and strings in Clozure CL fully support Unicode by using UTF-32. There is only one CHARACTER type and one STRING type in Clozure CL. There has been a lot of discussion about this decision which can be found by searching the openmcl-devel archives at http://clozure.com/pipermail/openmcl-devel/. Suffice it to say that we decided that the simplicity and speed advantages of only supporting UTF-32 outweigh the space disadvantage.

In release 1.2 and later, Clozure CL supports memory-mapped files. On operating systems that support memory-mapped files (including Mac OS X, Linux, and FreeBSD), the operating system can arrange for a range of virtual memory addresses to refer to the contents of an open file. As long as the file remains open, programs can read values from the file by reading addresses in the mapped range.

Using memory-mapped files may in some cases be more efficient than reading the contents of a file into a data structure in memory.

Clozure CL provides the functions CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR to support memory-mapping. These functions return vectors whose contents are the contents of memory-mapped files. Reading an element of such a vector returns data from the corresponding position in the file.

Without memory-mapped files, a common idiom for reading the contents of files might be something like this:

(let* ((stream (open pathname :direction :input :element-type '(unsigned-byte 8)))
       (vector (make-array (file-size-to-vector-size stream)
                           :element-type '(unsigned-byte 8))))
  (read-sequence vector stream))
    

Using a memory-mapped files has a result that is the same in that, like the above example, it returns a vector whose contents are the same as the contents of the file. It differs in that the above example creates a new vector in memory and copies the file's contents into it; using a memory-mapped file instead arranges for the vector's elements to point to the file's contents on disk directly, without copying them into memory first.

The vectors returned by CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR are read-only; any attempt to change an element of a vector returned by these functions results in a memory-access error. Clozure CL does not currently support writing data to memory-mapped files.

Vectors created by CCL:MAP-FILE-TO-IVECTOR and CCL:MAP-FILE-TO-OCTET-VECTOR are required to respect Clozure CL's limit on the total size of an array. That means that you cannot use these functions to create a vector longer than ARRAY-TOTAL-SIZE-LIMIT, even if the filesystem supports file sizes that are larger. The value of ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) on 32-but platforms; and (EXPT 2 56) on 64-bit platforms.

CCL:MAP-FILE-TO-IVECTOR pathname element-type [Function]

The map-file-to-ivector function tries to open the file at pathname for reading. If successful, the function maps the file's contents to a range of virtual addresses. If successful, it returns a read-only vector whose element-type is given by element-type, and whose contents are the contents of the memory-mapped file.

The returned vector is a displaced-array whose element-type is (UPGRADED-ARRAY-ELEMENT-TYPE element-type). The target of the displaced array is a vector of type (SIMPLE-ARRAY element-type (*)) whose elements are the contents of the memory-mapped file.

Because of alignment issues, the mapped file's contents start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit platforms) into the vector. The displaced array returned by CCL:MAP-FILE-TO-IVECTOR hides this overhead, but it's usually more efficient to operate on the underlying simple 1-dimensional array. Given a displaced array (like the value returned by CCL:MAP-FILE-TO-IVECTOR), the function ARRAY-DISPLACEMENT returns the underlying array and the displacement index in elements.

Currently, Clozure CL supports only read operations on memory-mapped files. If you try to change the contents of an array returned by map-file-to-ivector, Clozure CL signals a memory error.

CCL:UNMAP-IVECTOR displaced-array [Function]

If the argument is a displaced-array returned by map-file-to-ivector, and if it has not yet been unmapped by this function, then unmap-ivector undoes the memory mapping, closes the mapped file, and changes the displaced-array so that its target is an empty vector (of length zero).

CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]

This function is a synonym for (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)) It is provided as a convenience for the common case of memory-mapping a file as a vector of bytes.

CCL:UNMAP-OCTET-VECTOR displaced-array [Function]

This function is a synonym for (CCL:UNMAP-IVECTOR)

Clozure CL supports the definition of static variables, whose values are the same across threads, and which may not be dynamically bound. The value of a static variable is thus the same across all threads; changing the value in one thread changes it for all threads.

Attempting to dynamically rebind a static variable (for instance, by using LET, or using the variable name as a parameter in a LAMBDA form) signals an error. Static variables are shared global resources; a dynamic binding is private to a single thread.

Static variables therefore provide a simple way to share mutable state across threads. They also provide a simple way to introduce race conditions and obscure bugs into your code, since every thread reads and writes the same instance of a given static variable. You must take care, therefore, in how you change the values of static variables, and use normal multithreaded programming techniques, such as locks or semaphores, to protect against race conditions.

In Clozure CL, access to a static variable is usually faster than access to a special variable that has not been declared static.

DEFSTATIC var value &key doc-string [Macro]

Proclaims the variable special, assigns the variable the supplied value, and assigns the doc-string to the variable's VARIABLE documentation. Marks the variable static, preventing any attempt to dynamically rebind it. Any attempt to dynamically rebind var signals an error.

Clozure CL provides the function CCL:SAVE-APPLICATION, which creates a file containing an archived Lisp memory image.

Clozure CL consists of a small executable called the Lisp kernel, which implements the very lowest level features of the Lisp system, and an image, which contains the in-memory representation of most of the Lisp system, including functions, data structures, variables, and so on. When you start Clozure CL, you are launching the kernel, which then locates and reads an image file, restoring the archived image in memory. Once the image is fully restored, the Lisp system is running.

Using CCL:SAVE-APPLICATION, you can create a file that contains a modified image, one that includes any changes you've made to the running Lisp system. If you later pass your image file to the Clozure CL kernel as a command-line parameter, it then loads your image file instead of its default one, and Clozure CL starts up with your modifications.

If this scenario seems to you like a convenient way to create an application, that's just as intended. You can create an application by modifying the running Lisp until it does what you want, then use CCL:SAVE-APPLICATION to preserve your changes and later load them for use.

In fact, you can go further than that. You can replace Clozure CL's toplevel function with your own, and then, when the image is loaded, the Lisp system immediately performs your tasks rather than the default tasks that make it a Lisp development system. If you save an image in which you have done this, the resulting Lisp system is your tool rather than a Lisp development system.

You can go a step further still. You can tell CCL:SAVE-APPLICATION to prepend the Lisp kernel to the image file. Doing this makes the resulting image into a self-contained executable binary. When you run the resulting file, the Lisp kernel immediately loads the attached image file and runs your saved system. The Lisp system that starts up can have any behavior you choose. It can be a Lisp development system, but with your customizations; or it can immediately perform some task of your design, making it a specialized tool rather than a general development system.

In other words, you can develop any application you like by interactively modifying Clozure CL until it does what you want, then using CCL:SAVE-APPLICATION to preserve your changes in an executable image.

On Mac OS X, the application builder uses CCL:SAVE-APPLICATION to create the executable portion of the application bundle. Double-clicking the application bundle runs the executable image created by CCL:SAVE-APPLICATION.

Also on Mac OS X, Clozure CL supports an object type called MACPTR, which is the type of pointers into the foreign (Mac OS) heap. Examples of commonly-user MACPTR objects are Cocoa windows and other dynamically-allocated Mac OS system objects.

Because a MACPTR object is a pointer into a foreign heap that exists for the lifetime of the running Lisp process, and because a saved image is used by loading it into a brand new Lisp process, saved MACPTR objects cannot be relied on to point to the same things when reconstituted from a saved image. In fact, a restored MACPTR object might point to anything at all—for example an arbitrary location in the middle of a block of code, or a completely nonexistent virtual address.

For that reason, CCL:SAVE-APPLICATION converts all MACPTR objects to DEAD-MACPTR objects when writing them to an image file. A DEAD-MACPTR is functionally identical to a MACPTR, except that code that operates on MACPTR objects distinguishes them from DEAD-MACPTR objects and can handle them appropriately—signaling errors, for example.

As of Clozure CL 1.2, there is one exception to the conversion of MACPTR to DEAD-MACPTR objects: a MACPTR object that points to the address 0 is not converted, because address 0 can always be relied upon to refer to the same thing.

As of Clozure CL 1.2, the constant CCL:+NULL-PTR+ refers to a MACPTR object that points to address 0.

On all supported platforms, you can use CCL:SAVE-APPLICATION to create a command-line tool that runs the same way any command-line program does. Alternatively, if you choose not to prepend the kernel, you can save an image and then later run it by passing it as a command-line parameter to the opencml or opencml64 script.

SAVE-APPLICATION filename &key toplevel-function init-file error-handler application-class clear-clos-caches (purify t) impurify (mode #o644) prepend-kernel [Function]

Multiple fasl files can be concatenated into a single file.

In Clozure CL, the Common Lisp types short-float and single-float are implemented as IEEE single precision values; double-float and long-float are IEEE double precision values. On 64-bit platforms, single-floats are immediate values (like fixnums and characters).

Floating-point exceptions are generally enabled and detected. By default, threads start up with overflow, division-by-zero, and invalid enabled, and the rounding mode is set to nearest. The functions SET-FPU-MODE and GET-FPU-MODE provide user control over floating-point behavior.

As of release 1.4, Clozure CL provides a way for lisp objects to be watched so that a condition will be signaled when a thread attempts to write to the watched object. For a certain class of bugs (someone is changing this value, but I don't know who), this can be extremely helpful.

? (defvar *s* "xxxxx")
*S*
? (watch *s*)
"xxxxx"
? (setf (char *s* 3) #\o)
> Error: Write to watched uvector "xxxxx" at index 3
>        Faulting instruction: (movl (% eax) (@ -5 (% r15) (% rcx)))
> While executing: SET-CHAR, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.

    

? (defclass foo ()
    (slot-a slot-b slot-c))
#<STANDARD-CLASS FOO>
? (defvar *a-foo* (make-instance 'foo))
*A-FOO*
? (watch *a-foo*)
#<SLOT-VECTOR #xDB00D>
;;; Note that WATCH has watched the internal slot-vector object
? (setf (slot-value *a-foo* 'slot-a) 'foo)
> Error: Write to watched uvector #<SLOT-VECTOR #xDB00D> at index 1
>        Faulting instruction: (movq (% rsi) (@ -5 (% r8) (% rdi)))
> While executing: %MAYBE-STD-SETF-SLOT-VALUE-USING-CLASS, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.

    

1 > (uvref (write-to-watched-object-object *break-condition*) 0)
#<FOO #x30004113502D>
1 > (describe *)
#<FOO #x30004113502D>
Class: #<STANDARD-CLASS FOO>
Wrapper: #<CLASS-WRAPPER FOO #x300041135EBD>
Instance slots
SLOT-A: #<Unbound>
SLOT-B: #<Unbound>
SLOT-C: #<Unbound>
1 >
 
    

?  (defvar *f* (make-array '(2 3) :element-type 'double-float))
*F*
? (watch *f*)
#(0.0D0 0.0D0 0.0D0 0.0D0 0.0D0 0.0D0)
;;; Note that the above vector is the underlying data-vector for the array
? (setf (aref *f* 1 2) pi)
> Error: Write to watched uvector #<VECTOR 6 type DOUBLE-FLOAT, simple> at index 5
>        Faulting instruction: (movq (% rax) (@ -5 (% r8) (% rdi)))
> While executing: ASET, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
1 > 
  

? (defvar *h* (make-hash-table))
*H*
? (setf (gethash 'noise *h*) 'feep)
FEEP
? (watch *h*)
#<HASH-TABLE-VECTOR #xDD00D>
;;; underlying hash-table-vector
? (setf (gethash 'noise *h*) 'ding)
> Error: Write to watched uvector #<HASH-TABLE-VECTOR #xDD00D> at index 35
>        Faulting instruction: (lock)
>          (cmpxchgq (% rsi) (@ (% r8) (% rdx)))
> While executing: %STORE-NODE-CONDITIONAL, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
;;; see what value is being replaced...
1 > (uvref (write-to-watched-object-object *break-condition*) 35)
FEEP
;;; backtrace shows useful context
1 > :b
*(1A109F8) : 0 (%STORE-NODE-CONDITIONAL ???) NIL
 (1A10A50) : 1 (LOCK-FREE-PUTHASH NOISE #<HASH-TABLE :TEST EQL size 1/60 #x30004117D47D> DING) 653
 (1A10AC8) : 2 (CALL-CHECK-REGS PUTHASH NOISE #<HASH-TABLE :TEST EQL size 1/60 #x30004117D47D> DING) 229
 (1A10B00) : 3 (TOPLEVEL-EVAL (SETF (GETHASH # *H*) 'DING) NIL) 709
 ...
  

? (defun watch-list (list)
    (maplist #'watch list))
WATCH-LIST
? (defvar *l* (list 1 2 3))
*L*
? (watch-list *l*)
((1 2 3) (2 3) (3))
? (setf (nth 2 *l*) 'foo)
> Error: Write to the CAR of watched cons cell (3)
>        Faulting instruction: (movq (% rsi) (@ 5 (% rdi)))
> While executing: %SETNTH, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
  

Clozure CL ships with the complete source code for an integrated development environment written using Cocoa on Mac OS X. This chapter describes how to build and use that environment, referred to hereafter simply as "the IDE".

The IDE provides a programmable text editor, listener windows, an inspector for Lisp data structures, and a means of easily building a Cocoa application in Lisp. In addition, its source code provides an example of a fairly complex Cocoa application written in Lisp.

The current version of the IDE has seen the addition of numerous features and many bugfixes. Although it's by no means a finished product, we hope it will prove more useful than previous versions, and we plan additional work on the IDE for future releases.

Building the Clozure CL IDE is now a very simple process.

For example, assuming that the Clozure CL distribution is installed in "/usr/local/ccl", the following sequence of shell interactions builds the IDE:

oshirion:ccl mikel$ ccl64
Welcome to Clozure Common Lisp Version 1.2-r9198M-trunk  (DarwinX8664)!
? (require :cocoa-application)
;Loading #P"ccl:cocoa-ide;fasls;cocoa-utils.dx64fsl.newest"...
;Loading #P"ccl:cocoa-ide;fasls;cocoa-defaults.dx64fsl.newest"...

[...many lines of "Compiling" and "Loading" omitted...]

Saving application to /usr/local/ccl/Clozure CL.app/

oshirion:ccl mikel$ 

    

Clozure CL compiles and loads the various subsystems that make up the IDE, then constructs a Cocoa application bundle named "Clozure CL.app" and saves the Lisp image into it. Normally Clozure CL creates the application bundle in the root directory of the Clozure CL distribution.

After it has been built, you can run the "Clozure CL.app" application normally, by double-clicking its icon. When launched, the IDE initially displays a single listener window that you can use to interact with Lisp. You can type Lisp expressions for evaluation at the prompt in the listener window. You can also use Hemlock editing commands to edit the text of expressions in the listener window.

Clozure CL builds the IDE from sources in the "objc-bridge" and "cocoa-ide" directories in the Clozure CL distribution. The IDE as a whole is a relatively complicated application, and is probably not the best place to look when you are first trying to understand how to build Cocoa applications. For that, you might benefit more from the examples in the "examples/cocoa/" directory. Once you are familiar with those examples, though, and have some experience building your own application features using Cocoa and the Objective-C bridge, you might browse through the IDE sources to see how it implements its features.

The search path for Clozure CL's REQUIRE feature includes the "objc-bridge" and "cocoa-ide" directories. You can load features defined in these directories by using REQUIRE. For example, if you want to use the Cocoa features of Clozure CL from a terminal session (or from an Emacs session using SLIME or ILISP), you can evaluate (require :cocoa).

One important feature of the IDE currently has no Cocoa user interface: the application builder. The application builder constructs a Cocoa application bundle that runs a Lisp image when double-clicked. You can use the application builder to create Cocoa applications in Lisp. These applications are exactly like Cocoa applications created with XCode and Objective-C, except that they are written in Lisp.

To make the application builder available, evaluate the expression (require :build-application). Clozure CL loads the required subsystems, if necessary.

BUILD-APPLICATION &key (name "MyApplication") (type-string "APPL") (creator-string "OMCL") (directory (current-directory)) (copy-ide-resources t) (info-plist NIL) (nibfiles NIL) (main-nib-name NIL) (application-class 'GUI::COCOA-APPLICATION) (toplevel-function NIL) [Function]

The build-application function constructs an application bundle, populates it with the files needed to satisfy Mac OS X that the bundle is a launchable application, and saves an executable Lisp image to the proper subdirectory of the bundle. Assuming that the saved Lisp image contains correct code, a user can subsequently launch the resulting Cocoa application by double-clicking its icon in the Finder, and the saved Lisp environment runs.

The keyword arguments control various aspects of application bundle as BUILD-APPLICATION builds it.

BUILD-APPLICATION creates a folder named "name.app" in the directory directory. Inside that folder, it creates the "Contents" folder that Mac OS X application bundles are expected to contain, and populates it with the "MacOS" and "Resources" folders, and the "Info.plist" and "PkgInfo" files that must be present in a working application bundle. It takes the contents of the "Info.plist" and "PkgInfo" files from the parameters to BUILD-APPLICATION. If copy-ide-resources is true then it copies the contents of the "Resources" folder from the "Resources" folder of the running IDE.

The work needed to produce a running Cocoa application is very minimal. In fact, if you supply BUILD-APPLICATION with a valid nibfile and pathnames, it builds a running Cocoa application that displays your UI. It doesn't need you to write any code at all to do this. Of course, the resulting application doesn't do anything apart from displaying the UI defined in the nibfile. If you want your UI to accomplish anything, you need to write the code to handle its events. But the path to a running application with your UI in it is very short indeed.

Please note that BUILD-APPLICATION is a work in progress. It can easily build a working Cocoa application, but it still has limitations that may in some cases prove inconvenient. For example, in the current version it provides no easy way to specify an application delegate different from the default. If you find the current limitations of BUILD-APPLICATION too restrictive, and want to try extending it for your use, you can find the source code for it in "ccl/cocoa-ide/build-application.lisp". You can see the default values used to populate the "Info.plist" file in "ccl/cocoa-ide/builder-utilities.lisp".

For more information on how to use BUILD-APPLICATION, see the Currency Converter example in "ccl/examples/cocoa/currency-converter/".

ccl -b -e "(require :cocoa)" -e "(require :build-application)" -e "(ccl::build-application :name \"Foo\")"
      

Clozure CL provides facilities which enable multiple threads of execution (threads, sometimes called lightweight processes or just processes, though the latter term shouldn't be confused with the OS's notion of a process) within a lisp session. This document describes those facilities and issues related to multithreaded programming in Clozure CL.

Wherever possible, I'll try to use the term "thread" to denote a lisp thread, even though many of the functions in the API have the word "process" in their name. A lisp-process is a lisp object (of type CCL:PROCESS) which is used to control and communicate with an underlying native thread. Sometimes, the distinction between these two (quite different) objects can be blurred; other times, it's important to maintain.

Lisp threads share the same address space, but maintain their own execution context (stacks and registers) and their own dynamic binding context.

Traditionally, Clozure CL's threads have been cooperatively scheduled: through a combination of compiler and runtime support, the currently executing lisp thread arranged to be interrupted at certain discrete points in its execution (typically on entry to a function and at the beginning of any looping construct). This interrupt occurred several dozen times per second; in response, a handler function might observe that the current thread had used up its time slice and another function (the lisp scheduler) would be called to find some other thread that was in a runnable state, suspend execution of the current thread, and resume execution of the newly executed thread. The process of switching contexts between the outgoing and incoming threads happened in some mixture of Lisp and assembly language code; as far as the OS was concerned, there was one native thread running in the Lisp image and its stack pointer and other registers just happened to change from time to time.

Under Clozure CL's cooperative scheduling model, it was possible (via the use of the CCL:WITHOUT-INTERRUPTS construct) to defer handling of the periodic interrupt that invoked the lisp scheduler; it was not uncommon to use WITHOUT-INTERRUPTS to gain safe, exclusive access to global data structures. In some code (including much of Clozure CL itself) this idiom was very common: it was (justifiably) believed to be an efficient way of inhibiting the execution of other threads for a short period of time.

The timer interrupt that drove the cooperative scheduler was only able to (pseudo-)preempt lisp code: if any thread called a blocking OS I/O function, no other thread could be scheduled until that thread resumed execution of lisp code. Lisp library functions were generally attuned to this constraint, and did a complicated mixture of polling and "timed blocking" in an attempt to work around it. Needless to say, this code is complicated and less efficient than it might be; it meant that the lisp was a little busier than it should have been when it was "doing nothing" (waiting for I/O to be possible.)

For a variety of reasons - better utilization of CPU resources on single and multiprocessor systems and better integration with the OS in general - threads in Clozure CL 0.14 and later are preemptively scheduled. In this model, lisp threads are native threads and all scheduling decisions involving them are made by the OS kernel. (Those decisions might involve scheduling multiple lisp threads simultaneously on multiple processors on SMP systems.) This change has a number of subtle effects:

As a broad generalization: code that's been aggressively tuned to the constraints of the cooperative scheduler may need to be redesigned to work well with the preemptive scheduler (and code written to run under Clozure CL's interface to the native scheduler may be less portable to other CL implementations, many of which offer a cooperative scheduler and an API similar to Clozure CL (< 0.14) 's.) At the same time, there's a large overlap in functionality in the two scheduling models, and it'll hopefully be possible to write interesting and useful MP code that's largely independent of the underlying scheduling details.

The keyword :OPENMCL-NATIVE-THREADS is on *FEATURES* in 0.14 and later and can be used for conditionalization where required.

Much of the functionality described above is similar to that provided by Clozure CL's cooperative scheduler, some other parts of which make no sense in a native threads implementation.

Older versions of Clozure CL used what are often called "user-mode threads", a less versatile threading model which does not require specific support from the operating system. This section discusses how to port code which was written for that mode.

It's hard to give step-by-step instructions; there are certainly a few things that one should look at carefully:

In the "tty world", Clozure CL starts out with 2 lisp-level threads:

? :proc
1 : -> listener     [Active]
0 :    Initial      [Active]
    

If you look at a running Clozure CL with a debugging tool, such as GDB, or Apple's Thread Viewer.app, you'll see an additional kernel-level thread on Darwin; this is used by the Mach exception-handling mechanism.

The initial thread, conveniently named "initial", is the one that was created by the operating system when it launched Clozure CL. It maps the heap image into memory, does some Lisp-level initialization, and, when the Cocoa IDE isn't being used, creates the thread "listener", which runs the top-level loop that reads input, evaluates it, and prints the result.

After the listener thread is created, the initial thread does "housekeeping": it sits in a loop, sleeping most of the time and waking up occasionally to do "periodic tasks". These tasks include forcing output on specified interactive streams, checking for and handling control-C interrupts, etc. Currently, those tasks also include polling for the exit status of external processes and handling some kinds of I/O to and from those processes.

In this environment, the initial thread does these "housekeeping" activities as necessary, until ccl:quit is called; quitting interrupts the initial thread, which then ends all other threads in as orderly a fashion as possible and calls the C function #_exit.

The short-term plan is to handle each external-process in a dedicated thread; the worst-case behavior of the current scheme can involve busy-waiting and excessive CPU utilization while waiting for an external process to terminate in some cases.

The Cocoa features use more threads. Adding a Cocoa listener creates two threads:

      ? :proc
      3 : -> Listener     [Active]
      2 :    housekeeping  [Active]
      1 :    listener     [Active]
      0 :    Initial      [Active]
    

The Cocoa event loop has to run in the initial thread; when the event loop starts up, it creates a new thread to do the "housekeeping" tasks which the initial thread would do in the terminal-only mode. The initial thread then becomes the one to receive all Cocoa events from the window server; it's the only thread which can.

It also creates one "Listener" (capital-L) thread for each listener window, with a lifetime that lasts as long as the thread does. So, if you open a second listener, you'll see five threads all together:

      ? :proc
      4 : -> Listener-2   [Active]
      3 :    Listener     [Active]
      2 :    housekeeping  [Active]
      1 :    listener     [Active]
      0 :    Initial      [Active]
    

Unix signals, such as SIGINT (control-C), invoke a handler installed by the Lisp kernel. Although the OS doesn't make any specific guarantee about which thread will receive the signal, in practice, it seems to be the initial thread. The handler just sets a flag and returns; the housekeeping thread (which may be the initial thread, if Cocoa's not being used) will check for the flag and take whatever action is appropriate to the signal.

In the case of SIGINT, the action is to enter a break loop, by calling on the thread being interrupted. When there's more than one Lisp listener active, it's not always clear what thread that should be, since it really depends on the user's intentions, which there's no way to divine programmatically. To make its best guess, the handler first checks whether the value of ccl:*interactive-abort-process* is a thread, and, if so, uses it. If that fails, it chooses the thread which currently "owns" the default terminal input stream; see .

In the bleeding-edge version of the Cocoa support which is based on Hemlock, an Emacs-like editor, each editor window has a dedicated thread associated with it. When a keypress event comes in which affects that specific window the initial thread sends it to the window's dedicated thread. The dedicated thread is responsible for trying to interpret keypresses as Hemlock commands, applying those commands to the active buffer; it repeats this in a loop, until the window closes. The initial thread handles all other events, such as mouse clicks and drags.

This thread-per-window scheme makes many things simpler, including the process of entering a "recursive command loop" in commands like "Incremental Search Forward", etc. (It might be possible to handle all Hemlock commands in the Cocoa event thread, but these "recursive command loops" would have to maintain a lot of context/state information; threads are a straightforward way of maintaining that information.)

Currently (August 2004), when a dedicated thread needs to alter the contents of the buffer or the selection, it does so by invoking methods in the initial thread, for synchronization purposes, but this is probably overkill and will likely be replaced by a more efficient scheme in the future.

The per-window thread could probably take more responsibility for drawing and handling the screen than it currently does; -something- needs to be done to buffer screen updates a bit better in some cases: you don't need to see everything that happens during something like indentation; you do need to see the results...

When Hemlock is being used, listener windows are editor windows, so in addition to each "Listener" thread, you should also see a thread which handles Hemlock command processing.

The Cocoa runtime may make additional threads in certain special situations; these threads usually don't run lisp code, and rarely if ever run much of it.

Clozure CL supports the socket abstraction for interprocess communication. A socket represents a connection to another process, typically (but not necessarily) a TCP/IP network connection to a client or server running on some other machine on the network.

All symbols mentioned in this chapter are exported from the CCL package. As of version 0.13, these symbols are additionally exported from the OPENMCL-SOCKET package.

Clozure CL supports three types of sockets: TCP sockets, UDP sockets, and Unix-domain sockets. This should be enough for all but the most esoteric network situations. All sockets are created by make-socket. The type of socket depends on the arguments to it, as follows: