Clozure CL Documentation


1. About Clozure CL
1.1. Introduction to Clozure CL
2. Obtaining, Installing, and Running Clozure CL
2.1. Releases and System Requirements
2.2. Obtaining Clozure CL
2.3. Command Line Set Up
2.4. Personal Customization with the Init File
2.5. Command Line Options
2.6. Using Clozure CL with GNU Emacs and SLIME
2.7. Example Programs
3. Building Clozure CL from its Source Code
3.1. Building Definitions
3.2. Setting Up to Build
3.3. Building Everything
3.4. Building the kernel
3.5. Building the heap image
4. Using Clozure CL
4.1. Introduction
4.2. Trace
4.3. Unicode
4.4. Pathnames
4.5. Memory-mapped Files
4.6. Static Variables
4.7. Saving Applications
5. The Clozure CL IDE
5.1. Introduction
5.2. Building the IDE
5.3. Running the IDE
5.4. IDE Features
5.5. IDE Sources
5.6. The Application Builder
6. Programming with Threads
6.1. Threads Overview
6.2. (Intentionally) Missing Functionality
6.3. Implementation Decisions and Open Questions
6.4. Porting Code from the Old Thread Model
6.5. Background Terminal Input
6.6. The Threads which Clozure CL Uses for Its Own Purposes
6.7. Threads Dictionary
7. Programming with Sockets
7.1. Overview
7.2. Sockets Dictionary
8. Running Other Programs as Subprocesses
8.1. Overview
8.2. Examples
8.3. Limitations and known bugs
8.4. External-Program Dictionary
9. Streams
9.1. Stream Extensions
9.2. Creating Your Own Stream Classes with Gray Streams
10. Writing Portable Extensions to the Object System using the MetaObject Protocol
10.1. Overview
10.2. Implementation status
10.3. Concurrency issues
11. Profiling
11.1. Using the Linux oprofile system-level profiler
11.2. Using Apple's CHUD metering tools
12. The Foreign-Function Interface
12.1. Specifying And Using Foreign Types
12.2. Foreign Function Calls
12.3. Referencing and Using Foreign Memory Addresses
12.4. The Interface Database
12.5. Using Interface Directories
12.6. Using Shared Libraries
12.7. The Interface Translator
12.8. Case-sensitivity of foreign names in CCL
12.9. Reading Foreign Names
12.10. Tutorial: Using Basic Calls and Types
12.11. Tutorial: Allocating Foreign Data on the Lisp Heap
12.12. The Foreign-Function-Interface Dictionary
13. The Objective-C Bridge
13.1. Changes in 1.2
13.2. Using Objective-C Classes
13.3. Instantiating Objective-C Objects
13.4. Calling Objective-C Methods
13.5. Defining Objective-C Classes
13.6. Defining Objective-C Methods
13.7. Loading Frameworks
13.8. How Objective-C Names are Mapped to Lisp Symbols
14. Platform-specific notes
14.1. Overview
14.2. Unix/Posix/Darwin Features
14.3. Cocoa Programming in Clozure CL
14.4. Building an Application Bundle
14.5. Recommended Reading>
14.6. Operating-System Dictionary
15. Understanding and Configuring the Garbage Collector
15.1. Heap space allocation
15.2. The Ephemeral GC
15.3. GC Page reclamation policy
15.4. "Pure" areas are read-only, paged from image file
15.5. Weak Hash Tables
15.6. Garbage-Collection Dictionary
16. Implementation Details of Clozure CL
16.1. Threads and exceptions
16.2. Register usage and tagging
16.3. Heap Allocation
16.4. GC details
16.5. The ephemeral GC
16.6. Fasl files
16.7. The Objective-C Bridge
17. Modifying Clozure CL
17.1. Contributing Code Back to the Clozure CL Project
17.2. Using Clozure CL in "development" and in "user" mode
17.3. The Kernel Debugger
17.4. Using AltiVec in Clozure CL LAP functions
17.5. Development-Mode Dictionary
18. Questions and Answers
18.1. How can I do nonblocking (aka "unbuffered" and "raw") IO?
18.2. I'm using the graphics demos. Why doesn't the menubar change?
18.3. I'm using Slime and Cocoa. Why doesn't *standard-output* seem to work?
Glossary of Terms
Symbol Index

List of Tables

3.1. Platform-specific filename conventions
4.1. Line Termination Keywords

Chapter 1. About Clozure CL

1.1. Introduction to Clozure CL

Clozure CL is a fast, mature, open source Common Lisp implementation that runs on Linux, Mac OS X and BSD on either Intel x86-64 or PPC. Clozure CL was forked from Macintosh Common Lisp (MCL) in 1998 and the development has been entirely separate since. Ports to IA32 and Windows are under development.

When it was forked from MCL in 1998, the new Lisp was named OpenMCL. Recently, Clozure renamed its Lisp to Clozure CL, partly because its ancestor MCL has lately been released as open source. Clozure thought it might be confusing for users if there were two independent open-source projects with such similar names. The new name also reflects Clozure CL's current status as the flagship product of Clozure Associates.

Furthermore, the new name refers to Clozure CL's ancestry: in its early years, MCL was known as Coral Common Lisp, or "CCL". For years the package that contains most of Clozure CL's implementation-specific symbols has been named "CCL", an acronym that once stood for the name of the Lisp product. It seems fitting that "CCL" once again stands for the name of the product.

Some commands and source files may still refer to "OpenMCL" instead of Clozure CL.

Clozure CL compiles to native code and supports multithreading using native OS threads. It includes a foreign-function interface, and supports both Lisp code that calls external code, and external code that calls Lisp code. Clozure CL can create standalone executables on all supported platforms.

On Mac OS X, Clozure CL supports building GUI applications that use OS X's native Cocoa frameworks, and the OS X distributions include an IDE written with Cocoa, and distributed with complete sources.

On all supported platforms, Clozure CL can run as a command-line process, or as an inferior Emacs process using either SLIME or ILISP.

Features of Clozure CL include

  • Very fast compilation speed.

  • A fast, precise, compacting, generational garbage collector written in hand-optimized C. The sizes of the generations are fully configurable. Typically, a generation can be collected in a millisecond on modern systems.

  • Fast execution speed, competitive with other Common Lisp implementations on most benchmarks.

  • Robust and stable. Customers report that their CPU-intensive, multi-threaded applications run for extended periods on Clozure CL without difficulty.

  • Full native OS threads on all platforms. Threads are automatically distributed across multiple cores. The API includes support for shared memory, locking, and blocking for OS operations such as I/O.

  • Full Unicode support.

  • Full SLIME integration.

  • An IDE on Mac OS X, fully integrated with the Macintosh window system and User Interface standards.

  • Excellent debugging facilities. The names of all local variables are available in a backtrace.

  • A complete, mature foreign function interface, including a powerful bridge to Objective-C and Cocoa on Mac OS X.

  • Many extensions including: files mapped to Common Lisp vectors for fast file I/O; thread-local hash tables and streams to eliminate locking overhead; cons hashing support; and much more

  • Very efficient use of memory

Although it's an open-source project, available free of charge under a liberal license, Clozure CL is also a fully-supported product of Clozure Associates. Clozure continues to extend, improve, and develop Clozure CL in response to customer and user needs, and offers full support and development services for Clozure CL.

Chapter 2. Obtaining, Installing, and Running Clozure CL

2.1. Releases and System Requirements

Version 1.2 is the latest release of Clozure CL as of April 2008. It is intended to be a more stable release and follow a more regular release schedule than previous versions. It is easier for users who wish to track the "bleeding edge" of development to do so.

Versions 1.2 is available for five platform configurations:

  • Linux on PowerPC (32-bit and 64-bit implementations)

  • Mac OS X on PowerPC (32-bit and 64-bit implementations)

  • Linux on X86-64 (64-bit implementation)

  • Mac OS X on X86-64 (64-bit implementation)

  • FreeBSD on X86-64 (64-bit implementation)

A 64-bit version of Clozure CL requires a 64-bit processor running a 64-bit OS variant.

There are ongoing efforts to port Clozure CL to the Windows operating system and to 32-bit x86 processors.

Additional platform-specific information is given in the following subsections.

Older versions are still available for downloading as tarballs. Version 1.0 was a stable version released in late 2005. Version 1.1 was under active development until late 2007. A final 1.1 release was never made. It was distributed as a series of development "snapshots" and CVS updates. 1.1 snapshots introduced support for x86-64 platforms, internal use of Unicode, and many other features, but were moving targets.

2.1.1. LinuxPPC

Clozure CL requires version 2.2.13 (or later) of the Linux kernel and version 2.1.3 (or later) of the GNU C library (glibc) at a bare minimum.

2.1.2. Linux X8664

Clozure CL runs on relatively recent Linux distributions for the x86-64 architecture. It requires a Linux with Thread Local Storage support in the toolchain and standard libraries, and the New Posix Thread Library (NPTL). Fortunately, these features seem to be present in all current Linux distributions for x86-64, though there may be some problems with early Linux distributions for x86-64. Some GCC versions older than 4.0 on Linux have been known to have problems compiling some of the C code in the kernel; some very old Linux distributions don't follow the current ABI standards with regard to segment register usage; some early Linux kernels for x86-64 had problems mapping large regions of the address space; and so on. It's difficult to enumerate exactly what versions of which Linux distributions have what problems. A rule of thumb is that—because much of the development of Clozure CL for x86-64 took place in that time frame—Linux distributions released earlier than early 2006 may have problems running Clozure CL.

2.1.3. FreeBSD-amd64

Clozure CL runs on FreeBSD on x86-64 (FreeBSD releases generally call the platform "amd64"). Clozure CL should run under FreeBSD 6.0 or later; as of this writing, FreeBSD 7.0 is about to be released and it may be necessary for FreeBSD 7 users to install the "compat6x" package in order to use a version of Clozure CL built on FreeBSD 6.x.

2.1.4. DarwinPPC-MacOS-X

Clozure CL runs under OS X versions 10.4 and 10.5 and requires at least version 10.3.9

The 64-bit DarwinPPC version of Clozure CL requires functionality introduced in OSX 10.4 (namely, the ability to run 64-bit binaries). The 64-bit DarwinPPC version also, obviously, requires a G5 processor.

Clozure CL hasn't been tested under Darwin proper, but Clozure CL doesn't intentionally use any Mac OS X features beyond the Darwin subset and therefore it seems likely that Clozure CL would run on PPC Darwin versions that correspond to recent OSX versions.

2.1.5. Darwinx8664-MacOS-X

Clozure CL runs on 64-bit DarwinX86 (Mac OS X on Intel).

Clozure CL Darwinx8664/MacOS X requires a 64-bit processor. All Macintoshes currently sold by Apple (as of early 2008) and all Macintoshes introduced by Apple since August 2006 have such processors. However, the original MacBooks, MacBook Pros and Intel iMacs (models introduced in early 2006) used 32-bit Core Duo processors, and so Clozure CL will not (yet) run on them.

2.2. Obtaining Clozure CL

There two main ways to obtain Clozure CL. For Mac OS X, there are disk images that can be used to install Clozure CL in the usual Macintosh way. For other OSes, Subversion is the best way to obtain Clozure CL. Mac OS X users can also use Subversion if they prefer. Tarballs are available for those who prefer them, but if you have Subversion installed, it is simpler and more flexible to use Subversion than tarballs. It is easier to keep up with the bleeding edge if you are using Subversion, since disk images and tarballs are generated much less frequently than changes to Subversion.

There are three popular ways to use Clozure CL: as a stand-alone double-clickable application (Mac OS X only), as a command-line application, or with EMACS and SLIME. The following sections describe these options.

2.2.1. The Mac Way

If you are using Mac OS X then you can install and use Clozure CL in the usual Macintosh way. Download and mount a disk image, then drag Clozure CL to the Applications folder. After that you can double-click the Clozure CL application to run it. The disk images are available at ftp://clozure.com/pub/testing/

So that Clozure CL can locate its source code, and for other reasons explained in Section 4.4.2, “Predefined Logical Hosts”, you should either put the ccl directory in the same directory as the Clozure CL application, or else put the Clozure CL application in the ccl directory. If you use a shell, you can set the value of the CCL_DEFAULT_DIRECTORY environment variable to explicitly indicate the location of the ccl directory. If you choose to do that, then the ccl and the Clozure CL application can each be in any location you find convenient.

2.2.2. Getting Clozure CL with Subversion

It is very easy to download, install, and build Clozure CL using Subversion. This is the preferred way to get either the latest, or a specific version of Clozure CL, unless you prefer the Mac Way. Subversion is a source code control system that is in wide usage. Most modern OSes come with subversion pre-installed. A complete, buildable and runnable set of Clozure CL sources and binaries can be retrieved by doing one subversion checkout.

One subversion command creates a ccl directory with runnable binaries, and a complete set of buildable sources. To get the bleeding edge Clozure CL for Darwin x8664, at the command line type:

          
svn co http://svn.clozure.com/publicsvn/openmcl/trunk/darwinx8664/ccl
        

To get the 1.2 version of Clozure CL type:

          
svn co http://svn.clozure.com/publicsvn/openmcl/release/1.2/darwinx8664/ccl
        

These examples fetch the complete sources and binaries for the Darwin X8664 build of Clozure CL. You can fetch a different version by substituting its name in place of "darwinx8664". Current available versions are:

  • darwinppc

  • darwinx8664

  • freebsdx8664

  • linuxppc

  • linuxx8664

These distributions contain complete sources and binaries. They use Subversion's "externals" features to share common sources; the majority of source code is the same across all versions.

Once the checkout is complete you can build Clozure CL by running the lisp kernel and executing the rebuild-ccl function. For example:

          
joe:ccl> ./dx86cl64
Welcome to Clozure Common Lisp Version 1.2  (DarwinX8664)!
? (rebuild-ccl :full t)

<lots of compilation output>

  ? (quit)
  joe:ccl>
        

2.2.2.1. Checking Subversion Installation

If svn co doesn't work, then make sure that Subversion is installed on your system. Bring up a command line shell and type:

          
shell> svn
        

If Subversion is installed, you will see something like:

          
Type 'svn help' for usage
        

If Subversion is not installed, you will see something like:

          
-bash: svn: command not found
        

If Subversion is not installed, you'll need to figure out how to install it on your OS. You can find information about obtaining and installing Subversion at the Subversion Packages page.

2.2.3. Tarballs

Tarballs are available at ftp://clozure.com/pub/testing/. Download and extract one on your local disk. Then edit the Clozure CL shell script to set the value of CCL_DEFAULT_DIRECTORY and start up the appropriate Clozure CL kernel. See Section 2.3.1, “The ccl Shell Script” for more information about the Clozure CL shell scripts.

2.3. Command Line Set Up

Sometimes it's convenient to use Clozure CL from a Unix shell command line. This is especially true when using Clozure CL as a way to run Common Lisp utilities.

2.3.1. The ccl Shell Script

Clozure CL needs to be able to find the ccl directory in order to support features such as require and provide, access to foreign interface information (see The Interface Database) and the Lisp build process (see Building Clozure CL from its Source Code). Specifically, it needs to set up logical pathname translations for the "ccl:" logical host. If this logical host isn't defined (or isn't defined correctly), some things might work, some things might not, and it'll generally be hard to invoke and use Clozure CL productively.

Clozure CL uses the value of the environment variable CCL_DEFAULT_DIRECTORY to determine the filesystem location of the ccl directory; the ccl shell script is intended to provide a way to invoke Clozure CL with that environment variable set correctly.

There are two versions of the shell script: "ccl/scripts/ccl" is used to invoke 32-bit implementations of Clozure CL and "ccl/scripts/ccl64" is used to invoke 64-bit implementations.

To use the script:

  1. Copy the script to a directory that is on your PATH. This is often /usr/local/bin or ~/bin. It is better to do this than to add ccl/scripts to your PATH, because the script needs to be edited, and editing it in-place means that Subversion sees the script as modified..

  2. Edit the definition of CCL_DEFAULT_DIRECTORY near the beginning of the shell script so that it refers to your ccl directory. Alternately, set the value of the CCL_DEFAULT_DIRECTORY environment variable in your .cshrc, .tcshrc, .bashrc,.bash_profile, .MacOSX/environment.plist, or wherever you usually set environment variables. If there is an existing definition of the variable, the ccl script will not override it. The shell script sets a local variable (OPENMCL_KERNEL) to the standard name of the Clozure CL kernel approprate for the platform, as determined by 'uname -s'. You might prefer to set this variable manually in the shell script.

  3. Ensure that the shell script is executable, for example:

    $ chmod +x ~/ccl/ccl/scripts/ccl64

    This command grants execute permission to the named script. If you are using a 32-bit platform, substitute "ccl" in place of "ccl64".

    Warning

    The above command won't work if you are not the owner of the installed copy of Clozure CL. In that case, you can use the "sudo" command like this:

    $ sudo chmod +x ~/ccl/ccl/scripts/ccl64

    Give your password when prompted.

    If the "sudo" command doesn't work, then you are not an administrator on the system you're using, and you don't have the appropriate "sudo" permissions. In that case you'll need to get help from the system's administrator.

Note that most people won't need both ccl and ccl64 scripts. You only need both if you sometimes run 32-bit Clozure CL and sometimes run 64-bit Clozure CL. You can rename the script that you use to whatever you want. For example, if you are on a 64-bit system, and you only use Clozure CL in 64-bit mode, then you can rename ccl64 to ccl so that you only need to type "ccl" to run it.

Once this is done, it should be possible to invoke Clozure CL by typing ccl or ccl64 at a shell prompt:

> ccl [args ...]
Welcome to Clozure CL Version 1.2 (DarwinPPC32)!
?
      

The ccl shell script passes all of its arguments to the Clozure CL kernel. See Section 2.3.2, “Invocation” for more information about these arguments. When invoked this way, the Lisp should be able to initialize the "ccl:" logical host so that its translations refer to the "ccl" directory. To test this, you can call probe-file in Clozure CL's read-eval-print loop:

? (probe-file "ccl:level-1;level-1.lisp")  ;returns the physical pathname of the file
#P"/Users/alms/my_lisp_stuff/ccl/level-1/level-1.lisp"
      

2.3.2. Invocation

Assuming that the shell script is properly installed, it can be used to invoke Clozure CL from a shell prompt:

shell>ccl args
	    

ccl runs a 32-bit session; ccl64 runs a 64-bit session.

2.4. Personal Customization with the Init File

By default Clozure CL tries to load the file "home:ccl-init.lisp" or the compiled "home:ccl-init.fasl" upon starting up. For the sake of backward compatibility, it also tries to load the file "home:openmcl-init.lisp", or its compiled equivalent. Clozure CL does this by executing (load "home:ccl-init"). If it's unable to load the file (for example because the file doesn't exist), Clozure CL doesn't signal an error or warning, it just completes its startup normally.

The "home:" prefix to the filename is a Common Lisp logical host, which Clozure CL initializes to refer to your home directory. Clozure CL therefore looks for either of the files ~/ccl-init.lisp or ~/ccl-init.fasl.

Because the init file is loaded the same way as normal Lisp code is, you can put anything you want in it. For example, you can change the working directory, and load packages that you use frequently.

To suppress the loading of this init-file, invoke Clozure CL with the --no-init option.

2.5. Command Line Options

When using Clozure CL from the command line, the following options may be used to modify its behavior. The exact set of Clozure CL command-line arguments may vary per platform and slowly changes over time. The current set of command line options may be retrieved by using the --help option.

  • -h (or --help). Provides a definitive (if somewhat terse) summary of the command line options accepted by the Clozure CL implementation and then exits.

  • -V (or --version). Prints the version of Clozure CL then exits. The version string is the same value that is returned by LISP-IMPLEMENTATION-VERSION.

  • -K character-encoding-name (or --terminal-encoding character-encoding-name). Specifies the character encoding to use for *TERMINAL-IO* (see Section 4.3.4, “Character Encodings”). Specifically, the character-encoding-name string is uppercased and interned in the KEYWORD package. If an encoding named by that keyword exists, CCL:*TERMINAL-CHARACTER-ENCODING-NAME* is set to the name of that encoding. CCL:*TERMINAL-CHARACTER-ENCODING-NAME* defaults to NIL, which is a synonym for :ISO-8859-1.

    For example:

    shell> ccl -K utf-8
    	      

    has the effect of making the standard CL streams use :UTF-8 as their character encoding.

  • -n (or --no-init). If this option is given, the init file is not loaded. This is useful if Clozure CL is being invoked by a shell script that should not be affected by whatever customizations a user might have in place.

  • -e form (or --eval). An expression is read (via READ-FROM-STRING) from the string form and evaluated. If form contains shell metacharacters, it may be necessary to escape or quote them to prevent the shell from interpreting them.

  • -l path (or --load path). Loads file specified by path.

  • -T n (or --set-lisp-heap-gc-threshold n). Sets the Lisp gc threshold to n. (see Section 15.3, “GC Page reclamation policy”

  • -Q (or --quiet). Suppresses printing of heralds and prompts when the --batch command line option is specified.

  • -R n (or --heap-reserve). Reserves n bytes for heap expansion. The default is 549755813888. (see Section 15.1, “Heap space allocation”)

  • -S n (or --stack-size n). Sets the size of the initial control stack to n. (see Section 6.3.1, “Thread Stack Sizes”)

  • -Z n (or --thread-stack-size n). Sets the size of the first thread's stack to n. (see Section 6.3.1, “Thread Stack Sizes”)

  • -b (or --batch). Execute in "batch mode". End-of-file from *STANDARD-INPUT* causes Clozure CL to exit, as do attempts to enter a break loop.

  • --no-sigtrap An obscure option for running under GDB.

  • -I image-name (or --image-name image-name). Specifies the image name for the kernel to load. Defaults to the kernel name with ".image" appended.

The --load and --eval options can each be provided multiple times. They're executed in the order specified on the command line, after the init file (if there is one) is loaded and before the toplevel read-eval-print loop is entered.

2.6. Using Clozure CL with GNU Emacs and SLIME

A very common way to use Clozure CL is to run it within the GNU Emacs editor, using a Lisp interface called SLIME ("Superior Lisp Interaction Mode for Emacs"). SLIME is an Emacs package designed to provide good support within Emacs for any of several Common Lisp implementations; one of the supported implementations is Clozure CL. This page describes how you can download SLIME and set it up to work with your Clozure CL installation.

Why use SLIME? With SLIME, you can do the following things from within an Emacs editing session:

  • run and control Lisp

  • evaluate, compile, and load files or expressions

  • macroexpand expressions

  • fetch documentation and source code for Lisp symbols

  • autocomplete symbols and package names

  • cross-reference function calls

  • examine stack traces and debug errors

For complete information about SLIME, see the SLIME home page. The SLIME home page provides up-to-date downloads, plus documentation, tutorials, and instructional screencasts.

2.6.1. Assumptions and Requirements

In order to simplify these instructions, we'll make several assumptions about your system. Specifically, we assume:

  • You have a working installation of GNU Emacs. If you don't have a working copy of GNU Emacs, see the web page on obtaining Emacs. If you prefer to use XEmacs instead of GNU Emacs, these instructions should still work; SLIME supports XEmacs Version21. Mac OS X includes an Emacs installation. If you want to look into different versions, you can check out theEmacsWiki, which maintains a page, EmacsForMacOS, that provides much more information about using Emacs on the Mac.

    A popular version of Emacs among Mac users is Aquamacs. This application is a version of GNU Emacs with a number of customizations meant to make it behave more like a standard Macintosh application, with windows, a menubar, etc. Aquamacs includes SLIME; if you like Aquamacs then you can use SLIME right away, without getting and installing it separately. You just need to tell SLIME where to find your installation of Clozure CL.

  • You have a working copy of Clozure CL, installed in "~/ccl"If you prefer to install Clozure CL in some directory other than"~/ccl" then these instructions still work, but you must remember to use your path to your ccl directory instead of the one that we give here.

  • You install emacs add-ons in the folder "~/emacs/site/"If this directory doesn't exist on your system, you can just create it.If you prefer to install Emacs add-ons in some place other than"~/emacs/site/" then you must remember to use your path to Emacs add-ons in place of ours.

2.6.2. Getting SLIME

You can get SLIME from the SLIME Home Page. Stable releases and CVS snapshots are available as archive files, or you can follow the instructions on the SLIME Home Page to check out the latest version from their CVS repository.

It's worth noting that stable SLIME releases happen very seldom, but the SLIME developers often make changes and improvements that are available through CVS updates. If you asked the SLIM developers, they would most likely recommend that you get SLIME from their CVS repository and update it frequently.

Whether you get it from CVS, or download and unpack one of the available archives, you should end up with a folder named "slime" that contains the SLIME distribution.

2.6.3. Installing SLIME

Once you have the "slime" folder described in the previous section, installation is a simple matter of copying the folder to the proper place. You can drag it into the "~/emacs/site/" folder, or you can use a terminal command to copy it there. For example, assuming your working directory contains the unpacked "slime" folder:

$ cp -R slime ~/emacs/site/

That's all it takes.

2.6.4. Telling Emacs About SLIME

Once SLIME and Clozure CL are installed, you just need to add a line to your "~/.emacs" file that tells SLIME where to find the script that runs Clozure CL:

(setq inferior-lisp-program "~/ccl/scripts/ccl64")

or

(setq inferior-lisp-program "~/ccl/scripts/ccl")

Warning

Aquamacs users should add this line to the file "~/Library/Preferences/Aquamacs Emacs/Preferences.el".

2.6.5. Running Clozure CL with SLIME

Once the preparations in the previous section are complete, exit Emacs and restart it, to ensure that it reads the changes you made in your ".emacs" file (alternatively, you could tell Emacs to reload the ".emacs" file). If all went well, you should now be ready to run Clozure CL using SLIME.

To run Clozure CL, execute the command "M-x slime". SLIME should start an Clozure CL session in a new buffer. (If you are unfamiliar with the Emacs notation "M-x command", see the GNU Emacs FAQ; specifically, take a look at questions 1, 2, and 128.)

2.6.6. What if a New Version of Clozure CL Breaks SLIME?

Sometimes you'll get a new version of Clozure CL, set up Emacs to use it with SLIME, and SLIME will fail. Most likely what has happened is that the new version of Clozure CL has a change in the output files produced by the compiler (Clozure CL developers will say "the fasl version has changed." fasl stands for "fast load" aka compiled files). This problem is easy to fix: just delete the existing SLIME fasl files. The next time you launch Emacs and start SLIME, it will automatically recompile the Lisp files, and that should fix the problem.

SLIME's load process stores its fasl files in a hidden folder inside your home folder. The path is

~/.slime/fasl

You can use a shell command to remove the fasl files, or remove them using your system's file browser.

Note for Macintosh Users: The leading "." character in the ".slime" folder's name prevents the Finder from showing this folder to you. If you use the "Go To Folder" menu item in the Finder's "Go" menu, you can type in "~/.slime" and the Finder will show it to you. You can then drag the "fasl" folder to the trash.

2.6.7. Known Bugs

SLIME has not been updated to account for recent changes made in Clozure CL to support x86-64 processors. You may run into bugs running on those platforms.

The SLIME backtrace sometimes shows incorrect information.

return-from-frame and apply-in-frame do not work reliably. (If they work at all, it's pure luck.)

Some versions of Emacs on the Macintosh may have trouble finding the shell script that runs Clozure CL unless you specify a full path to it. See the above section "Telling Emacs About SLIME" to learn how to specify the path to the shell script.

For more help with Clozure CL on Mac OS X, consult the Clozure CL mailing lists. You can find information about the mailing lists on the Clozure CL wiki.

2.7. Example Programs

A number (ok, a small number), of example programs are distributed in the "ccl:examples;" directory of the source distribution. See the README-OPENMCL-EXAMPLES text file in that directory for information about prerequisites and usage.

Some of the example programs are derived from C examples in textbooks, etc.; in those cases, the original author and work are cited in the source code.

Unless the original author or contributor claims other rights, you're free to incorporate any of this example code or derivative thereof in any of your own works without restriction. In doing so, you agree that the code was provided "as is", and that no other party is legally or otherwise responsible for any consequences of your decision to use it.

If you've developed Clozure CL examples that you'd like to see added to the distribution, please send mail to the Clozure CL mailing lists. Any such contributions would be welcome and appreciated (as would bug fixes and improvements to the existing examples.)

Chapter 3. Building Clozure CL from its Source Code

Clozure CL, like many other Lisp implementations, consists of a kernel and a heap image. The kernel is an ordinary C program, and is built with a C compiler. It provides very basic and fundamental facilities, such as memory management, garbage collection, and bootstrapping. All the higher-level features are written in Lisp, and compiled into the heap image. Both parts are needed to have a working Lisp implementation; neither the kernel nor the heap image can stand alone.

You may already know that, when you have a C compiler which is written in C, you need a working C compiler to build the compiler. Similarly, the Clozure CL heap image includes a Lisp compiler, which is written in Lisp. You therefore need a working Lisp compiler in order to build the Lisp heap image.

Where will you get a working Lisp compiler? No worries; you can use a precompiled copy of a (slightly older and compatible) version of Clozure CL. This section explains how to do all this.

In principle it should be possible to use another implementation of Common Lisp as the host compiler, rather than an older Clozure CL; this would be a challenging and experimental way to build, and is not described here.

3.1. Building Definitions

The following terms are used in subsequent sections; it may be helpful to refer to these definitions.

fasl files are the object files produced by compile-file. fasl files store the machine code associated with function definitions and the external representation of other lisp objects in a compact, machine-readable form. fasl is short for “FASt Loading”. Clozure CL uses different pathname types (extensions) to name fasl files on different platforms; see Table 3.1, “Platform-specific filename conventions”

The Lisp kernel is a C program with a fair amount of platform-specific assembly language code. Its basic job is to map a lisp heap image into memory, transfer control to some compiled lisp code that the image contains, handle any exceptions that occur during the execution of that lisp code, and provide various other forms of runtime support for that code. Clozure CL uses different filenames to name the lisp kernel files on different platforms; see Table 3.1, “Platform-specific filename conventions”.

A heap image is a file that can be quickly mapped into a process' address space. Conceptually, it's not too different from an executable file or shared library in the OS's native format (ELF orMach-O/dyld format); for historical reasons, Clozure CL's own heap images are in their own (fairly simple) format. The term full heap image refers to a heap image file that contains all of the code and data that comprise Clozure CL. Clozure CL uses different filenames to name the standard full heap image files on different platforms; see Table 3.1, “Platform-specific filename conventions”.

A bootstrapping image is a minimal heap image used in the process of building Clozure CL itself. The bootstrapping image contains just enough code to load the rest of Clozure CL from fasl files. It may help to think of the bootstrapping image as the egg and the full heap image as the chicken. Clozure CL uses different filenames to name the standard bootstrapping image files on different platforms; see Table 3.1, “Platform-specific filename conventions” .

Each supported platform (and possibly a few as-yet-unsupported ones) has a uniquely named subdirectory of ccl/lisp-kernel/; each such contains a Makefile and may contain some auxiliary files (linker scripts, etc.) that are used to build the lisp kernel on a particular platform.The platform-specific name of the kernel build directory is described in Table 3.1, “Platform-specific filename conventions”.

3.1.1. Platform-specific filename conventions

Table 3.1. Platform-specific filename conventions

Platform kernel full-image boot-image fasl extension kernel-build directory
DarwinPPC32 dppccl dppccl.image ppc-boot.image .dfsl darwinppc
LinuxPPC32 ppccl PPCCL ppc-boot .pfsl linuxppc
DarwinPPC64 dppccl64 dppccl64.image ppc-boot64.image .d64fsl darwinppc64
LinuxPPC64 ppccl64 PPCCL64 ppc-boot64 .p64fsl linuxppc64
LinuxX8664 lx86cl64 LX86CL64 x86-boot64 .lx64fsl linuxx8664
DarwinX8664 dx86cl64 dx86cl64.image x86-boot64.image .dx64fsl darwinx8664
FreeBSDX8664 fx86cl64 FX86CL64 fx86-boot64 .fx64fsl freebsdx8664

3.2. Setting Up to Build

There are currently three versions of Clozure CL that you might want to use (and therefore might want to build from source):

  • Version 1.0 - the more stable version

  • Version 1.1 - the more recent version, which runs on more platforms (including x86-64 platforms) and supports Unicode

  • Version 1.2 - supports (at least) all of the features and platforms of 1.1, but is distributed and updated differently

All versions are available for download from the Clozure CL website in the form of archives that contain everything you need to work with Clozure CL, including the complete sources, a full heap image, and the foreign-function interface database.

Version 1.0 archives are named openmcl-platform-all-1.0.tar.gz, where platform is either darwinppc, darwinppc64, or linuxppc. Because version 1.0 is no longer undergoing active development, you won't ever need to update these sources.

Version 1.1 archives are named openmcl-platform-snapshot-yymmdd.tar.gz, where platform is either darwinppc, darwinx8664, linuxppc, linuxx8664, or freebsdx8664, and where yymmdd is the year, month, and day the snapshot was released.

Because version 1.1 is undergoing active development, there may be times when you want to get sources that are more recent than the most recent snapshot and use them to build yourself a new bleeding-edge Clozure CL. In that case, you should download and install the latest snapshot, and then update your sources via CVS. At that point you can rebuild and you'll have the latest and greatest Clozure CL. The snapshot has CVS working-copy information in it, so all you need to do to update is

$ cd ccl
$ cvs login             # password is "cvs"
# this step only needs to be done once,
# that'll store the trivially encrypted
# password in ~/.cvspas
$ cvs update
    

Unless you tell it to, cvs won't delete ("prune") empty directories or create new ones when the repository changes. It's generally a good habit to use

$ cvs update -d -P      # create dirs as needed, prune empty ones
    

Version 1.2 archives follow naming conventions that are similar to those used by 1.0 (though more platforms are supported.) However, rather than containing CVS working-copy information, the 1.2 (and, presumably, later) archives contain metainformation used by the Subversion (svn) source-code control system.

Subversion client programs are pre-installed on OSX 10.5 and later and are typically either pre-installed or readily available on Linux and FreeBSD platforms. The Subversion web page contains links to subversion client programs for many platforms; users of OSX versions 10.4 and earlier can also install Subversion clients via Fink or MacPorts.

3.3. Building Everything

Given that you now have everything you need, do the following in a running Clozure CL to bring your Lisp system completely up to date.

? (ccl:rebuild-ccl :full t)
    

That call to the function rebuild-ccl performs the following steps:

  • Deletes all fasl files and other object files in the ccl directory tree

  • Runs an external process that does a make in the current platform's kernel build directory to create a new kernel

  • Does (compile-ccl t) in the running lisp, to produce a set of fasl files from the “higher level” lisp sources.

  • Does (xload-level-0 :force) in the running lisp, to compile the lisp sources in the “ccl:level-0;” directory into fasl files and then create a bootstrapping image from those fasl files.

  • Runs another external process, which causes the newly compiled lisp kernel to load the new bootstrapping image. The bootsrtrapping image then loads the “higher level” fasl files and a new copy of the platform's full heap image is then saved.

If all goes well, it'll all happen without user intervention and with some simple progress messages. If anything goes wrong during execution of either of the external processes, the process output is displayed as part of a lisp error message.

rebuild-ccl is essentially just a short cut for running all the individual steps involved in rebuilding the system. You can also execute these steps individually, as described below.

3.4. Building the kernel

The Lisp kernel is the executable that you run to use Lisp. It doesn't actually contain the entire Lisp implementation; rather, it loads a heap image which contains the specifics—the "library", as it might be called if this was a C program. The kernel also provides runtime support to the heap image, such as garbage collection, memory allocation, exception handling, and the OS interface.

The Lisp kernel file has different names on different platforms. See Table 3.1, “Platform-specific filename conventions”. On all platforms the lisp kernel sources reside in ccl/lisp-kernel.

This section gives directions on how to rebuild the Lisp kernel from its source code. Most Clozure CL users will rarely have to do this. You probably will only need to do it if you are attempting to port Clozure CL to a new architecture or extend or enhance its kernel in some way. As mentioned above, this step happens automatically when you do

? (rebuild-ccl :full t)
      

3.4.1. Kernel build prerequisites

The Clozure CL kernel can be bult with the following widely available tools:

  • cc or gcc- the GNU C compiler

  • ld - the GNU linker

  • m4 or gm4- the GNU m4 macro processor

  • as - the GNU assembler (version 2.10.1 or later)

  • make - either GNU make or, on FreeBSD, the default BSD make program

In general, the more recent the versions of those tools, the better; some versions of gcc 3.x on Linux have difficulty compiling some of the kernel source code correctly (so gcc 4.0 should be used, if possible.) On OSX, the versions of the tools distributed with XCode should work fine; on Linux, the versions of the tools installed with the OS (or available through its package management system) should work fine if they're "recent enough". On FreeBSD, the installed version of the m4 program doesn't support some features that the kernel build process depends on; the GNU version of the m4 macroprocessor (called gm4 on FreeBSD) should be installed.

3.4.2. Using "make" to build the lisp kernel

With those tools in place, do:

shell> cd ccl/lisp-kernel/PLATFORM
shell> make
	    

That'll assemble several assembly language source files, compile several C source files, and link ../../the kernel.

3.5. Building the heap image

The initial heap image is loaded by the Lisp kernel, and provides most of the language implementation The heap image captures the entire state of a running Lisp (except for external resources, such as open files and TCP sockets). After it is loaded, the contents of the new Lisp process's memory are exactly the same as those of the old Lisp process when the image was created.

The heap image is how we get around the fact that we can't run Lisp code until we have a working Lisp implementation, and we can't make our Lisp implementation work until we can run Lisp code. Since the heap image already contains a fully-working implementation, all we need to do is load it into memory and start using it.

If you're building a new version of Clozure CL, you need to build a new heap image.

(You might also wish to build a heap image if you have a large program that is very complicated or time-consuming to load, so that you will be able to load it once, save an image, and thenceforth never have to load it again. At any time, a heap image capturing the entire memory state of a running Lisp can be created by calling the function ccl:save-application.)

3.5.1. Development cycle

Creating a new Clozure CL full heap image consists of the following steps:

  1. Using your existing Clozure CL, create a bootstrapping image

  2. Using your existing Clozure CL, recompile your updated Clozure CL sources

  3. Invoke Clozure CL with the bootstrapping image you just created (rather than with the existing full heap image).

When you invoke Clozure CL with the bootstrapping image, it starts up, loads all of the Clozure CL fasl files, and saves out a new full heap image. Voila. You've created a new heap image.

A few points worth noting:

  • There's a circular dependency between the full heap image and the bootstrapping image, in that each is used to build the other.

  • There are some minor implementation differences, but the environment in effect after the bootstrapping image has loaded its fasl files is essentially equivalent to the environment provided by the full heap image; the latter loads a lot faster and is easier to distribute, of course.

  • If the full heap image doesn't work (because of an OS compatibilty problem or other bug), it's very likely that the bootstrapping image will suffer the same problems.

Given a bootstrapping image and a set of up-to-date fasl files, the development cycle usually involves editing lisp sources (or updating those sources via cvs update), recompiling modified files, and using the bootstrapping image to produce a new heap image.

3.5.2. Generating a bootstrapping image

The bootstrapping image isn't provided in Clozure CL distributions. It can be built from the source code provided in distributions (using a lisp image and kernel provided in those distributions) using the procedure described below.

The bootstrapping image is built by invoking a special utility inside a running Clozure CL heap image to load files contained in the ccl/level-0 directory. The bootstrapping image loads several dozen fasl files. After it's done so, it saves a heap image via save-application. This process is called "cross-dumping".

Given a source distribution, a lisp kernel, and a heap image, one can produce a bootstrapping image by first invoking Clozure CL from the shell:

shell> ccl
Welcome to Clozure CL .... !
?
	  

then calling ccl:xload-level-0 at the lisp prompt

? (ccl:xload-level-0)
	  

This function compiles the lisp sources in the ccl/level-0 directory if they're newer than the corresponding fasl files and then loads the resulting fasl files into a simulated lisp heap contained in data structures inside the running lisp. That simulated heap image is then written to disk.

xload-level-0 should be called whenever your existing boot image is out-of-date with respect to the source files in ccl:level-0; :

? (ccl:xload-level-0 :force)
      

forces recompilation of the level-0 sources.

3.5.3. Generating fasl files

Calling:

? (ccl:compile-ccl)
	  

at the lisp prompt compiles any fasl files that are out-of-date with respect to the corresponding lisp sources; (ccl:compile-ccl t) forces recompilation. ccl:compile-ccl reloads newly-compiled versions of some files; ccl:xcompile-ccl is analogous, but skips this reloading step.

Unless there are bootstrapping considerations involved, it usually doesn't matter whether these files are reloaded after they're recompiled.

Calling compile-ccl or xcompile-ccl in an environment where fasl files don't yet exist may produce warnings to that effect whenever files are required during compilation; those warnings can be safely ignored. Depending on the maturity of the Clozure CL release, calling compile-ccl or xcompile-ccl may also produce several warnings about undefined functions, etc. They should be cleaned up at some point.

3.5.4. Building a full image from a bootstrapping image

To build a full image from a bootstrapping image, just invoke the kernel with the bootstrapping image as an argument

$ cd ccl                        # wherever your ccl directory is
$ ./KERNEL BOOT_IMAGE
	  

Where KERNEL and BOOT_IMAGE are the names of the kernel and boot image appropriate to the platform you are running on. See Table 3.1, “Platform-specific filename conventions”

That should load a few dozen fasl files (printing a message as each file is loaded.) If all of these files successfully load, the lisp will print a prompt. You should be able to do essentially everything in that environment that you can in the environment provided by a "real" heap image. If you're confident that things loaded OK, you can save that image.

? (ccl:save-application "image_name") ; Overwiting the existing heap image
	  

Where image_name is the name of the full heap image for your platform. See Table 3.1, “Platform-specific filename conventions”.

If things go wrong in the early stages of the loading sequence, errors are often difficult to debug; until a fair amount of code (CLOS, the CL condition system, streams, the reader, the read-eval-print loop) is loaded, it's generally not possible for the lisp to report an error. Errors that occur during these early stages ("the cold load") sometimes cause the lisp kernel debugger (see ) to be invoked; it's primitive, but can sometimes help one to get oriented.

Chapter 4. Using Clozure CL

4.1. Introduction

The Common Lisp standard allows considerable latitude in the details of an implementation, and each particular Common Lisp system has some idiosyncrasies. This chapter describes ordinary user-level features of Clozure CL, including features that may be part of the Common Lisp standard, but which may have quirks or details in the Clozure CL implementation that are not described by the standard. It also describes extensions to the standard; that is, features of Clozure CL that are not part of the Common Lisp standard at all.

4.2. Trace

Clozure CL's tracing facility is invoked by an extended version of the Common Lisp trace macro. Extensions allow tracing of methods, as well as finer control over tracing actions.

TRACE {keyword global-value}* {spec | (spec {keyword local-value}*)}* [Macro]

The trace macro encapsulates the functions named by specs, causing trace actions to take place on entry and exit from each function. The default actions print a message on function entry and exit. Keyword/value options can be used to specify changes in the default behavior.

Invoking (trace) without arguments returns a list of functions being traced.

A spec is either a symbol that is the name of a function, or an expression of the form (setf symbol), or a specific method of a generic function in the form (:method gf-name {qualifier}* ({specializer}*)), where a specializer can be the name of a class or an EQL specializer.

A spec can also be a string naming a package, or equivalently a list (:package package-name), in order to request that all functions in the package to be traced.

By default, whenever a traced function is entered or exited, a short message is printed on *trace-output* showing the arguments on entry and values on exit. Options specified as key/value pairs can be used to modify this behavior. Options preceding the function specs apply to all the functions being traced. Options specified along with a spec apply to that spec only and override any global options. The following options are supported:

:methods {T | nil}

If true, and if applied to a spec naming a generic function, arranges to trace all the methods of the generic function in addition to the generic function itself.

:inside outside-spec | ({outside-spec}*)

Inhibits all trace actions unless the current invocation of the function being traced is inside one of the outside-spec's, i.e. unless a function named by one of the outside-spec's is currently on the stack. outside-spec can name a function, a method, or a package, as above.

:if form, :condition form

Evaluates form whenever the function being traced is about to be entered, and inhibits all trace actions if